foxhunt/docs/plans/2026-03-02-additional-training-metrics.md

Additional Training Metrics Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Add 17 new Prometheus metrics (12 always-on gauges + 5 opt-in verbose) to provide full RL diagnostics, gradient health, and hyperopt intra-trial visibility across the monitoring pipeline.

Architecture: Two-tier metrics system. Tier 1 (12 gauges) — zero-cost atomic writes in every trainer's epoch loop. Tier 2 (5 metrics) — gated behind FOXHUNT_VERBOSE_METRICS=1. All metrics flow through the existing Prometheus → monitoring_service → gRPC → fxt CLI pipeline. No new queries needed (wildcard foxhunt_training_*|foxhunt_hyperopt_* already captures new metrics).

Tech Stack: Rust, prometheus crate, tonic/prost (gRPC), protobuf, Candle ML framework

Build command: SQLX_OFFLINE=true cargo check --workspace Test command: SQLX_OFFLINE=true cargo test -p <crate> --lib Clippy rules: #![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::indexing_slicing)] — use .get(), ?, .ok_or() instead.

---

Task 1: Register Tier 1 metrics + verbose gate in training_metrics.rs

Files:

• Modify: crates/common/src/metrics/training_metrics.rs

This task adds 12 new gauge registrations, 5 verbose metric registrations with env-var gating, and typed helper functions for all 17 new metrics.

Context: The file currently has 24 metrics registered in init() (lines 32-168) and typed helpers (lines 174-297). Tests are at lines 299-340. The module uses helper functions from super:: (register_gauge_vec, set_gauge_vec, register_histogram_vec, observe_histogram_vec). Labels use &["model", "fold"] for training metrics and &["model"] for hyperopt metrics.

Step 1: Add the verbose gate

At the top of the file, after the use super::{...} block (line 23), add:

use std::sync::OnceLock;

static VERBOSE: OnceLock<bool> = OnceLock::new();

/// Returns true if `FOXHUNT_VERBOSE_METRICS` env var is set.
/// Checked once at first call and cached.
pub fn verbose_enabled() -> bool {
    *VERBOSE.get_or_init(|| std::env::var("FOXHUNT_VERBOSE_METRICS").is_ok())
}

Step 2: Register Tier 1 gauges in init()

After the existing register_gauge("foxhunt_training_active_workers", ...) block (line 130) and before the // Hyperopt gauges comment (line 132), add these 9 training gauges:

    // RL diagnostic gauges (model + fold)
    _ = register_gauge_vec("foxhunt_training_q_value_mean", "Mean Q-value per epoch (DQN)", mf);
    _ = register_gauge_vec("foxhunt_training_q_value_max", "Max Q-value per epoch (DQN)", mf);
    _ = register_gauge_vec("foxhunt_training_policy_entropy", "Policy entropy (PPO)", mf);
    _ = register_gauge_vec("foxhunt_training_kl_divergence", "KL divergence old vs new policy (PPO)", mf);
    _ = register_gauge_vec("foxhunt_training_advantage_mean", "Mean advantage estimate (PPO)", mf);
    _ = register_gauge_vec("foxhunt_training_replay_buffer_size", "Replay buffer occupancy (DQN)", mf);

    // Gradient & training health gauges (model + fold)
    _ = register_gauge_vec("foxhunt_training_gradient_norm", "L2 gradient norm after backward", mf);
    _ = register_gauge_vec("foxhunt_training_learning_rate", "Effective learning rate", mf);
    _ = register_gauge_vec("foxhunt_training_epoch_duration_seconds", "Wall-clock time per epoch", mf);

After the existing register_gauge_vec("foxhunt_hyperopt_mode", ...) block (line 152) and before the // Hyperopt counters comment (line 154), add these 3 hyperopt gauges:

    // Hyperopt intra-trial gauges (model only)
    _ = register_gauge_vec("foxhunt_hyperopt_trial_epoch", "Current epoch within active trial", m);
    _ = register_gauge_vec("foxhunt_hyperopt_trial_best_loss", "Best loss in current trial", m);
    _ = register_gauge_vec("foxhunt_hyperopt_elapsed_seconds", "Total wall-clock time since hyperopt start", m);

Step 3: Register Tier 2 verbose metrics in init()

At the end of init(), before the closing }, add:

    // Tier 2: Verbose metrics (only registered when FOXHUNT_VERBOSE_METRICS is set)
    if verbose_enabled() {
        let mfl: &[&str] = &["model", "fold", "layer"];
        _ = register_histogram_vec(
            "foxhunt_training_batch_loss",
            "Per-batch loss distribution",
            mf,
            vec![0.001, 0.01, 0.05, 0.1, 0.5, 1.0, 5.0, 10.0],
        );
        _ = register_histogram_vec(
            "foxhunt_training_gradient_norm_per_layer",
            "Per-layer gradient L2 norm",
            mfl,
            vec![0.0001, 0.001, 0.01, 0.1, 1.0, 10.0, 100.0],
        );
        _ = register_gauge_vec("foxhunt_training_replay_priority_mean", "Mean PER priority", mf);
        _ = register_gauge_vec("foxhunt_training_advantage_std", "Advantage standard deviation (PPO)", mf);
        _ = register_gauge_vec("foxhunt_training_value_explained_variance", "Value function explained variance (PPO)", mf);
    }

Step 4: Add typed helpers for Tier 1

After the existing set_hyperopt_mode helper (line 297), add:

// ---------------------------------------------------------------------------
// Tier 1: RL diagnostics
// ---------------------------------------------------------------------------

pub fn set_q_value_stats(model: &str, fold: &str, mean: f64, max: f64) {
    set_gauge_vec("foxhunt_training_q_value_mean", &[model, fold], mean);
    set_gauge_vec("foxhunt_training_q_value_max", &[model, fold], max);
}

pub fn set_policy_entropy(model: &str, fold: &str, entropy: f64) {
    set_gauge_vec("foxhunt_training_policy_entropy", &[model, fold], entropy);
}

pub fn set_kl_divergence(model: &str, fold: &str, kl: f64) {
    set_gauge_vec("foxhunt_training_kl_divergence", &[model, fold], kl);
}

pub fn set_advantage_mean(model: &str, fold: &str, mean: f64) {
    set_gauge_vec("foxhunt_training_advantage_mean", &[model, fold], mean);
}

pub fn set_replay_buffer_size(model: &str, fold: &str, size: f64) {
    set_gauge_vec("foxhunt_training_replay_buffer_size", &[model, fold], size);
}

// ---------------------------------------------------------------------------
// Tier 1: Gradient & training health
// ---------------------------------------------------------------------------

pub fn set_gradient_norm(model: &str, fold: &str, norm: f64) {
    set_gauge_vec("foxhunt_training_gradient_norm", &[model, fold], norm);
}

pub fn set_learning_rate(model: &str, fold: &str, lr: f64) {
    set_gauge_vec("foxhunt_training_learning_rate", &[model, fold], lr);
}

pub fn set_epoch_duration(model: &str, fold: &str, secs: f64) {
    set_gauge_vec("foxhunt_training_epoch_duration_seconds", &[model, fold], secs);
}

// ---------------------------------------------------------------------------
// Tier 1: Hyperopt intra-trial
// ---------------------------------------------------------------------------

pub fn set_hyperopt_trial_epoch(model: &str, epoch: f64) {
    set_gauge_vec("foxhunt_hyperopt_trial_epoch", &[model], epoch);
}

pub fn set_hyperopt_trial_best_loss(model: &str, loss: f64) {
    set_gauge_vec("foxhunt_hyperopt_trial_best_loss", &[model], loss);
}

pub fn set_hyperopt_elapsed(model: &str, secs: f64) {
    set_gauge_vec("foxhunt_hyperopt_elapsed_seconds", &[model], secs);
}

// ---------------------------------------------------------------------------
// Tier 2: Verbose metrics (no-op when FOXHUNT_VERBOSE_METRICS not set)
// ---------------------------------------------------------------------------

pub fn observe_batch_loss(model: &str, fold: &str, loss: f64) {
    if verbose_enabled() {
        observe_histogram_vec("foxhunt_training_batch_loss", &[model, fold], loss);
    }
}

pub fn observe_gradient_norm_per_layer(model: &str, fold: &str, layer: &str, norm: f64) {
    if verbose_enabled() {
        observe_histogram_vec("foxhunt_training_gradient_norm_per_layer", &[model, fold, layer], norm);
    }
}

pub fn set_replay_priority_mean(model: &str, fold: &str, mean: f64) {
    if verbose_enabled() {
        set_gauge_vec("foxhunt_training_replay_priority_mean", &[model, fold], mean);
    }
}

pub fn set_advantage_std(model: &str, fold: &str, std: f64) {
    if verbose_enabled() {
        set_gauge_vec("foxhunt_training_advantage_std", &[model, fold], std);
    }
}

pub fn set_value_explained_variance(model: &str, fold: &str, ev: f64) {
    if verbose_enabled() {
        set_gauge_vec("foxhunt_training_value_explained_variance", &[model, fold], ev);
    }
}

Step 5: Add tests

Add these tests inside the existing #[cfg(test)] mod tests block, after the last test:

    #[test]
    fn test_verbose_disabled_by_default() {
        // FOXHUNT_VERBOSE_METRICS not set in test env
        // verbose helpers should not panic even when metrics aren't registered
        init();
        // These are no-ops when verbose is disabled
        observe_batch_loss("dqn", "0", 0.05);
        observe_gradient_norm_per_layer("dqn", "0", "fc1", 0.01);
        set_replay_priority_mean("dqn", "0", 1.5);
        set_advantage_std("ppo", "0", 0.8);
        set_value_explained_variance("ppo", "0", 0.7);
    }

    #[test]
    fn test_tier1_helpers_no_panic() {
        init();
        set_q_value_stats("dqn", "0", 12.3, 45.6);
        set_policy_entropy("ppo", "0", 1.23);
        set_kl_divergence("ppo", "0", 0.008);
        set_advantage_mean("ppo", "0", 0.001);
        set_replay_buffer_size("dqn", "0", 50000.0);
        set_gradient_norm("dqn", "0", 1.23);
        set_learning_rate("dqn", "0", 0.001);
        set_epoch_duration("dqn", "0", 12.5);
        set_hyperopt_trial_epoch("dqn", 5.0);
        set_hyperopt_trial_best_loss("dqn", 0.042);
        set_hyperopt_elapsed("dqn", 123.4);
    }

Step 6: Run tests

Run: SQLX_OFFLINE=true cargo test -p common --lib -- training_metrics Expected: All tests pass, including the 2 new ones.

Step 7: Clippy check

Run: SQLX_OFFLINE=true cargo clippy -p common -- -D warnings Expected: 0 errors, 0 warnings.

Step 8: Commit

git add crates/common/src/metrics/training_metrics.rs
git commit -m "feat(metrics): register 17 new Prometheus metrics with verbose gating"

---

Task 2: Extend monitoring.proto with new TrainingSession fields

Files:

• Modify: services/monitoring_service/proto/monitoring.proto

Context: TrainingSession currently has fields 1-23. The design doc assigns field numbers 24-35 for new fields. Note: the existing hyperopt fields already use numbers 20-23, so the new fields start at 24.

Step 1: Add new fields to TrainingSession

After the uint32 hyperopt_trials_failed = 23; line (line 65), add:

  // RL diagnostics
  float q_value_mean = 24;
  float q_value_max = 25;
  float policy_entropy = 26;
  float kl_divergence = 27;
  float advantage_mean = 28;
  uint32 replay_buffer_size = 29;

  // Gradient & training health
  float gradient_norm = 30;
  float learning_rate = 31;
  float epoch_duration_seconds = 32;

  // Hyperopt intra-trial
  uint32 hyperopt_trial_epoch = 33;
  float hyperopt_trial_best_loss = 34;
  float hyperopt_elapsed_seconds = 35;

Step 2: Verify proto compiles

Run: SQLX_OFFLINE=true cargo check -p monitoring_service Expected: Compiles (prost generates the new fields as f32/u32 with default 0).

Run: SQLX_OFFLINE=true cargo check -p fxt Expected: Compiles (fxt also compiles this proto).

Step 3: Commit

git add services/monitoring_service/proto/monitoring.proto
git commit -m "proto(monitoring): add 12 new TrainingSession fields for diagnostics"

---

Task 3: Wire new metrics through monitoring_service group_into_sessions

Files:

• Modify: services/monitoring_service/src/service.rs

Context: The group_into_sessions function (line 98) has a match s.name.as_str() block (lines 117-158) that maps Prometheus metric names to TrainingSession proto fields. We add 12 new match arms.

Step 1: Add match arms

Inside the match s.name.as_str() block, after the "foxhunt_hyperopt_mode" arm (line 153-157) and before the _ => {} wildcard (line 158), add:

            // RL diagnostics
            "foxhunt_training_q_value_mean" => session.q_value_mean = s.value as f32,
            "foxhunt_training_q_value_max" => session.q_value_max = s.value as f32,
            "foxhunt_training_policy_entropy" => session.policy_entropy = s.value as f32,
            "foxhunt_training_kl_divergence" => session.kl_divergence = s.value as f32,
            "foxhunt_training_advantage_mean" => session.advantage_mean = s.value as f32,
            "foxhunt_training_replay_buffer_size" => {
                session.replay_buffer_size = s.value as u32;
            }
            // Gradient & training health
            "foxhunt_training_gradient_norm" => session.gradient_norm = s.value as f32,
            "foxhunt_training_learning_rate" => session.learning_rate = s.value as f32,
            "foxhunt_training_epoch_duration_seconds" => {
                session.epoch_duration_seconds = s.value as f32;
            }
            // Hyperopt intra-trial
            "foxhunt_hyperopt_trial_epoch" => session.hyperopt_trial_epoch = s.value as u32,
            "foxhunt_hyperopt_trial_best_loss" => {
                session.hyperopt_trial_best_loss = s.value as f32;
            }
            "foxhunt_hyperopt_elapsed_seconds" => {
                session.hyperopt_elapsed_seconds = s.value as f32;
            }

Step 2: Add tests for new metric mapping

Add a new test in the #[cfg(test)] mod tests block:

    #[test]
    fn test_group_new_diagnostic_metrics() {
        let samples = vec![
            MetricSample {
                name: "foxhunt_training_current_epoch".to_owned(),
                model: "dqn".to_owned(),
                fold: "0".to_owned(),
                value: 25.0,
            },
            MetricSample {
                name: "foxhunt_training_q_value_mean".to_owned(),
                model: "dqn".to_owned(),
                fold: "0".to_owned(),
                value: 12.3,
            },
            MetricSample {
                name: "foxhunt_training_q_value_max".to_owned(),
                model: "dqn".to_owned(),
                fold: "0".to_owned(),
                value: 45.6,
            },
            MetricSample {
                name: "foxhunt_training_gradient_norm".to_owned(),
                model: "dqn".to_owned(),
                fold: "0".to_owned(),
                value: 1.23,
            },
            MetricSample {
                name: "foxhunt_training_learning_rate".to_owned(),
                model: "dqn".to_owned(),
                fold: "0".to_owned(),
                value: 0.001,
            },
            MetricSample {
                name: "foxhunt_training_epoch_duration_seconds".to_owned(),
                model: "dqn".to_owned(),
                fold: "0".to_owned(),
                value: 12.5,
            },
            MetricSample {
                name: "foxhunt_training_replay_buffer_size".to_owned(),
                model: "dqn".to_owned(),
                fold: "0".to_owned(),
                value: 50000.0,
            },
        ];
        let sessions = group_into_sessions(&samples, "");
        assert_eq!(sessions.len(), 1);
        let s = &sessions[0];
        assert_eq!(s.current_epoch, 25.0);
        assert!((s.q_value_mean - 12.3).abs() < 0.01);
        assert!((s.q_value_max - 45.6).abs() < 0.01);
        assert!((s.gradient_norm - 1.23).abs() < 0.01);
        assert!((s.learning_rate - 0.001).abs() < 0.0001);
        assert!((s.epoch_duration_seconds - 12.5).abs() < 0.01);
        assert_eq!(s.replay_buffer_size, 50000);
    }

    #[test]
    fn test_group_ppo_diagnostics() {
        let samples = vec![
            MetricSample {
                name: "foxhunt_training_policy_entropy".to_owned(),
                model: "ppo".to_owned(),
                fold: "0".to_owned(),
                value: 1.23,
            },
            MetricSample {
                name: "foxhunt_training_kl_divergence".to_owned(),
                model: "ppo".to_owned(),
                fold: "0".to_owned(),
                value: 0.008,
            },
            MetricSample {
                name: "foxhunt_training_advantage_mean".to_owned(),
                model: "ppo".to_owned(),
                fold: "0".to_owned(),
                value: 0.001,
            },
        ];
        let sessions = group_into_sessions(&samples, "");
        assert_eq!(sessions.len(), 1);
        let s = &sessions[0];
        assert!((s.policy_entropy - 1.23).abs() < 0.01);
        assert!((s.kl_divergence - 0.008).abs() < 0.001);
        assert!((s.advantage_mean - 0.001).abs() < 0.001);
    }

    #[test]
    fn test_group_hyperopt_intra_trial() {
        let samples = vec![
            MetricSample {
                name: "foxhunt_hyperopt_mode".to_owned(),
                model: "dqn".to_owned(),
                fold: "".to_owned(),
                value: 1.0,
            },
            MetricSample {
                name: "foxhunt_hyperopt_trial_epoch".to_owned(),
                model: "dqn".to_owned(),
                fold: "".to_owned(),
                value: 12.0,
            },
            MetricSample {
                name: "foxhunt_hyperopt_trial_best_loss".to_owned(),
                model: "dqn".to_owned(),
                fold: "".to_owned(),
                value: 0.042,
            },
            MetricSample {
                name: "foxhunt_hyperopt_elapsed_seconds".to_owned(),
                model: "dqn".to_owned(),
                fold: "".to_owned(),
                value: 123.4,
            },
        ];
        let sessions = group_into_sessions(&samples, "");
        assert_eq!(sessions.len(), 1);
        let s = &sessions[0];
        assert!(s.is_hyperopt);
        assert_eq!(s.hyperopt_trial_epoch, 12);
        assert!((s.hyperopt_trial_best_loss - 0.042).abs() < 0.001);
        assert!((s.hyperopt_elapsed_seconds - 123.4).abs() < 0.1);
    }

Step 3: Run tests

Run: SQLX_OFFLINE=true cargo test -p monitoring_service --lib Expected: All 7 tests pass (4 existing + 3 new).

Step 4: Commit

git add services/monitoring_service/src/service.rs
git commit -m "feat(monitoring): wire 12 new metric→proto field mappings in group_into_sessions"

---

Task 4: Enhance fxt monitor CLI display

Files:

• Modify: bin/fxt/src/commands/train/monitor.rs

Context: The CLI has print_session_table (line 127) for the main table, print_hyperopt_summary (line 154) for hyperopt, and print_health_summary (line 172) for health counters. We add gradient_norm to the main table, an RL diagnostics section, and inline hyperopt trial progress.

Step 1: Add gradient norm column to main table

Replace the print_session_table function (lines 127-152) with:

fn print_session_table(sessions: &[TrainingSession]) {
    println!(
        "{:<10} {:<6} {:<7} {:<10} {:<10} {:<9} {:<10} {:<10}",
        "Model".bright_cyan(),
        "Fold".bright_cyan(),
        "Epoch".bright_cyan(),
        "Loss".bright_cyan(),
        "Val Loss".bright_cyan(),
        "Batch/s".bright_cyan(),
        "Grad Norm".bright_cyan(),
        "Eval Acc".bright_cyan(),
    );
    println!("{}", "-".repeat(82).bright_black());

    for s in sessions {
        let acc = if s.eval_accuracy > 0.0 {
            format!("{:.1}%", s.eval_accuracy * 100.0)
        } else {
            "-".to_owned()
        };
        let grad = if s.gradient_norm > 0.0 {
            format!("{:.4}", s.gradient_norm)
        } else {
            "-".to_owned()
        };
        println!(
            "{:<10} {:<6} {:<7.0} {:<10.4} {:<10.4} {:<9.1} {:<10} {:<10}",
            s.model, s.fold, s.current_epoch, s.epoch_loss, s.validation_loss,
            s.batches_per_second, grad, acc,
        );
    }
}

Step 2: Add RL diagnostics section

After print_session_table, add a new function:

fn print_rl_diagnostics(sessions: &[TrainingSession]) {
    let rl_sessions: Vec<_> = sessions
        .iter()
        .filter(|s| s.q_value_mean != 0.0 || s.policy_entropy != 0.0)
        .collect();
    if rl_sessions.is_empty() {
        return;
    }
    println!();
    println!("{}", "RL Diagnostics:".bright_cyan());
    for s in &rl_sessions {
        if s.q_value_mean != 0.0 || s.q_value_max != 0.0 {
            // DQN-style
            println!(
                "  {}: Q-mean={:.2} Q-max={:.2} | buffer={}",
                s.model.bright_white(),
                s.q_value_mean,
                s.q_value_max,
                s.replay_buffer_size,
            );
        }
        if s.policy_entropy != 0.0 || s.kl_divergence != 0.0 {
            // PPO-style
            println!(
                "  {}: entropy={:.3} KL={:.4} adv-mean={:.4}",
                s.model.bright_white(),
                s.policy_entropy,
                s.kl_divergence,
                s.advantage_mean,
            );
        }
    }
}

Step 3: Enhance hyperopt summary with intra-trial progress

Replace the print_hyperopt_summary function (lines 154-169) with:

fn print_hyperopt_summary(sessions: &[TrainingSession]) {
    let hyperopt: Vec<_> = sessions.iter().filter(|s| s.is_hyperopt).collect();
    if hyperopt.is_empty() {
        return;
    }
    println!();
    for s in &hyperopt {
        let trial_detail = if s.hyperopt_trial_epoch > 0 {
            format!(
                " (epoch {}, loss {:.4})",
                s.hyperopt_trial_epoch, s.hyperopt_trial_best_loss
            )
        } else {
            String::new()
        };
        let elapsed = if s.hyperopt_elapsed_seconds > 0.0 {
            format!(" | {:.0}s elapsed", s.hyperopt_elapsed_seconds)
        } else {
            String::new()
        };
        println!(
            "Hyperopt ({}): trial {}/{}{} | best Sharpe {:.2} | {} failures{}",
            s.model.bright_magenta(),
            s.hyperopt_trial_current,
            s.hyperopt_trial_total,
            trial_detail,
            s.hyperopt_best_objective,
            s.hyperopt_trials_failed,
            elapsed,
        );
    }
}

Step 4: Wire RL diagnostics into render_snapshot

In render_snapshot (line 81), add a call to print_rl_diagnostics between print_session_table and print_hyperopt_summary:

Change lines 100-102 from:

    print_session_table(&resp.sessions);
    print_hyperopt_summary(&resp.sessions);
    print_health_summary(&resp.sessions);

to:

    print_session_table(&resp.sessions);
    print_rl_diagnostics(&resp.sessions);
    print_hyperopt_summary(&resp.sessions);
    print_health_summary(&resp.sessions);

Step 5: Verify compilation

Run: SQLX_OFFLINE=true cargo check -p fxt Expected: Compiles cleanly.

Step 6: Commit

git add bin/fxt/src/commands/train/monitor.rs
git commit -m "feat(fxt): add grad norm, RL diagnostics, and hyperopt trial detail to monitor"

---

Task 5: Record new metrics in DQN trainer

Files:

• Modify: crates/ml/src/trainers/dqn/trainer.rs

Context: The DQN trainer already has use common::metrics::training_metrics; and records 4 metrics at line 2653. The epoch loop computes avg_q_value and avg_grad_norm at line 2473. Q-value stats (min, max, mean) are available at line 2580 via monitor.get_q_value_stats(). Buffer size is available via agent.get_replay_buffer_size() (line 993).

Step 1: Add Tier 1 metrics after existing per-epoch recording

After the existing training_metrics::set_batches_per_second(...) block (~line 2661), add:

            // Tier 1: RL diagnostics
            let (q_min, q_max, q_mean) = monitor.get_q_value_stats();
            training_metrics::set_q_value_stats("dqn", "current", q_mean as f64, q_max as f64);
            training_metrics::set_gradient_norm("dqn", "current", avg_grad_norm);
            training_metrics::set_epoch_duration("dqn", "current", epoch_duration.as_secs_f64());
            {
                let agent = self.agent.read().await;
                if let Ok(buf_size) = agent.get_replay_buffer_size() {
                    training_metrics::set_replay_buffer_size("dqn", "current", buf_size as f64);
                }
            }

Note: q_min, q_max, q_mean are already used at line 2580. You need to move/duplicate the call here. The monitor.get_q_value_stats() call at line 2580 already exists — use the variables from there or call it again. The simplest approach is to add the metrics recording right after the existing let (q_min, q_max, q_mean) = monitor.get_q_value_stats(); at line 2580.

Actually, the cleanest approach: add metrics recording right after line 2580 where (q_min, q_max, q_mean) is already bound:

            // After: let (q_min, q_max, q_mean) = monitor.get_q_value_stats();
            training_metrics::set_q_value_stats("dqn", "current", q_mean as f64, q_max as f64);
            training_metrics::set_gradient_norm("dqn", "current", avg_grad_norm);
            training_metrics::set_epoch_duration("dqn", "current", epoch_duration.as_secs_f64());
            {
                let agent = self.agent.read().await;
                if let Ok(buf_size) = agent.get_replay_buffer_size() {
                    training_metrics::set_replay_buffer_size("dqn", "current", buf_size as f64);
                }
            }

Note: _ = q_min; may be needed to suppress unused warning if q_min was only used in the info log. Check — if q_min is already used in the info!() macro at line 2582, no suppression needed.

Step 2: Verify compilation

Run: SQLX_OFFLINE=true cargo check -p ml Expected: Compiles. Check for unused variable warnings.

Step 3: Commit

git add crates/ml/src/trainers/dqn/trainer.rs
git commit -m "feat(dqn): record Q-value, gradient norm, epoch duration, buffer size metrics"

---

Task 6: Record new metrics in PPO trainer

Files:

• Modify: crates/ml/src/trainers/ppo.rs

Context: PPO trainer already records 3 metrics at line 668. The epoch loop computes kl_divergence, explained_variance, mean_reward, std_reward, entropy at line 649 via self.compute_metrics(...). The PpoEpochMetrics struct (line 196) has all these fields.

Step 1: Add metrics after existing recording

After the existing training_metrics::set_validation_loss(...) at line 670, add:

            // Tier 1: PPO diagnostics
            training_metrics::set_policy_entropy("ppo", "current", entropy as f64);
            training_metrics::set_kl_divergence("ppo", "current", kl_divergence as f64);
            training_metrics::set_advantage_mean("ppo", "current", mean_reward as f64);

            // Tier 2: Verbose PPO metrics (no-op when disabled)
            training_metrics::set_advantage_std("ppo", "current", std_reward as f64);
            training_metrics::set_value_explained_variance("ppo", "current", explained_variance as f64);

Note: The variables entropy, kl_divergence, mean_reward, std_reward, explained_variance are computed at line 649 and available in scope.

Step 2: Add gradient norm and epoch duration

The PPO trainer doesn't currently track gradient norms per epoch. We need to find where the optimizer step happens. Search for the backward/optimizer step in the PPO update logic. If a gradient norm is available after the optimizer step, record it. If not, skip this for PPO (the gradient_norm metric will only be populated for DQN and supervised models).

Check if the PPO update at self.ppo_update_step(...) or similar returns a gradient norm. If not, we only add epoch_duration:

After the metrics block, add:

            // Epoch duration (epoch_start must be captured at top of loop)
            // Note: PPO epochs are measured by the progress_callback timing

Actually — looking at the PPO trainer structure, the epoch loop at line 492 iterates over 0..self.hyperparams.epochs. We need to add timing. At the top of the epoch loop body (after for epoch in 0..self.hyperparams.epochs {), add:

            let epoch_start = std::time::Instant::now();

Then after the metrics block:

            training_metrics::set_epoch_duration("ppo", "current", epoch_start.elapsed().as_secs_f64());

Step 3: Verify compilation

Run: SQLX_OFFLINE=true cargo check -p ml Expected: Compiles.

Step 4: Commit

git add crates/ml/src/trainers/ppo.rs
git commit -m "feat(ppo): record entropy, KL, advantage, explained variance, epoch duration metrics"

---

Task 7: Record new metrics in supervised training binary

Files:

• Modify: crates/ml/examples/train_baseline_supervised.rs

Context: The supervised binary already records epoch/loss/val_loss/iteration_seconds at lines 564-567. The adapter.get_learning_rate() value is printed at line 576 but not recorded as a metric. Epoch timing is already measured at line 554 (epoch_start).

Step 1: Add learning rate and epoch duration metrics

After the existing metrics block (line 567), add:

        metrics::set_learning_rate(model_name, &fold_str, adapter.get_learning_rate());
        metrics::set_epoch_duration(model_name, &fold_str, elapsed.as_secs_f64());

Note: metrics is the alias for training_metrics (imported as use common::metrics::training_metrics as metrics; at line 46). elapsed is computed at line 561. adapter.get_learning_rate() returns f64.

Step 2: Verify compilation

Run: SQLX_OFFLINE=true cargo check --example train_baseline_supervised Expected: Compiles. (Note: this example may need the full cargo check -p ml due to workspace deps.)

Actually, the correct command is: Run: SQLX_OFFLINE=true cargo check -p ml --example train_baseline_supervised Expected: Compiles.

Step 3: Commit

git add crates/ml/examples/train_baseline_supervised.rs
git commit -m "feat(supervised): record learning rate and epoch duration metrics"

---

Task 8: Record hyperopt intra-trial metrics in both hyperopt binaries

Files:

• Modify: crates/ml/examples/hyperopt_baseline_rl.rs
• Modify: crates/ml/examples/hyperopt_baseline_supervised.rs

Context: Both binaries already have use common::metrics::training_metrics; and record hyperopt trial/best_objective. They each have a let start = Instant::now(); at the beginning of each model's optimization function. The elapsed is computed after optimization completes. We need to add elapsed_seconds recording after optimization completes, before returning results.

Step 1: Add elapsed_seconds to hyperopt_baseline_rl.rs

In run_dqn_hyperopt(), after training_metrics::set_hyperopt_best_objective("dqn", result.best_objective); (~line 185), add:

    training_metrics::set_hyperopt_elapsed("dqn", elapsed);

In run_ppo_hyperopt(), after training_metrics::set_hyperopt_best_objective("ppo", result.best_objective); (~line 254), add:

    training_metrics::set_hyperopt_elapsed("ppo", elapsed);

Step 2: Add elapsed_seconds to hyperopt_baseline_supervised.rs

The supervised hyperopt binary has 8 model functions (TFT, Mamba2, Liquid, TGGN, TLOB, KAN, xLSTM, Diffusion). In each function, after the training_metrics::set_hyperopt_best_objective(MODEL, result.best_objective); line, add:

    training_metrics::set_hyperopt_elapsed(MODEL, elapsed);

Where MODEL is the lowercase model name string ("tft", "mamba2", "liquid", "tggn", "tlob", "kan", "xlstm", "diffusion").

Specific locations:

• run_tft_hyperopt(): after line 150
• run_mamba2_hyperopt(): after line 208
• run_liquid_hyperopt(): after line 267
• run_tggn_hyperopt(): after line 326
• run_tlob_hyperopt(): after line 385
• run_kan_hyperopt(): after line 444
• run_xlstm_hyperopt(): after line 503
• run_diffusion_hyperopt(): after line 562

Step 3: Verify compilation

Run: SQLX_OFFLINE=true cargo check -p ml --example hyperopt_baseline_rl Run: SQLX_OFFLINE=true cargo check -p ml --example hyperopt_baseline_supervised Expected: Both compile.

Step 4: Commit

git add crates/ml/examples/hyperopt_baseline_rl.rs crates/ml/examples/hyperopt_baseline_supervised.rs
git commit -m "feat(hyperopt): record elapsed_seconds metric in all hyperopt binaries"

---

Task 9: Full workspace check + clippy

Files: None (verification only)

Step 1: Full workspace check

Run: SQLX_OFFLINE=true cargo check --workspace Expected: 0 errors.

Step 2: Clippy

Run: SQLX_OFFLINE=true cargo clippy --workspace -- -D warnings Expected: 0 errors, 0 warnings.

Step 3: Run all affected tests

Run: SQLX_OFFLINE=true cargo test -p common --lib -- training_metrics Run: SQLX_OFFLINE=true cargo test -p monitoring_service --lib Expected: All tests pass.

Step 4: Commit (if any fixes needed)

git commit -am "fix: resolve clippy/test issues from metrics additions"

Only create this commit if there were issues to fix. If everything passes, skip this step.