# Additional Training Metrics Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Add 17 new Prometheus metrics (12 always-on gauges + 5 opt-in verbose) to provide full RL diagnostics, gradient health, and hyperopt intra-trial visibility across the monitoring pipeline.

**Architecture:** Two-tier metrics system. Tier 1 (12 gauges) — zero-cost atomic writes in every trainer's epoch loop. Tier 2 (5 metrics) — gated behind `FOXHUNT_VERBOSE_METRICS=1`. All metrics flow through the existing Prometheus → monitoring_service → gRPC → fxt CLI pipeline. No new queries needed (wildcard `foxhunt_training_*|foxhunt_hyperopt_*` already captures new metrics).

**Tech Stack:** Rust, prometheus crate, tonic/prost (gRPC), protobuf, Candle ML framework

**Build command:** `SQLX_OFFLINE=true cargo check --workspace`
**Test command:** `SQLX_OFFLINE=true cargo test -p <crate> --lib`
**Clippy rules:** `#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::indexing_slicing)]` — use `.get()`, `?`, `.ok_or()` instead.

---

### Task 1: Register Tier 1 metrics + verbose gate in training_metrics.rs

**Files:**
- Modify: `crates/common/src/metrics/training_metrics.rs`

This task adds 12 new gauge registrations, 5 verbose metric registrations with env-var gating, and typed helper functions for all 17 new metrics.

**Context:** The file currently has 24 metrics registered in `init()` (lines 32-168) and typed helpers (lines 174-297). Tests are at lines 299-340. The module uses helper functions from `super::` (`register_gauge_vec`, `set_gauge_vec`, `register_histogram_vec`, `observe_histogram_vec`). Labels use `&["model", "fold"]` for training metrics and `&["model"]` for hyperopt metrics.

**Step 1: Add the verbose gate**

At the top of the file, after the `use super::{...}` block (line 23), add:

```rust
use std::sync::OnceLock;

static VERBOSE: OnceLock<bool> = OnceLock::new();

/// Returns true if `FOXHUNT_VERBOSE_METRICS` env var is set.
/// Checked once at first call and cached.
pub fn verbose_enabled() -> bool {
    *VERBOSE.get_or_init(|| std::env::var("FOXHUNT_VERBOSE_METRICS").is_ok())
}
```

**Step 2: Register Tier 1 gauges in `init()`**

After the existing `register_gauge("foxhunt_training_active_workers", ...)` block (line 130) and before the `// Hyperopt gauges` comment (line 132), add these 9 training gauges:

```rust
    // RL diagnostic gauges (model + fold)
    _ = register_gauge_vec("foxhunt_training_q_value_mean", "Mean Q-value per epoch (DQN)", mf);
    _ = register_gauge_vec("foxhunt_training_q_value_max", "Max Q-value per epoch (DQN)", mf);
    _ = register_gauge_vec("foxhunt_training_policy_entropy", "Policy entropy (PPO)", mf);
    _ = register_gauge_vec("foxhunt_training_kl_divergence", "KL divergence old vs new policy (PPO)", mf);
    _ = register_gauge_vec("foxhunt_training_advantage_mean", "Mean advantage estimate (PPO)", mf);
    _ = register_gauge_vec("foxhunt_training_replay_buffer_size", "Replay buffer occupancy (DQN)", mf);

    // Gradient & training health gauges (model + fold)
    _ = register_gauge_vec("foxhunt_training_gradient_norm", "L2 gradient norm after backward", mf);
    _ = register_gauge_vec("foxhunt_training_learning_rate", "Effective learning rate", mf);
    _ = register_gauge_vec("foxhunt_training_epoch_duration_seconds", "Wall-clock time per epoch", mf);
```

After the existing `register_gauge_vec("foxhunt_hyperopt_mode", ...)` block (line 152) and before the `// Hyperopt counters` comment (line 154), add these 3 hyperopt gauges:

```rust
    // Hyperopt intra-trial gauges (model only)
    _ = register_gauge_vec("foxhunt_hyperopt_trial_epoch", "Current epoch within active trial", m);
    _ = register_gauge_vec("foxhunt_hyperopt_trial_best_loss", "Best loss in current trial", m);
    _ = register_gauge_vec("foxhunt_hyperopt_elapsed_seconds", "Total wall-clock time since hyperopt start", m);
```

**Step 3: Register Tier 2 verbose metrics in `init()`**

At the end of `init()`, before the closing `}`, add:

```rust
    // Tier 2: Verbose metrics (only registered when FOXHUNT_VERBOSE_METRICS is set)
    if verbose_enabled() {
        let mfl: &[&str] = &["model", "fold", "layer"];
        _ = register_histogram_vec(
            "foxhunt_training_batch_loss",
            "Per-batch loss distribution",
            mf,
            vec![0.001, 0.01, 0.05, 0.1, 0.5, 1.0, 5.0, 10.0],
        );
        _ = register_histogram_vec(
            "foxhunt_training_gradient_norm_per_layer",
            "Per-layer gradient L2 norm",
            mfl,
            vec![0.0001, 0.001, 0.01, 0.1, 1.0, 10.0, 100.0],
        );
        _ = register_gauge_vec("foxhunt_training_replay_priority_mean", "Mean PER priority", mf);
        _ = register_gauge_vec("foxhunt_training_advantage_std", "Advantage standard deviation (PPO)", mf);
        _ = register_gauge_vec("foxhunt_training_value_explained_variance", "Value function explained variance (PPO)", mf);
    }
```

**Step 4: Add typed helpers for Tier 1**

After the existing `set_hyperopt_mode` helper (line 297), add:

```rust
// ---------------------------------------------------------------------------
// Tier 1: RL diagnostics
// ---------------------------------------------------------------------------

pub fn set_q_value_stats(model: &str, fold: &str, mean: f64, max: f64) {
    set_gauge_vec("foxhunt_training_q_value_mean", &[model, fold], mean);
    set_gauge_vec("foxhunt_training_q_value_max", &[model, fold], max);
}

pub fn set_policy_entropy(model: &str, fold: &str, entropy: f64) {
    set_gauge_vec("foxhunt_training_policy_entropy", &[model, fold], entropy);
}

pub fn set_kl_divergence(model: &str, fold: &str, kl: f64) {
    set_gauge_vec("foxhunt_training_kl_divergence", &[model, fold], kl);
}

pub fn set_advantage_mean(model: &str, fold: &str, mean: f64) {
    set_gauge_vec("foxhunt_training_advantage_mean", &[model, fold], mean);
}

pub fn set_replay_buffer_size(model: &str, fold: &str, size: f64) {
    set_gauge_vec("foxhunt_training_replay_buffer_size", &[model, fold], size);
}

// ---------------------------------------------------------------------------
// Tier 1: Gradient & training health
// ---------------------------------------------------------------------------

pub fn set_gradient_norm(model: &str, fold: &str, norm: f64) {
    set_gauge_vec("foxhunt_training_gradient_norm", &[model, fold], norm);
}

pub fn set_learning_rate(model: &str, fold: &str, lr: f64) {
    set_gauge_vec("foxhunt_training_learning_rate", &[model, fold], lr);
}

pub fn set_epoch_duration(model: &str, fold: &str, secs: f64) {
    set_gauge_vec("foxhunt_training_epoch_duration_seconds", &[model, fold], secs);
}

// ---------------------------------------------------------------------------
// Tier 1: Hyperopt intra-trial
// ---------------------------------------------------------------------------

pub fn set_hyperopt_trial_epoch(model: &str, epoch: f64) {
    set_gauge_vec("foxhunt_hyperopt_trial_epoch", &[model], epoch);
}

pub fn set_hyperopt_trial_best_loss(model: &str, loss: f64) {
    set_gauge_vec("foxhunt_hyperopt_trial_best_loss", &[model], loss);
}

pub fn set_hyperopt_elapsed(model: &str, secs: f64) {
    set_gauge_vec("foxhunt_hyperopt_elapsed_seconds", &[model], secs);
}

// ---------------------------------------------------------------------------
// Tier 2: Verbose metrics (no-op when FOXHUNT_VERBOSE_METRICS not set)
// ---------------------------------------------------------------------------

pub fn observe_batch_loss(model: &str, fold: &str, loss: f64) {
    if verbose_enabled() {
        observe_histogram_vec("foxhunt_training_batch_loss", &[model, fold], loss);
    }
}

pub fn observe_gradient_norm_per_layer(model: &str, fold: &str, layer: &str, norm: f64) {
    if verbose_enabled() {
        observe_histogram_vec("foxhunt_training_gradient_norm_per_layer", &[model, fold, layer], norm);
    }
}

pub fn set_replay_priority_mean(model: &str, fold: &str, mean: f64) {
    if verbose_enabled() {
        set_gauge_vec("foxhunt_training_replay_priority_mean", &[model, fold], mean);
    }
}

pub fn set_advantage_std(model: &str, fold: &str, std: f64) {
    if verbose_enabled() {
        set_gauge_vec("foxhunt_training_advantage_std", &[model, fold], std);
    }
}

pub fn set_value_explained_variance(model: &str, fold: &str, ev: f64) {
    if verbose_enabled() {
        set_gauge_vec("foxhunt_training_value_explained_variance", &[model, fold], ev);
    }
}
```

**Step 5: Add tests**

Add these tests inside the existing `#[cfg(test)] mod tests` block, after the last test:

```rust
    #[test]
    fn test_verbose_disabled_by_default() {
        // FOXHUNT_VERBOSE_METRICS not set in test env
        // verbose helpers should not panic even when metrics aren't registered
        init();
        // These are no-ops when verbose is disabled
        observe_batch_loss("dqn", "0", 0.05);
        observe_gradient_norm_per_layer("dqn", "0", "fc1", 0.01);
        set_replay_priority_mean("dqn", "0", 1.5);
        set_advantage_std("ppo", "0", 0.8);
        set_value_explained_variance("ppo", "0", 0.7);
    }

    #[test]
    fn test_tier1_helpers_no_panic() {
        init();
        set_q_value_stats("dqn", "0", 12.3, 45.6);
        set_policy_entropy("ppo", "0", 1.23);
        set_kl_divergence("ppo", "0", 0.008);
        set_advantage_mean("ppo", "0", 0.001);
        set_replay_buffer_size("dqn", "0", 50000.0);
        set_gradient_norm("dqn", "0", 1.23);
        set_learning_rate("dqn", "0", 0.001);
        set_epoch_duration("dqn", "0", 12.5);
        set_hyperopt_trial_epoch("dqn", 5.0);
        set_hyperopt_trial_best_loss("dqn", 0.042);
        set_hyperopt_elapsed("dqn", 123.4);
    }
```

**Step 6: Run tests**

Run: `SQLX_OFFLINE=true cargo test -p common --lib -- training_metrics`
Expected: All tests pass, including the 2 new ones.

**Step 7: Clippy check**

Run: `SQLX_OFFLINE=true cargo clippy -p common -- -D warnings`
Expected: 0 errors, 0 warnings.

**Step 8: Commit**

```bash
git add crates/common/src/metrics/training_metrics.rs
git commit -m "feat(metrics): register 17 new Prometheus metrics with verbose gating"
```

---

### Task 2: Extend monitoring.proto with new TrainingSession fields

**Files:**
- Modify: `services/monitoring_service/proto/monitoring.proto`

**Context:** `TrainingSession` currently has fields 1-23. The design doc assigns field numbers 24-35 for new fields. Note: the existing hyperopt fields already use numbers 20-23, so the new fields start at 24.

**Step 1: Add new fields to TrainingSession**

After the `uint32 hyperopt_trials_failed = 23;` line (line 65), add:

```protobuf

  // RL diagnostics
  float q_value_mean = 24;
  float q_value_max = 25;
  float policy_entropy = 26;
  float kl_divergence = 27;
  float advantage_mean = 28;
  uint32 replay_buffer_size = 29;

  // Gradient & training health
  float gradient_norm = 30;
  float learning_rate = 31;
  float epoch_duration_seconds = 32;

  // Hyperopt intra-trial
  uint32 hyperopt_trial_epoch = 33;
  float hyperopt_trial_best_loss = 34;
  float hyperopt_elapsed_seconds = 35;
```

**Step 2: Verify proto compiles**

Run: `SQLX_OFFLINE=true cargo check -p monitoring_service`
Expected: Compiles (prost generates the new fields as `f32`/`u32` with default 0).

Run: `SQLX_OFFLINE=true cargo check -p fxt`
Expected: Compiles (fxt also compiles this proto).

**Step 3: Commit**

```bash
git add services/monitoring_service/proto/monitoring.proto
git commit -m "proto(monitoring): add 12 new TrainingSession fields for diagnostics"
```

---

### Task 3: Wire new metrics through monitoring_service group_into_sessions

**Files:**
- Modify: `services/monitoring_service/src/service.rs`

**Context:** The `group_into_sessions` function (line 98) has a `match s.name.as_str()` block (lines 117-158) that maps Prometheus metric names to TrainingSession proto fields. We add 12 new match arms.

**Step 1: Add match arms**

Inside the `match s.name.as_str()` block, after the `"foxhunt_hyperopt_mode"` arm (line 153-157) and before the `_ => {}` wildcard (line 158), add:

```rust
            // RL diagnostics
            "foxhunt_training_q_value_mean" => session.q_value_mean = s.value as f32,
            "foxhunt_training_q_value_max" => session.q_value_max = s.value as f32,
            "foxhunt_training_policy_entropy" => session.policy_entropy = s.value as f32,
            "foxhunt_training_kl_divergence" => session.kl_divergence = s.value as f32,
            "foxhunt_training_advantage_mean" => session.advantage_mean = s.value as f32,
            "foxhunt_training_replay_buffer_size" => {
                session.replay_buffer_size = s.value as u32;
            }
            // Gradient & training health
            "foxhunt_training_gradient_norm" => session.gradient_norm = s.value as f32,
            "foxhunt_training_learning_rate" => session.learning_rate = s.value as f32,
            "foxhunt_training_epoch_duration_seconds" => {
                session.epoch_duration_seconds = s.value as f32;
            }
            // Hyperopt intra-trial
            "foxhunt_hyperopt_trial_epoch" => session.hyperopt_trial_epoch = s.value as u32,
            "foxhunt_hyperopt_trial_best_loss" => {
                session.hyperopt_trial_best_loss = s.value as f32;
            }
            "foxhunt_hyperopt_elapsed_seconds" => {
                session.hyperopt_elapsed_seconds = s.value as f32;
            }
```

**Step 2: Add tests for new metric mapping**

Add a new test in the `#[cfg(test)] mod tests` block:

```rust
    #[test]
    fn test_group_new_diagnostic_metrics() {
        let samples = vec![
            MetricSample {
                name: "foxhunt_training_current_epoch".to_owned(),
                model: "dqn".to_owned(),
                fold: "0".to_owned(),
                value: 25.0,
            },
            MetricSample {
                name: "foxhunt_training_q_value_mean".to_owned(),
                model: "dqn".to_owned(),
                fold: "0".to_owned(),
                value: 12.3,
            },
            MetricSample {
                name: "foxhunt_training_q_value_max".to_owned(),
                model: "dqn".to_owned(),
                fold: "0".to_owned(),
                value: 45.6,
            },
            MetricSample {
                name: "foxhunt_training_gradient_norm".to_owned(),
                model: "dqn".to_owned(),
                fold: "0".to_owned(),
                value: 1.23,
            },
            MetricSample {
                name: "foxhunt_training_learning_rate".to_owned(),
                model: "dqn".to_owned(),
                fold: "0".to_owned(),
                value: 0.001,
            },
            MetricSample {
                name: "foxhunt_training_epoch_duration_seconds".to_owned(),
                model: "dqn".to_owned(),
                fold: "0".to_owned(),
                value: 12.5,
            },
            MetricSample {
                name: "foxhunt_training_replay_buffer_size".to_owned(),
                model: "dqn".to_owned(),
                fold: "0".to_owned(),
                value: 50000.0,
            },
        ];
        let sessions = group_into_sessions(&samples, "");
        assert_eq!(sessions.len(), 1);
        let s = &sessions[0];
        assert_eq!(s.current_epoch, 25.0);
        assert!((s.q_value_mean - 12.3).abs() < 0.01);
        assert!((s.q_value_max - 45.6).abs() < 0.01);
        assert!((s.gradient_norm - 1.23).abs() < 0.01);
        assert!((s.learning_rate - 0.001).abs() < 0.0001);
        assert!((s.epoch_duration_seconds - 12.5).abs() < 0.01);
        assert_eq!(s.replay_buffer_size, 50000);
    }

    #[test]
    fn test_group_ppo_diagnostics() {
        let samples = vec![
            MetricSample {
                name: "foxhunt_training_policy_entropy".to_owned(),
                model: "ppo".to_owned(),
                fold: "0".to_owned(),
                value: 1.23,
            },
            MetricSample {
                name: "foxhunt_training_kl_divergence".to_owned(),
                model: "ppo".to_owned(),
                fold: "0".to_owned(),
                value: 0.008,
            },
            MetricSample {
                name: "foxhunt_training_advantage_mean".to_owned(),
                model: "ppo".to_owned(),
                fold: "0".to_owned(),
                value: 0.001,
            },
        ];
        let sessions = group_into_sessions(&samples, "");
        assert_eq!(sessions.len(), 1);
        let s = &sessions[0];
        assert!((s.policy_entropy - 1.23).abs() < 0.01);
        assert!((s.kl_divergence - 0.008).abs() < 0.001);
        assert!((s.advantage_mean - 0.001).abs() < 0.001);
    }

    #[test]
    fn test_group_hyperopt_intra_trial() {
        let samples = vec![
            MetricSample {
                name: "foxhunt_hyperopt_mode".to_owned(),
                model: "dqn".to_owned(),
                fold: "".to_owned(),
                value: 1.0,
            },
            MetricSample {
                name: "foxhunt_hyperopt_trial_epoch".to_owned(),
                model: "dqn".to_owned(),
                fold: "".to_owned(),
                value: 12.0,
            },
            MetricSample {
                name: "foxhunt_hyperopt_trial_best_loss".to_owned(),
                model: "dqn".to_owned(),
                fold: "".to_owned(),
                value: 0.042,
            },
            MetricSample {
                name: "foxhunt_hyperopt_elapsed_seconds".to_owned(),
                model: "dqn".to_owned(),
                fold: "".to_owned(),
                value: 123.4,
            },
        ];
        let sessions = group_into_sessions(&samples, "");
        assert_eq!(sessions.len(), 1);
        let s = &sessions[0];
        assert!(s.is_hyperopt);
        assert_eq!(s.hyperopt_trial_epoch, 12);
        assert!((s.hyperopt_trial_best_loss - 0.042).abs() < 0.001);
        assert!((s.hyperopt_elapsed_seconds - 123.4).abs() < 0.1);
    }
```

**Step 3: Run tests**

Run: `SQLX_OFFLINE=true cargo test -p monitoring_service --lib`
Expected: All 7 tests pass (4 existing + 3 new).

**Step 4: Commit**

```bash
git add services/monitoring_service/src/service.rs
git commit -m "feat(monitoring): wire 12 new metric→proto field mappings in group_into_sessions"
```

---

### Task 4: Enhance fxt monitor CLI display

**Files:**
- Modify: `bin/fxt/src/commands/train/monitor.rs`

**Context:** The CLI has `print_session_table` (line 127) for the main table, `print_hyperopt_summary` (line 154) for hyperopt, and `print_health_summary` (line 172) for health counters. We add gradient_norm to the main table, an RL diagnostics section, and inline hyperopt trial progress.

**Step 1: Add gradient norm column to main table**

Replace the `print_session_table` function (lines 127-152) with:

```rust
fn print_session_table(sessions: &[TrainingSession]) {
    println!(
        "{:<10} {:<6} {:<7} {:<10} {:<10} {:<9} {:<10} {:<10}",
        "Model".bright_cyan(),
        "Fold".bright_cyan(),
        "Epoch".bright_cyan(),
        "Loss".bright_cyan(),
        "Val Loss".bright_cyan(),
        "Batch/s".bright_cyan(),
        "Grad Norm".bright_cyan(),
        "Eval Acc".bright_cyan(),
    );
    println!("{}", "-".repeat(82).bright_black());

    for s in sessions {
        let acc = if s.eval_accuracy > 0.0 {
            format!("{:.1}%", s.eval_accuracy * 100.0)
        } else {
            "-".to_owned()
        };
        let grad = if s.gradient_norm > 0.0 {
            format!("{:.4}", s.gradient_norm)
        } else {
            "-".to_owned()
        };
        println!(
            "{:<10} {:<6} {:<7.0} {:<10.4} {:<10.4} {:<9.1} {:<10} {:<10}",
            s.model, s.fold, s.current_epoch, s.epoch_loss, s.validation_loss,
            s.batches_per_second, grad, acc,
        );
    }
}
```

**Step 2: Add RL diagnostics section**

After `print_session_table`, add a new function:

```rust
fn print_rl_diagnostics(sessions: &[TrainingSession]) {
    let rl_sessions: Vec<_> = sessions
        .iter()
        .filter(|s| s.q_value_mean != 0.0 || s.policy_entropy != 0.0)
        .collect();
    if rl_sessions.is_empty() {
        return;
    }
    println!();
    println!("{}", "RL Diagnostics:".bright_cyan());
    for s in &rl_sessions {
        if s.q_value_mean != 0.0 || s.q_value_max != 0.0 {
            // DQN-style
            println!(
                "  {}: Q-mean={:.2} Q-max={:.2} | buffer={}",
                s.model.bright_white(),
                s.q_value_mean,
                s.q_value_max,
                s.replay_buffer_size,
            );
        }
        if s.policy_entropy != 0.0 || s.kl_divergence != 0.0 {
            // PPO-style
            println!(
                "  {}: entropy={:.3} KL={:.4} adv-mean={:.4}",
                s.model.bright_white(),
                s.policy_entropy,
                s.kl_divergence,
                s.advantage_mean,
            );
        }
    }
}
```

**Step 3: Enhance hyperopt summary with intra-trial progress**

Replace the `print_hyperopt_summary` function (lines 154-169) with:

```rust
fn print_hyperopt_summary(sessions: &[TrainingSession]) {
    let hyperopt: Vec<_> = sessions.iter().filter(|s| s.is_hyperopt).collect();
    if hyperopt.is_empty() {
        return;
    }
    println!();
    for s in &hyperopt {
        let trial_detail = if s.hyperopt_trial_epoch > 0 {
            format!(
                " (epoch {}, loss {:.4})",
                s.hyperopt_trial_epoch, s.hyperopt_trial_best_loss
            )
        } else {
            String::new()
        };
        let elapsed = if s.hyperopt_elapsed_seconds > 0.0 {
            format!(" | {:.0}s elapsed", s.hyperopt_elapsed_seconds)
        } else {
            String::new()
        };
        println!(
            "Hyperopt ({}): trial {}/{}{} | best Sharpe {:.2} | {} failures{}",
            s.model.bright_magenta(),
            s.hyperopt_trial_current,
            s.hyperopt_trial_total,
            trial_detail,
            s.hyperopt_best_objective,
            s.hyperopt_trials_failed,
            elapsed,
        );
    }
}
```

**Step 4: Wire RL diagnostics into render_snapshot**

In `render_snapshot` (line 81), add a call to `print_rl_diagnostics` between `print_session_table` and `print_hyperopt_summary`:

Change lines 100-102 from:
```rust
    print_session_table(&resp.sessions);
    print_hyperopt_summary(&resp.sessions);
    print_health_summary(&resp.sessions);
```
to:
```rust
    print_session_table(&resp.sessions);
    print_rl_diagnostics(&resp.sessions);
    print_hyperopt_summary(&resp.sessions);
    print_health_summary(&resp.sessions);
```

**Step 5: Verify compilation**

Run: `SQLX_OFFLINE=true cargo check -p fxt`
Expected: Compiles cleanly.

**Step 6: Commit**

```bash
git add bin/fxt/src/commands/train/monitor.rs
git commit -m "feat(fxt): add grad norm, RL diagnostics, and hyperopt trial detail to monitor"
```

---

### Task 5: Record new metrics in DQN trainer

**Files:**
- Modify: `crates/ml/src/trainers/dqn/trainer.rs`

**Context:** The DQN trainer already has `use common::metrics::training_metrics;` and records 4 metrics at line 2653. The epoch loop computes `avg_q_value` and `avg_grad_norm` at line 2473. Q-value stats (min, max, mean) are available at line 2580 via `monitor.get_q_value_stats()`. Buffer size is available via `agent.get_replay_buffer_size()` (line 993).

**Step 1: Add Tier 1 metrics after existing per-epoch recording**

After the existing `training_metrics::set_batches_per_second(...)` block (~line 2661), add:

```rust
            // Tier 1: RL diagnostics
            let (q_min, q_max, q_mean) = monitor.get_q_value_stats();
            training_metrics::set_q_value_stats("dqn", "current", q_mean as f64, q_max as f64);
            training_metrics::set_gradient_norm("dqn", "current", avg_grad_norm);
            training_metrics::set_epoch_duration("dqn", "current", epoch_duration.as_secs_f64());
            {
                let agent = self.agent.read().await;
                if let Ok(buf_size) = agent.get_replay_buffer_size() {
                    training_metrics::set_replay_buffer_size("dqn", "current", buf_size as f64);
                }
            }
```

Note: `q_min`, `q_max`, `q_mean` are already used at line 2580. You need to move/duplicate the call here. The `monitor.get_q_value_stats()` call at line 2580 already exists — use the variables from there or call it again. The simplest approach is to add the metrics recording right after the existing `let (q_min, q_max, q_mean) = monitor.get_q_value_stats();` at line 2580.

Actually, the cleanest approach: add metrics recording right after line 2580 where `(q_min, q_max, q_mean)` is already bound:

```rust
            // After: let (q_min, q_max, q_mean) = monitor.get_q_value_stats();
            training_metrics::set_q_value_stats("dqn", "current", q_mean as f64, q_max as f64);
            training_metrics::set_gradient_norm("dqn", "current", avg_grad_norm);
            training_metrics::set_epoch_duration("dqn", "current", epoch_duration.as_secs_f64());
            {
                let agent = self.agent.read().await;
                if let Ok(buf_size) = agent.get_replay_buffer_size() {
                    training_metrics::set_replay_buffer_size("dqn", "current", buf_size as f64);
                }
            }
```

Note: `_ = q_min;` may be needed to suppress unused warning if `q_min` was only used in the info log. Check — if `q_min` is already used in the `info!()` macro at line 2582, no suppression needed.

**Step 2: Verify compilation**

Run: `SQLX_OFFLINE=true cargo check -p ml`
Expected: Compiles. Check for unused variable warnings.

**Step 3: Commit**

```bash
git add crates/ml/src/trainers/dqn/trainer.rs
git commit -m "feat(dqn): record Q-value, gradient norm, epoch duration, buffer size metrics"
```

---

### Task 6: Record new metrics in PPO trainer

**Files:**
- Modify: `crates/ml/src/trainers/ppo.rs`

**Context:** PPO trainer already records 3 metrics at line 668. The epoch loop computes `kl_divergence`, `explained_variance`, `mean_reward`, `std_reward`, `entropy` at line 649 via `self.compute_metrics(...)`. The `PpoEpochMetrics` struct (line 196) has all these fields.

**Step 1: Add metrics after existing recording**

After the existing `training_metrics::set_validation_loss(...)` at line 670, add:

```rust
            // Tier 1: PPO diagnostics
            training_metrics::set_policy_entropy("ppo", "current", entropy as f64);
            training_metrics::set_kl_divergence("ppo", "current", kl_divergence as f64);
            training_metrics::set_advantage_mean("ppo", "current", mean_reward as f64);

            // Tier 2: Verbose PPO metrics (no-op when disabled)
            training_metrics::set_advantage_std("ppo", "current", std_reward as f64);
            training_metrics::set_value_explained_variance("ppo", "current", explained_variance as f64);
```

Note: The variables `entropy`, `kl_divergence`, `mean_reward`, `std_reward`, `explained_variance` are computed at line 649 and available in scope.

**Step 2: Add gradient norm and epoch duration**

The PPO trainer doesn't currently track gradient norms per epoch. We need to find where the optimizer step happens. Search for the backward/optimizer step in the PPO update logic. If a gradient norm is available after the optimizer step, record it. If not, skip this for PPO (the gradient_norm metric will only be populated for DQN and supervised models).

Check if the PPO update at `self.ppo_update_step(...)` or similar returns a gradient norm. If not, we only add epoch_duration:

After the metrics block, add:

```rust
            // Epoch duration (epoch_start must be captured at top of loop)
            // Note: PPO epochs are measured by the progress_callback timing
```

Actually — looking at the PPO trainer structure, the epoch loop at line 492 iterates over `0..self.hyperparams.epochs`. We need to add timing. At the top of the epoch loop body (after `for epoch in 0..self.hyperparams.epochs {`), add:

```rust
            let epoch_start = std::time::Instant::now();
```

Then after the metrics block:

```rust
            training_metrics::set_epoch_duration("ppo", "current", epoch_start.elapsed().as_secs_f64());
```

**Step 3: Verify compilation**

Run: `SQLX_OFFLINE=true cargo check -p ml`
Expected: Compiles.

**Step 4: Commit**

```bash
git add crates/ml/src/trainers/ppo.rs
git commit -m "feat(ppo): record entropy, KL, advantage, explained variance, epoch duration metrics"
```

---

### Task 7: Record new metrics in supervised training binary

**Files:**
- Modify: `crates/ml/examples/train_baseline_supervised.rs`

**Context:** The supervised binary already records epoch/loss/val_loss/iteration_seconds at lines 564-567. The `adapter.get_learning_rate()` value is printed at line 576 but not recorded as a metric. Epoch timing is already measured at line 554 (`epoch_start`).

**Step 1: Add learning rate and epoch duration metrics**

After the existing metrics block (line 567), add:

```rust
        metrics::set_learning_rate(model_name, &fold_str, adapter.get_learning_rate());
        metrics::set_epoch_duration(model_name, &fold_str, elapsed.as_secs_f64());
```

Note: `metrics` is the alias for `training_metrics` (imported as `use common::metrics::training_metrics as metrics;` at line 46). `elapsed` is computed at line 561. `adapter.get_learning_rate()` returns `f64`.

**Step 2: Verify compilation**

Run: `SQLX_OFFLINE=true cargo check --example train_baseline_supervised`
Expected: Compiles. (Note: this example may need the full `cargo check -p ml` due to workspace deps.)

Actually, the correct command is:
Run: `SQLX_OFFLINE=true cargo check -p ml --example train_baseline_supervised`
Expected: Compiles.

**Step 3: Commit**

```bash
git add crates/ml/examples/train_baseline_supervised.rs
git commit -m "feat(supervised): record learning rate and epoch duration metrics"
```

---

### Task 8: Record hyperopt intra-trial metrics in both hyperopt binaries

**Files:**
- Modify: `crates/ml/examples/hyperopt_baseline_rl.rs`
- Modify: `crates/ml/examples/hyperopt_baseline_supervised.rs`

**Context:** Both binaries already have `use common::metrics::training_metrics;` and record hyperopt trial/best_objective. They each have a `let start = Instant::now();` at the beginning of each model's optimization function. The `elapsed` is computed after optimization completes. We need to add `elapsed_seconds` recording after optimization completes, before returning results.

**Step 1: Add elapsed_seconds to hyperopt_baseline_rl.rs**

In `run_dqn_hyperopt()`, after `training_metrics::set_hyperopt_best_objective("dqn", result.best_objective);` (~line 185), add:

```rust
    training_metrics::set_hyperopt_elapsed("dqn", elapsed);
```

In `run_ppo_hyperopt()`, after `training_metrics::set_hyperopt_best_objective("ppo", result.best_objective);` (~line 254), add:

```rust
    training_metrics::set_hyperopt_elapsed("ppo", elapsed);
```

**Step 2: Add elapsed_seconds to hyperopt_baseline_supervised.rs**

The supervised hyperopt binary has 8 model functions (TFT, Mamba2, Liquid, TGGN, TLOB, KAN, xLSTM, Diffusion). In each function, after the `training_metrics::set_hyperopt_best_objective(MODEL, result.best_objective);` line, add:

```rust
    training_metrics::set_hyperopt_elapsed(MODEL, elapsed);
```

Where MODEL is the lowercase model name string ("tft", "mamba2", "liquid", "tggn", "tlob", "kan", "xlstm", "diffusion").

Specific locations:
- `run_tft_hyperopt()`: after line 150
- `run_mamba2_hyperopt()`: after line 208
- `run_liquid_hyperopt()`: after line 267
- `run_tggn_hyperopt()`: after line 326
- `run_tlob_hyperopt()`: after line 385
- `run_kan_hyperopt()`: after line 444
- `run_xlstm_hyperopt()`: after line 503
- `run_diffusion_hyperopt()`: after line 562

**Step 3: Verify compilation**

Run: `SQLX_OFFLINE=true cargo check -p ml --example hyperopt_baseline_rl`
Run: `SQLX_OFFLINE=true cargo check -p ml --example hyperopt_baseline_supervised`
Expected: Both compile.

**Step 4: Commit**

```bash
git add crates/ml/examples/hyperopt_baseline_rl.rs crates/ml/examples/hyperopt_baseline_supervised.rs
git commit -m "feat(hyperopt): record elapsed_seconds metric in all hyperopt binaries"
```

---

### Task 9: Full workspace check + clippy

**Files:** None (verification only)

**Step 1: Full workspace check**

Run: `SQLX_OFFLINE=true cargo check --workspace`
Expected: 0 errors.

**Step 2: Clippy**

Run: `SQLX_OFFLINE=true cargo clippy --workspace -- -D warnings`
Expected: 0 errors, 0 warnings.

**Step 3: Run all affected tests**

Run: `SQLX_OFFLINE=true cargo test -p common --lib -- training_metrics`
Run: `SQLX_OFFLINE=true cargo test -p monitoring_service --lib`
Expected: All tests pass.

**Step 4: Commit (if any fixes needed)**

```bash
git commit -am "fix: resolve clippy/test issues from metrics additions"
```

Only create this commit if there were issues to fix. If everything passes, skip this step.