docs: implementation plan for 17 additional training metrics
9 tasks covering: training_metrics.rs (definitions + verbose gate), monitoring.proto (12 new fields), service.rs (match arms + tests), fxt monitor CLI (grad norm, RL diagnostics, hyperopt trial detail), DQN/PPO trainers (per-epoch recording), supervised binary (LR + duration), hyperopt binaries (elapsed_seconds), and full workspace verification. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
875
docs/plans/2026-03-02-additional-training-metrics.md
Normal file
875
docs/plans/2026-03-02-additional-training-metrics.md
Normal file
@@ -0,0 +1,875 @@
|
||||
# Additional Training Metrics Implementation Plan
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Add 17 new Prometheus metrics (12 always-on gauges + 5 opt-in verbose) to provide full RL diagnostics, gradient health, and hyperopt intra-trial visibility across the monitoring pipeline.
|
||||
|
||||
**Architecture:** Two-tier metrics system. Tier 1 (12 gauges) — zero-cost atomic writes in every trainer's epoch loop. Tier 2 (5 metrics) — gated behind `FOXHUNT_VERBOSE_METRICS=1`. All metrics flow through the existing Prometheus → monitoring_service → gRPC → fxt CLI pipeline. No new queries needed (wildcard `foxhunt_training_*|foxhunt_hyperopt_*` already captures new metrics).
|
||||
|
||||
**Tech Stack:** Rust, prometheus crate, tonic/prost (gRPC), protobuf, Candle ML framework
|
||||
|
||||
**Build command:** `SQLX_OFFLINE=true cargo check --workspace`
|
||||
**Test command:** `SQLX_OFFLINE=true cargo test -p <crate> --lib`
|
||||
**Clippy rules:** `#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::indexing_slicing)]` — use `.get()`, `?`, `.ok_or()` instead.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Register Tier 1 metrics + verbose gate in training_metrics.rs
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/common/src/metrics/training_metrics.rs`
|
||||
|
||||
This task adds 12 new gauge registrations, 5 verbose metric registrations with env-var gating, and typed helper functions for all 17 new metrics.
|
||||
|
||||
**Context:** The file currently has 24 metrics registered in `init()` (lines 32-168) and typed helpers (lines 174-297). Tests are at lines 299-340. The module uses helper functions from `super::` (`register_gauge_vec`, `set_gauge_vec`, `register_histogram_vec`, `observe_histogram_vec`). Labels use `&["model", "fold"]` for training metrics and `&["model"]` for hyperopt metrics.
|
||||
|
||||
**Step 1: Add the verbose gate**
|
||||
|
||||
At the top of the file, after the `use super::{...}` block (line 23), add:
|
||||
|
||||
```rust
|
||||
use std::sync::OnceLock;
|
||||
|
||||
static VERBOSE: OnceLock<bool> = OnceLock::new();
|
||||
|
||||
/// Returns true if `FOXHUNT_VERBOSE_METRICS` env var is set.
|
||||
/// Checked once at first call and cached.
|
||||
pub fn verbose_enabled() -> bool {
|
||||
*VERBOSE.get_or_init(|| std::env::var("FOXHUNT_VERBOSE_METRICS").is_ok())
|
||||
}
|
||||
```
|
||||
|
||||
**Step 2: Register Tier 1 gauges in `init()`**
|
||||
|
||||
After the existing `register_gauge("foxhunt_training_active_workers", ...)` block (line 130) and before the `// Hyperopt gauges` comment (line 132), add these 9 training gauges:
|
||||
|
||||
```rust
|
||||
// RL diagnostic gauges (model + fold)
|
||||
_ = register_gauge_vec("foxhunt_training_q_value_mean", "Mean Q-value per epoch (DQN)", mf);
|
||||
_ = register_gauge_vec("foxhunt_training_q_value_max", "Max Q-value per epoch (DQN)", mf);
|
||||
_ = register_gauge_vec("foxhunt_training_policy_entropy", "Policy entropy (PPO)", mf);
|
||||
_ = register_gauge_vec("foxhunt_training_kl_divergence", "KL divergence old vs new policy (PPO)", mf);
|
||||
_ = register_gauge_vec("foxhunt_training_advantage_mean", "Mean advantage estimate (PPO)", mf);
|
||||
_ = register_gauge_vec("foxhunt_training_replay_buffer_size", "Replay buffer occupancy (DQN)", mf);
|
||||
|
||||
// Gradient & training health gauges (model + fold)
|
||||
_ = register_gauge_vec("foxhunt_training_gradient_norm", "L2 gradient norm after backward", mf);
|
||||
_ = register_gauge_vec("foxhunt_training_learning_rate", "Effective learning rate", mf);
|
||||
_ = register_gauge_vec("foxhunt_training_epoch_duration_seconds", "Wall-clock time per epoch", mf);
|
||||
```
|
||||
|
||||
After the existing `register_gauge_vec("foxhunt_hyperopt_mode", ...)` block (line 152) and before the `// Hyperopt counters` comment (line 154), add these 3 hyperopt gauges:
|
||||
|
||||
```rust
|
||||
// Hyperopt intra-trial gauges (model only)
|
||||
_ = register_gauge_vec("foxhunt_hyperopt_trial_epoch", "Current epoch within active trial", m);
|
||||
_ = register_gauge_vec("foxhunt_hyperopt_trial_best_loss", "Best loss in current trial", m);
|
||||
_ = register_gauge_vec("foxhunt_hyperopt_elapsed_seconds", "Total wall-clock time since hyperopt start", m);
|
||||
```
|
||||
|
||||
**Step 3: Register Tier 2 verbose metrics in `init()`**
|
||||
|
||||
At the end of `init()`, before the closing `}`, add:
|
||||
|
||||
```rust
|
||||
// Tier 2: Verbose metrics (only registered when FOXHUNT_VERBOSE_METRICS is set)
|
||||
if verbose_enabled() {
|
||||
let mfl: &[&str] = &["model", "fold", "layer"];
|
||||
_ = register_histogram_vec(
|
||||
"foxhunt_training_batch_loss",
|
||||
"Per-batch loss distribution",
|
||||
mf,
|
||||
vec![0.001, 0.01, 0.05, 0.1, 0.5, 1.0, 5.0, 10.0],
|
||||
);
|
||||
_ = register_histogram_vec(
|
||||
"foxhunt_training_gradient_norm_per_layer",
|
||||
"Per-layer gradient L2 norm",
|
||||
mfl,
|
||||
vec![0.0001, 0.001, 0.01, 0.1, 1.0, 10.0, 100.0],
|
||||
);
|
||||
_ = register_gauge_vec("foxhunt_training_replay_priority_mean", "Mean PER priority", mf);
|
||||
_ = register_gauge_vec("foxhunt_training_advantage_std", "Advantage standard deviation (PPO)", mf);
|
||||
_ = register_gauge_vec("foxhunt_training_value_explained_variance", "Value function explained variance (PPO)", mf);
|
||||
}
|
||||
```
|
||||
|
||||
**Step 4: Add typed helpers for Tier 1**
|
||||
|
||||
After the existing `set_hyperopt_mode` helper (line 297), add:
|
||||
|
||||
```rust
|
||||
// ---------------------------------------------------------------------------
|
||||
// Tier 1: RL diagnostics
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
pub fn set_q_value_stats(model: &str, fold: &str, mean: f64, max: f64) {
|
||||
set_gauge_vec("foxhunt_training_q_value_mean", &[model, fold], mean);
|
||||
set_gauge_vec("foxhunt_training_q_value_max", &[model, fold], max);
|
||||
}
|
||||
|
||||
pub fn set_policy_entropy(model: &str, fold: &str, entropy: f64) {
|
||||
set_gauge_vec("foxhunt_training_policy_entropy", &[model, fold], entropy);
|
||||
}
|
||||
|
||||
pub fn set_kl_divergence(model: &str, fold: &str, kl: f64) {
|
||||
set_gauge_vec("foxhunt_training_kl_divergence", &[model, fold], kl);
|
||||
}
|
||||
|
||||
pub fn set_advantage_mean(model: &str, fold: &str, mean: f64) {
|
||||
set_gauge_vec("foxhunt_training_advantage_mean", &[model, fold], mean);
|
||||
}
|
||||
|
||||
pub fn set_replay_buffer_size(model: &str, fold: &str, size: f64) {
|
||||
set_gauge_vec("foxhunt_training_replay_buffer_size", &[model, fold], size);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Tier 1: Gradient & training health
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
pub fn set_gradient_norm(model: &str, fold: &str, norm: f64) {
|
||||
set_gauge_vec("foxhunt_training_gradient_norm", &[model, fold], norm);
|
||||
}
|
||||
|
||||
pub fn set_learning_rate(model: &str, fold: &str, lr: f64) {
|
||||
set_gauge_vec("foxhunt_training_learning_rate", &[model, fold], lr);
|
||||
}
|
||||
|
||||
pub fn set_epoch_duration(model: &str, fold: &str, secs: f64) {
|
||||
set_gauge_vec("foxhunt_training_epoch_duration_seconds", &[model, fold], secs);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Tier 1: Hyperopt intra-trial
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
pub fn set_hyperopt_trial_epoch(model: &str, epoch: f64) {
|
||||
set_gauge_vec("foxhunt_hyperopt_trial_epoch", &[model], epoch);
|
||||
}
|
||||
|
||||
pub fn set_hyperopt_trial_best_loss(model: &str, loss: f64) {
|
||||
set_gauge_vec("foxhunt_hyperopt_trial_best_loss", &[model], loss);
|
||||
}
|
||||
|
||||
pub fn set_hyperopt_elapsed(model: &str, secs: f64) {
|
||||
set_gauge_vec("foxhunt_hyperopt_elapsed_seconds", &[model], secs);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Tier 2: Verbose metrics (no-op when FOXHUNT_VERBOSE_METRICS not set)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
pub fn observe_batch_loss(model: &str, fold: &str, loss: f64) {
|
||||
if verbose_enabled() {
|
||||
observe_histogram_vec("foxhunt_training_batch_loss", &[model, fold], loss);
|
||||
}
|
||||
}
|
||||
|
||||
pub fn observe_gradient_norm_per_layer(model: &str, fold: &str, layer: &str, norm: f64) {
|
||||
if verbose_enabled() {
|
||||
observe_histogram_vec("foxhunt_training_gradient_norm_per_layer", &[model, fold, layer], norm);
|
||||
}
|
||||
}
|
||||
|
||||
pub fn set_replay_priority_mean(model: &str, fold: &str, mean: f64) {
|
||||
if verbose_enabled() {
|
||||
set_gauge_vec("foxhunt_training_replay_priority_mean", &[model, fold], mean);
|
||||
}
|
||||
}
|
||||
|
||||
pub fn set_advantage_std(model: &str, fold: &str, std: f64) {
|
||||
if verbose_enabled() {
|
||||
set_gauge_vec("foxhunt_training_advantage_std", &[model, fold], std);
|
||||
}
|
||||
}
|
||||
|
||||
pub fn set_value_explained_variance(model: &str, fold: &str, ev: f64) {
|
||||
if verbose_enabled() {
|
||||
set_gauge_vec("foxhunt_training_value_explained_variance", &[model, fold], ev);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Step 5: Add tests**
|
||||
|
||||
Add these tests inside the existing `#[cfg(test)] mod tests` block, after the last test:
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn test_verbose_disabled_by_default() {
|
||||
// FOXHUNT_VERBOSE_METRICS not set in test env
|
||||
// verbose helpers should not panic even when metrics aren't registered
|
||||
init();
|
||||
// These are no-ops when verbose is disabled
|
||||
observe_batch_loss("dqn", "0", 0.05);
|
||||
observe_gradient_norm_per_layer("dqn", "0", "fc1", 0.01);
|
||||
set_replay_priority_mean("dqn", "0", 1.5);
|
||||
set_advantage_std("ppo", "0", 0.8);
|
||||
set_value_explained_variance("ppo", "0", 0.7);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_tier1_helpers_no_panic() {
|
||||
init();
|
||||
set_q_value_stats("dqn", "0", 12.3, 45.6);
|
||||
set_policy_entropy("ppo", "0", 1.23);
|
||||
set_kl_divergence("ppo", "0", 0.008);
|
||||
set_advantage_mean("ppo", "0", 0.001);
|
||||
set_replay_buffer_size("dqn", "0", 50000.0);
|
||||
set_gradient_norm("dqn", "0", 1.23);
|
||||
set_learning_rate("dqn", "0", 0.001);
|
||||
set_epoch_duration("dqn", "0", 12.5);
|
||||
set_hyperopt_trial_epoch("dqn", 5.0);
|
||||
set_hyperopt_trial_best_loss("dqn", 0.042);
|
||||
set_hyperopt_elapsed("dqn", 123.4);
|
||||
}
|
||||
```
|
||||
|
||||
**Step 6: Run tests**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo test -p common --lib -- training_metrics`
|
||||
Expected: All tests pass, including the 2 new ones.
|
||||
|
||||
**Step 7: Clippy check**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo clippy -p common -- -D warnings`
|
||||
Expected: 0 errors, 0 warnings.
|
||||
|
||||
**Step 8: Commit**
|
||||
|
||||
```bash
|
||||
git add crates/common/src/metrics/training_metrics.rs
|
||||
git commit -m "feat(metrics): register 17 new Prometheus metrics with verbose gating"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Extend monitoring.proto with new TrainingSession fields
|
||||
|
||||
**Files:**
|
||||
- Modify: `services/monitoring_service/proto/monitoring.proto`
|
||||
|
||||
**Context:** `TrainingSession` currently has fields 1-23. The design doc assigns field numbers 24-35 for new fields. Note: the existing hyperopt fields already use numbers 20-23, so the new fields start at 24.
|
||||
|
||||
**Step 1: Add new fields to TrainingSession**
|
||||
|
||||
After the `uint32 hyperopt_trials_failed = 23;` line (line 65), add:
|
||||
|
||||
```protobuf
|
||||
|
||||
// RL diagnostics
|
||||
float q_value_mean = 24;
|
||||
float q_value_max = 25;
|
||||
float policy_entropy = 26;
|
||||
float kl_divergence = 27;
|
||||
float advantage_mean = 28;
|
||||
uint32 replay_buffer_size = 29;
|
||||
|
||||
// Gradient & training health
|
||||
float gradient_norm = 30;
|
||||
float learning_rate = 31;
|
||||
float epoch_duration_seconds = 32;
|
||||
|
||||
// Hyperopt intra-trial
|
||||
uint32 hyperopt_trial_epoch = 33;
|
||||
float hyperopt_trial_best_loss = 34;
|
||||
float hyperopt_elapsed_seconds = 35;
|
||||
```
|
||||
|
||||
**Step 2: Verify proto compiles**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo check -p monitoring_service`
|
||||
Expected: Compiles (prost generates the new fields as `f32`/`u32` with default 0).
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo check -p fxt`
|
||||
Expected: Compiles (fxt also compiles this proto).
|
||||
|
||||
**Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add services/monitoring_service/proto/monitoring.proto
|
||||
git commit -m "proto(monitoring): add 12 new TrainingSession fields for diagnostics"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Wire new metrics through monitoring_service group_into_sessions
|
||||
|
||||
**Files:**
|
||||
- Modify: `services/monitoring_service/src/service.rs`
|
||||
|
||||
**Context:** The `group_into_sessions` function (line 98) has a `match s.name.as_str()` block (lines 117-158) that maps Prometheus metric names to TrainingSession proto fields. We add 12 new match arms.
|
||||
|
||||
**Step 1: Add match arms**
|
||||
|
||||
Inside the `match s.name.as_str()` block, after the `"foxhunt_hyperopt_mode"` arm (line 153-157) and before the `_ => {}` wildcard (line 158), add:
|
||||
|
||||
```rust
|
||||
// RL diagnostics
|
||||
"foxhunt_training_q_value_mean" => session.q_value_mean = s.value as f32,
|
||||
"foxhunt_training_q_value_max" => session.q_value_max = s.value as f32,
|
||||
"foxhunt_training_policy_entropy" => session.policy_entropy = s.value as f32,
|
||||
"foxhunt_training_kl_divergence" => session.kl_divergence = s.value as f32,
|
||||
"foxhunt_training_advantage_mean" => session.advantage_mean = s.value as f32,
|
||||
"foxhunt_training_replay_buffer_size" => {
|
||||
session.replay_buffer_size = s.value as u32;
|
||||
}
|
||||
// Gradient & training health
|
||||
"foxhunt_training_gradient_norm" => session.gradient_norm = s.value as f32,
|
||||
"foxhunt_training_learning_rate" => session.learning_rate = s.value as f32,
|
||||
"foxhunt_training_epoch_duration_seconds" => {
|
||||
session.epoch_duration_seconds = s.value as f32;
|
||||
}
|
||||
// Hyperopt intra-trial
|
||||
"foxhunt_hyperopt_trial_epoch" => session.hyperopt_trial_epoch = s.value as u32,
|
||||
"foxhunt_hyperopt_trial_best_loss" => {
|
||||
session.hyperopt_trial_best_loss = s.value as f32;
|
||||
}
|
||||
"foxhunt_hyperopt_elapsed_seconds" => {
|
||||
session.hyperopt_elapsed_seconds = s.value as f32;
|
||||
}
|
||||
```
|
||||
|
||||
**Step 2: Add tests for new metric mapping**
|
||||
|
||||
Add a new test in the `#[cfg(test)] mod tests` block:
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn test_group_new_diagnostic_metrics() {
|
||||
let samples = vec![
|
||||
MetricSample {
|
||||
name: "foxhunt_training_current_epoch".to_owned(),
|
||||
model: "dqn".to_owned(),
|
||||
fold: "0".to_owned(),
|
||||
value: 25.0,
|
||||
},
|
||||
MetricSample {
|
||||
name: "foxhunt_training_q_value_mean".to_owned(),
|
||||
model: "dqn".to_owned(),
|
||||
fold: "0".to_owned(),
|
||||
value: 12.3,
|
||||
},
|
||||
MetricSample {
|
||||
name: "foxhunt_training_q_value_max".to_owned(),
|
||||
model: "dqn".to_owned(),
|
||||
fold: "0".to_owned(),
|
||||
value: 45.6,
|
||||
},
|
||||
MetricSample {
|
||||
name: "foxhunt_training_gradient_norm".to_owned(),
|
||||
model: "dqn".to_owned(),
|
||||
fold: "0".to_owned(),
|
||||
value: 1.23,
|
||||
},
|
||||
MetricSample {
|
||||
name: "foxhunt_training_learning_rate".to_owned(),
|
||||
model: "dqn".to_owned(),
|
||||
fold: "0".to_owned(),
|
||||
value: 0.001,
|
||||
},
|
||||
MetricSample {
|
||||
name: "foxhunt_training_epoch_duration_seconds".to_owned(),
|
||||
model: "dqn".to_owned(),
|
||||
fold: "0".to_owned(),
|
||||
value: 12.5,
|
||||
},
|
||||
MetricSample {
|
||||
name: "foxhunt_training_replay_buffer_size".to_owned(),
|
||||
model: "dqn".to_owned(),
|
||||
fold: "0".to_owned(),
|
||||
value: 50000.0,
|
||||
},
|
||||
];
|
||||
let sessions = group_into_sessions(&samples, "");
|
||||
assert_eq!(sessions.len(), 1);
|
||||
let s = &sessions[0];
|
||||
assert_eq!(s.current_epoch, 25.0);
|
||||
assert!((s.q_value_mean - 12.3).abs() < 0.01);
|
||||
assert!((s.q_value_max - 45.6).abs() < 0.01);
|
||||
assert!((s.gradient_norm - 1.23).abs() < 0.01);
|
||||
assert!((s.learning_rate - 0.001).abs() < 0.0001);
|
||||
assert!((s.epoch_duration_seconds - 12.5).abs() < 0.01);
|
||||
assert_eq!(s.replay_buffer_size, 50000);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_group_ppo_diagnostics() {
|
||||
let samples = vec![
|
||||
MetricSample {
|
||||
name: "foxhunt_training_policy_entropy".to_owned(),
|
||||
model: "ppo".to_owned(),
|
||||
fold: "0".to_owned(),
|
||||
value: 1.23,
|
||||
},
|
||||
MetricSample {
|
||||
name: "foxhunt_training_kl_divergence".to_owned(),
|
||||
model: "ppo".to_owned(),
|
||||
fold: "0".to_owned(),
|
||||
value: 0.008,
|
||||
},
|
||||
MetricSample {
|
||||
name: "foxhunt_training_advantage_mean".to_owned(),
|
||||
model: "ppo".to_owned(),
|
||||
fold: "0".to_owned(),
|
||||
value: 0.001,
|
||||
},
|
||||
];
|
||||
let sessions = group_into_sessions(&samples, "");
|
||||
assert_eq!(sessions.len(), 1);
|
||||
let s = &sessions[0];
|
||||
assert!((s.policy_entropy - 1.23).abs() < 0.01);
|
||||
assert!((s.kl_divergence - 0.008).abs() < 0.001);
|
||||
assert!((s.advantage_mean - 0.001).abs() < 0.001);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_group_hyperopt_intra_trial() {
|
||||
let samples = vec![
|
||||
MetricSample {
|
||||
name: "foxhunt_hyperopt_mode".to_owned(),
|
||||
model: "dqn".to_owned(),
|
||||
fold: "".to_owned(),
|
||||
value: 1.0,
|
||||
},
|
||||
MetricSample {
|
||||
name: "foxhunt_hyperopt_trial_epoch".to_owned(),
|
||||
model: "dqn".to_owned(),
|
||||
fold: "".to_owned(),
|
||||
value: 12.0,
|
||||
},
|
||||
MetricSample {
|
||||
name: "foxhunt_hyperopt_trial_best_loss".to_owned(),
|
||||
model: "dqn".to_owned(),
|
||||
fold: "".to_owned(),
|
||||
value: 0.042,
|
||||
},
|
||||
MetricSample {
|
||||
name: "foxhunt_hyperopt_elapsed_seconds".to_owned(),
|
||||
model: "dqn".to_owned(),
|
||||
fold: "".to_owned(),
|
||||
value: 123.4,
|
||||
},
|
||||
];
|
||||
let sessions = group_into_sessions(&samples, "");
|
||||
assert_eq!(sessions.len(), 1);
|
||||
let s = &sessions[0];
|
||||
assert!(s.is_hyperopt);
|
||||
assert_eq!(s.hyperopt_trial_epoch, 12);
|
||||
assert!((s.hyperopt_trial_best_loss - 0.042).abs() < 0.001);
|
||||
assert!((s.hyperopt_elapsed_seconds - 123.4).abs() < 0.1);
|
||||
}
|
||||
```
|
||||
|
||||
**Step 3: Run tests**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo test -p monitoring_service --lib`
|
||||
Expected: All 7 tests pass (4 existing + 3 new).
|
||||
|
||||
**Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add services/monitoring_service/src/service.rs
|
||||
git commit -m "feat(monitoring): wire 12 new metric→proto field mappings in group_into_sessions"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Enhance fxt monitor CLI display
|
||||
|
||||
**Files:**
|
||||
- Modify: `bin/fxt/src/commands/train/monitor.rs`
|
||||
|
||||
**Context:** The CLI has `print_session_table` (line 127) for the main table, `print_hyperopt_summary` (line 154) for hyperopt, and `print_health_summary` (line 172) for health counters. We add gradient_norm to the main table, an RL diagnostics section, and inline hyperopt trial progress.
|
||||
|
||||
**Step 1: Add gradient norm column to main table**
|
||||
|
||||
Replace the `print_session_table` function (lines 127-152) with:
|
||||
|
||||
```rust
|
||||
fn print_session_table(sessions: &[TrainingSession]) {
|
||||
println!(
|
||||
"{:<10} {:<6} {:<7} {:<10} {:<10} {:<9} {:<10} {:<10}",
|
||||
"Model".bright_cyan(),
|
||||
"Fold".bright_cyan(),
|
||||
"Epoch".bright_cyan(),
|
||||
"Loss".bright_cyan(),
|
||||
"Val Loss".bright_cyan(),
|
||||
"Batch/s".bright_cyan(),
|
||||
"Grad Norm".bright_cyan(),
|
||||
"Eval Acc".bright_cyan(),
|
||||
);
|
||||
println!("{}", "-".repeat(82).bright_black());
|
||||
|
||||
for s in sessions {
|
||||
let acc = if s.eval_accuracy > 0.0 {
|
||||
format!("{:.1}%", s.eval_accuracy * 100.0)
|
||||
} else {
|
||||
"-".to_owned()
|
||||
};
|
||||
let grad = if s.gradient_norm > 0.0 {
|
||||
format!("{:.4}", s.gradient_norm)
|
||||
} else {
|
||||
"-".to_owned()
|
||||
};
|
||||
println!(
|
||||
"{:<10} {:<6} {:<7.0} {:<10.4} {:<10.4} {:<9.1} {:<10} {:<10}",
|
||||
s.model, s.fold, s.current_epoch, s.epoch_loss, s.validation_loss,
|
||||
s.batches_per_second, grad, acc,
|
||||
);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Step 2: Add RL diagnostics section**
|
||||
|
||||
After `print_session_table`, add a new function:
|
||||
|
||||
```rust
|
||||
fn print_rl_diagnostics(sessions: &[TrainingSession]) {
|
||||
let rl_sessions: Vec<_> = sessions
|
||||
.iter()
|
||||
.filter(|s| s.q_value_mean != 0.0 || s.policy_entropy != 0.0)
|
||||
.collect();
|
||||
if rl_sessions.is_empty() {
|
||||
return;
|
||||
}
|
||||
println!();
|
||||
println!("{}", "RL Diagnostics:".bright_cyan());
|
||||
for s in &rl_sessions {
|
||||
if s.q_value_mean != 0.0 || s.q_value_max != 0.0 {
|
||||
// DQN-style
|
||||
println!(
|
||||
" {}: Q-mean={:.2} Q-max={:.2} | buffer={}",
|
||||
s.model.bright_white(),
|
||||
s.q_value_mean,
|
||||
s.q_value_max,
|
||||
s.replay_buffer_size,
|
||||
);
|
||||
}
|
||||
if s.policy_entropy != 0.0 || s.kl_divergence != 0.0 {
|
||||
// PPO-style
|
||||
println!(
|
||||
" {}: entropy={:.3} KL={:.4} adv-mean={:.4}",
|
||||
s.model.bright_white(),
|
||||
s.policy_entropy,
|
||||
s.kl_divergence,
|
||||
s.advantage_mean,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Step 3: Enhance hyperopt summary with intra-trial progress**
|
||||
|
||||
Replace the `print_hyperopt_summary` function (lines 154-169) with:
|
||||
|
||||
```rust
|
||||
fn print_hyperopt_summary(sessions: &[TrainingSession]) {
|
||||
let hyperopt: Vec<_> = sessions.iter().filter(|s| s.is_hyperopt).collect();
|
||||
if hyperopt.is_empty() {
|
||||
return;
|
||||
}
|
||||
println!();
|
||||
for s in &hyperopt {
|
||||
let trial_detail = if s.hyperopt_trial_epoch > 0 {
|
||||
format!(
|
||||
" (epoch {}, loss {:.4})",
|
||||
s.hyperopt_trial_epoch, s.hyperopt_trial_best_loss
|
||||
)
|
||||
} else {
|
||||
String::new()
|
||||
};
|
||||
let elapsed = if s.hyperopt_elapsed_seconds > 0.0 {
|
||||
format!(" | {:.0}s elapsed", s.hyperopt_elapsed_seconds)
|
||||
} else {
|
||||
String::new()
|
||||
};
|
||||
println!(
|
||||
"Hyperopt ({}): trial {}/{}{} | best Sharpe {:.2} | {} failures{}",
|
||||
s.model.bright_magenta(),
|
||||
s.hyperopt_trial_current,
|
||||
s.hyperopt_trial_total,
|
||||
trial_detail,
|
||||
s.hyperopt_best_objective,
|
||||
s.hyperopt_trials_failed,
|
||||
elapsed,
|
||||
);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Step 4: Wire RL diagnostics into render_snapshot**
|
||||
|
||||
In `render_snapshot` (line 81), add a call to `print_rl_diagnostics` between `print_session_table` and `print_hyperopt_summary`:
|
||||
|
||||
Change lines 100-102 from:
|
||||
```rust
|
||||
print_session_table(&resp.sessions);
|
||||
print_hyperopt_summary(&resp.sessions);
|
||||
print_health_summary(&resp.sessions);
|
||||
```
|
||||
to:
|
||||
```rust
|
||||
print_session_table(&resp.sessions);
|
||||
print_rl_diagnostics(&resp.sessions);
|
||||
print_hyperopt_summary(&resp.sessions);
|
||||
print_health_summary(&resp.sessions);
|
||||
```
|
||||
|
||||
**Step 5: Verify compilation**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo check -p fxt`
|
||||
Expected: Compiles cleanly.
|
||||
|
||||
**Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add bin/fxt/src/commands/train/monitor.rs
|
||||
git commit -m "feat(fxt): add grad norm, RL diagnostics, and hyperopt trial detail to monitor"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Record new metrics in DQN trainer
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml/src/trainers/dqn/trainer.rs`
|
||||
|
||||
**Context:** The DQN trainer already has `use common::metrics::training_metrics;` and records 4 metrics at line 2653. The epoch loop computes `avg_q_value` and `avg_grad_norm` at line 2473. Q-value stats (min, max, mean) are available at line 2580 via `monitor.get_q_value_stats()`. Buffer size is available via `agent.get_replay_buffer_size()` (line 993).
|
||||
|
||||
**Step 1: Add Tier 1 metrics after existing per-epoch recording**
|
||||
|
||||
After the existing `training_metrics::set_batches_per_second(...)` block (~line 2661), add:
|
||||
|
||||
```rust
|
||||
// Tier 1: RL diagnostics
|
||||
let (q_min, q_max, q_mean) = monitor.get_q_value_stats();
|
||||
training_metrics::set_q_value_stats("dqn", "current", q_mean as f64, q_max as f64);
|
||||
training_metrics::set_gradient_norm("dqn", "current", avg_grad_norm);
|
||||
training_metrics::set_epoch_duration("dqn", "current", epoch_duration.as_secs_f64());
|
||||
{
|
||||
let agent = self.agent.read().await;
|
||||
if let Ok(buf_size) = agent.get_replay_buffer_size() {
|
||||
training_metrics::set_replay_buffer_size("dqn", "current", buf_size as f64);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Note: `q_min`, `q_max`, `q_mean` are already used at line 2580. You need to move/duplicate the call here. The `monitor.get_q_value_stats()` call at line 2580 already exists — use the variables from there or call it again. The simplest approach is to add the metrics recording right after the existing `let (q_min, q_max, q_mean) = monitor.get_q_value_stats();` at line 2580.
|
||||
|
||||
Actually, the cleanest approach: add metrics recording right after line 2580 where `(q_min, q_max, q_mean)` is already bound:
|
||||
|
||||
```rust
|
||||
// After: let (q_min, q_max, q_mean) = monitor.get_q_value_stats();
|
||||
training_metrics::set_q_value_stats("dqn", "current", q_mean as f64, q_max as f64);
|
||||
training_metrics::set_gradient_norm("dqn", "current", avg_grad_norm);
|
||||
training_metrics::set_epoch_duration("dqn", "current", epoch_duration.as_secs_f64());
|
||||
{
|
||||
let agent = self.agent.read().await;
|
||||
if let Ok(buf_size) = agent.get_replay_buffer_size() {
|
||||
training_metrics::set_replay_buffer_size("dqn", "current", buf_size as f64);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Note: `_ = q_min;` may be needed to suppress unused warning if `q_min` was only used in the info log. Check — if `q_min` is already used in the `info!()` macro at line 2582, no suppression needed.
|
||||
|
||||
**Step 2: Verify compilation**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo check -p ml`
|
||||
Expected: Compiles. Check for unused variable warnings.
|
||||
|
||||
**Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add crates/ml/src/trainers/dqn/trainer.rs
|
||||
git commit -m "feat(dqn): record Q-value, gradient norm, epoch duration, buffer size metrics"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 6: Record new metrics in PPO trainer
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml/src/trainers/ppo.rs`
|
||||
|
||||
**Context:** PPO trainer already records 3 metrics at line 668. The epoch loop computes `kl_divergence`, `explained_variance`, `mean_reward`, `std_reward`, `entropy` at line 649 via `self.compute_metrics(...)`. The `PpoEpochMetrics` struct (line 196) has all these fields.
|
||||
|
||||
**Step 1: Add metrics after existing recording**
|
||||
|
||||
After the existing `training_metrics::set_validation_loss(...)` at line 670, add:
|
||||
|
||||
```rust
|
||||
// Tier 1: PPO diagnostics
|
||||
training_metrics::set_policy_entropy("ppo", "current", entropy as f64);
|
||||
training_metrics::set_kl_divergence("ppo", "current", kl_divergence as f64);
|
||||
training_metrics::set_advantage_mean("ppo", "current", mean_reward as f64);
|
||||
|
||||
// Tier 2: Verbose PPO metrics (no-op when disabled)
|
||||
training_metrics::set_advantage_std("ppo", "current", std_reward as f64);
|
||||
training_metrics::set_value_explained_variance("ppo", "current", explained_variance as f64);
|
||||
```
|
||||
|
||||
Note: The variables `entropy`, `kl_divergence`, `mean_reward`, `std_reward`, `explained_variance` are computed at line 649 and available in scope.
|
||||
|
||||
**Step 2: Add gradient norm and epoch duration**
|
||||
|
||||
The PPO trainer doesn't currently track gradient norms per epoch. We need to find where the optimizer step happens. Search for the backward/optimizer step in the PPO update logic. If a gradient norm is available after the optimizer step, record it. If not, skip this for PPO (the gradient_norm metric will only be populated for DQN and supervised models).
|
||||
|
||||
Check if the PPO update at `self.ppo_update_step(...)` or similar returns a gradient norm. If not, we only add epoch_duration:
|
||||
|
||||
After the metrics block, add:
|
||||
|
||||
```rust
|
||||
// Epoch duration (epoch_start must be captured at top of loop)
|
||||
// Note: PPO epochs are measured by the progress_callback timing
|
||||
```
|
||||
|
||||
Actually — looking at the PPO trainer structure, the epoch loop at line 492 iterates over `0..self.hyperparams.epochs`. We need to add timing. At the top of the epoch loop body (after `for epoch in 0..self.hyperparams.epochs {`), add:
|
||||
|
||||
```rust
|
||||
let epoch_start = std::time::Instant::now();
|
||||
```
|
||||
|
||||
Then after the metrics block:
|
||||
|
||||
```rust
|
||||
training_metrics::set_epoch_duration("ppo", "current", epoch_start.elapsed().as_secs_f64());
|
||||
```
|
||||
|
||||
**Step 3: Verify compilation**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo check -p ml`
|
||||
Expected: Compiles.
|
||||
|
||||
**Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add crates/ml/src/trainers/ppo.rs
|
||||
git commit -m "feat(ppo): record entropy, KL, advantage, explained variance, epoch duration metrics"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 7: Record new metrics in supervised training binary
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml/examples/train_baseline_supervised.rs`
|
||||
|
||||
**Context:** The supervised binary already records epoch/loss/val_loss/iteration_seconds at lines 564-567. The `adapter.get_learning_rate()` value is printed at line 576 but not recorded as a metric. Epoch timing is already measured at line 554 (`epoch_start`).
|
||||
|
||||
**Step 1: Add learning rate and epoch duration metrics**
|
||||
|
||||
After the existing metrics block (line 567), add:
|
||||
|
||||
```rust
|
||||
metrics::set_learning_rate(model_name, &fold_str, adapter.get_learning_rate());
|
||||
metrics::set_epoch_duration(model_name, &fold_str, elapsed.as_secs_f64());
|
||||
```
|
||||
|
||||
Note: `metrics` is the alias for `training_metrics` (imported as `use common::metrics::training_metrics as metrics;` at line 46). `elapsed` is computed at line 561. `adapter.get_learning_rate()` returns `f64`.
|
||||
|
||||
**Step 2: Verify compilation**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo check --example train_baseline_supervised`
|
||||
Expected: Compiles. (Note: this example may need the full `cargo check -p ml` due to workspace deps.)
|
||||
|
||||
Actually, the correct command is:
|
||||
Run: `SQLX_OFFLINE=true cargo check -p ml --example train_baseline_supervised`
|
||||
Expected: Compiles.
|
||||
|
||||
**Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add crates/ml/examples/train_baseline_supervised.rs
|
||||
git commit -m "feat(supervised): record learning rate and epoch duration metrics"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 8: Record hyperopt intra-trial metrics in both hyperopt binaries
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml/examples/hyperopt_baseline_rl.rs`
|
||||
- Modify: `crates/ml/examples/hyperopt_baseline_supervised.rs`
|
||||
|
||||
**Context:** Both binaries already have `use common::metrics::training_metrics;` and record hyperopt trial/best_objective. They each have a `let start = Instant::now();` at the beginning of each model's optimization function. The `elapsed` is computed after optimization completes. We need to add `elapsed_seconds` recording after optimization completes, before returning results.
|
||||
|
||||
**Step 1: Add elapsed_seconds to hyperopt_baseline_rl.rs**
|
||||
|
||||
In `run_dqn_hyperopt()`, after `training_metrics::set_hyperopt_best_objective("dqn", result.best_objective);` (~line 185), add:
|
||||
|
||||
```rust
|
||||
training_metrics::set_hyperopt_elapsed("dqn", elapsed);
|
||||
```
|
||||
|
||||
In `run_ppo_hyperopt()`, after `training_metrics::set_hyperopt_best_objective("ppo", result.best_objective);` (~line 254), add:
|
||||
|
||||
```rust
|
||||
training_metrics::set_hyperopt_elapsed("ppo", elapsed);
|
||||
```
|
||||
|
||||
**Step 2: Add elapsed_seconds to hyperopt_baseline_supervised.rs**
|
||||
|
||||
The supervised hyperopt binary has 8 model functions (TFT, Mamba2, Liquid, TGGN, TLOB, KAN, xLSTM, Diffusion). In each function, after the `training_metrics::set_hyperopt_best_objective(MODEL, result.best_objective);` line, add:
|
||||
|
||||
```rust
|
||||
training_metrics::set_hyperopt_elapsed(MODEL, elapsed);
|
||||
```
|
||||
|
||||
Where MODEL is the lowercase model name string ("tft", "mamba2", "liquid", "tggn", "tlob", "kan", "xlstm", "diffusion").
|
||||
|
||||
Specific locations:
|
||||
- `run_tft_hyperopt()`: after line 150
|
||||
- `run_mamba2_hyperopt()`: after line 208
|
||||
- `run_liquid_hyperopt()`: after line 267
|
||||
- `run_tggn_hyperopt()`: after line 326
|
||||
- `run_tlob_hyperopt()`: after line 385
|
||||
- `run_kan_hyperopt()`: after line 444
|
||||
- `run_xlstm_hyperopt()`: after line 503
|
||||
- `run_diffusion_hyperopt()`: after line 562
|
||||
|
||||
**Step 3: Verify compilation**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo check -p ml --example hyperopt_baseline_rl`
|
||||
Run: `SQLX_OFFLINE=true cargo check -p ml --example hyperopt_baseline_supervised`
|
||||
Expected: Both compile.
|
||||
|
||||
**Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add crates/ml/examples/hyperopt_baseline_rl.rs crates/ml/examples/hyperopt_baseline_supervised.rs
|
||||
git commit -m "feat(hyperopt): record elapsed_seconds metric in all hyperopt binaries"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 9: Full workspace check + clippy
|
||||
|
||||
**Files:** None (verification only)
|
||||
|
||||
**Step 1: Full workspace check**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo check --workspace`
|
||||
Expected: 0 errors.
|
||||
|
||||
**Step 2: Clippy**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo clippy --workspace -- -D warnings`
|
||||
Expected: 0 errors, 0 warnings.
|
||||
|
||||
**Step 3: Run all affected tests**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo test -p common --lib -- training_metrics`
|
||||
Run: `SQLX_OFFLINE=true cargo test -p monitoring_service --lib`
|
||||
Expected: All tests pass.
|
||||
|
||||
**Step 4: Commit (if any fixes needed)**
|
||||
|
||||
```bash
|
||||
git commit -am "fix: resolve clippy/test issues from metrics additions"
|
||||
```
|
||||
|
||||
Only create this commit if there were issues to fix. If everything passes, skip this step.
|
||||
Reference in New Issue
Block a user