9105921f8d
Two-tier system: 12 always-on gauges (RL diagnostics, gradient health, hyperopt intra-trial) + 5 opt-in verbose metrics gated behind FOXHUNT_VERBOSE_METRICS=1. Extends proto, monitoring service, and CLI. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
152 lines
6.3 KiB
Markdown
152 lines
6.3 KiB
Markdown
# Additional Training Metrics Design
|
|
|
|
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
|
|
|
**Goal:** Add 17 new Prometheus metrics (12 always-on gauges + 5 opt-in verbose) to provide full training diagnostics and production monitoring visibility.
|
|
|
|
**Architecture:** Two-tier metrics system. Tier 1 metrics are zero-cost atomic gauge writes recorded in every trainer's epoch loop. Tier 2 metrics (histograms and per-layer stats) are gated behind `FOXHUNT_VERBOSE_METRICS=1` and disabled by default. All metrics flow through the existing Prometheus → monitoring_service → gRPC → fxt CLI pipeline.
|
|
|
|
---
|
|
|
|
## Current State
|
|
|
|
- 24 Prometheus metrics defined in `crates/common/src/metrics/training_metrics.rs`
|
|
- ~20 actively recorded across 5 training binaries
|
|
- Monitoring service queries all `foxhunt_training_*` and `foxhunt_hyperopt_*` via wildcard
|
|
- Proto has TrainingSession (19 fields) and GpuSnapshot (5 fields)
|
|
|
|
## Tier 1: Always-On Gauges (12 new)
|
|
|
|
### RL Diagnostics (6)
|
|
|
|
| Metric Name | Labels | Source | Purpose |
|
|
|---|---|---|---|
|
|
| `foxhunt_training_q_value_mean` | model, fold | DQN epoch loop | Detect value overestimation |
|
|
| `foxhunt_training_q_value_max` | model, fold | DQN epoch loop | Detect divergence (exploding Q-values) |
|
|
| `foxhunt_training_policy_entropy` | model, fold | PPO epoch loop | Detect premature convergence (entropy → 0) |
|
|
| `foxhunt_training_kl_divergence` | model, fold | PPO epoch loop | Detect unstable policy updates |
|
|
| `foxhunt_training_advantage_mean` | model, fold | PPO epoch loop | Detect advantage bias (should be ~0) |
|
|
| `foxhunt_training_replay_buffer_size` | model, fold | DQN epoch loop | Monitor replay buffer occupancy |
|
|
|
|
### Gradient & Training Health (3)
|
|
|
|
| Metric Name | Labels | Source | Purpose |
|
|
|---|---|---|---|
|
|
| `foxhunt_training_gradient_norm` | model, fold | All trainers post-backward | Trend before gradient explosion |
|
|
| `foxhunt_training_learning_rate` | model, fold | All trainers per epoch | Correlate LR schedule with loss plateaus |
|
|
| `foxhunt_training_epoch_duration_seconds` | model, fold | All trainers | Detect performance regressions |
|
|
|
|
### Hyperopt Intra-Trial Progress (3)
|
|
|
|
| Metric Name | Labels | Source | Purpose |
|
|
|---|---|---|---|
|
|
| `foxhunt_hyperopt_trial_epoch` | model | Hyperopt adapters | Current epoch within active trial |
|
|
| `foxhunt_hyperopt_trial_best_loss` | model | Hyperopt adapters | Best loss in current trial |
|
|
| `foxhunt_hyperopt_elapsed_seconds` | model | Hyperopt binaries | Wall-clock time since start |
|
|
|
|
## Tier 2: Opt-In Verbose Metrics (5 new, `FOXHUNT_VERBOSE_METRICS=1`)
|
|
|
|
| Metric Name | Type | Labels | Purpose |
|
|
|---|---|---|---|
|
|
| `foxhunt_training_batch_loss` | Histogram | model, fold | Per-batch loss distribution |
|
|
| `foxhunt_training_gradient_norm_per_layer` | Histogram | model, fold, layer | Per-layer gradient health |
|
|
| `foxhunt_training_replay_priority_mean` | Gauge | model, fold | PER staleness detection |
|
|
| `foxhunt_training_advantage_std` | Gauge | model, fold | Advantage normalization quality |
|
|
| `foxhunt_training_value_explained_variance` | Gauge | model, fold | Value function prediction quality |
|
|
|
|
### Gating Mechanism
|
|
|
|
```rust
|
|
// In training_metrics.rs
|
|
static VERBOSE: OnceLock<bool> = OnceLock::new();
|
|
|
|
pub fn verbose_enabled() -> bool {
|
|
*VERBOSE.get_or_init(|| std::env::var("FOXHUNT_VERBOSE_METRICS").is_ok())
|
|
}
|
|
```
|
|
|
|
Recording calls for Tier 2 metrics check `verbose_enabled()` and return early if disabled.
|
|
|
|
## Proto Changes (monitoring.proto)
|
|
|
|
Add to `TrainingSession`:
|
|
|
|
```protobuf
|
|
// RL diagnostics
|
|
float q_value_mean = 20;
|
|
float q_value_max = 21;
|
|
float policy_entropy = 22;
|
|
float kl_divergence = 23;
|
|
float advantage_mean = 24;
|
|
uint32 replay_buffer_size = 25;
|
|
|
|
// Gradient & training health
|
|
float gradient_norm = 26;
|
|
float learning_rate = 27;
|
|
float epoch_duration_seconds = 28;
|
|
|
|
// Hyperopt intra-trial
|
|
uint32 hyperopt_trial_epoch = 29;
|
|
float hyperopt_trial_best_loss = 30;
|
|
float hyperopt_elapsed_seconds = 31;
|
|
```
|
|
|
|
## Service Changes
|
|
|
|
### monitoring_service/src/service.rs
|
|
|
|
Extend `group_into_sessions` match arms for all 12 new metric names → proto fields. No new Prometheus queries needed (wildcard already covers `foxhunt_training_*` and `foxhunt_hyperopt_*`).
|
|
|
|
### monitoring_service/src/prometheus_client.rs
|
|
|
|
No changes needed.
|
|
|
|
## CLI Changes (fxt train monitor)
|
|
|
|
### Main table: add gradient_norm column
|
|
|
|
```
|
|
Model Fold Epoch Loss Val Loss Batch/s Grad Norm Eval Acc
|
|
----------------------------------------------------------------------
|
|
dqn 0 25 0.0234 0.0312 148.2 1.23 -
|
|
ppo 0 12 0.0156 0.0089 92.1 0.87 -
|
|
```
|
|
|
|
### RL diagnostics section (only when non-zero)
|
|
|
|
```
|
|
RL Diagnostics:
|
|
dqn: Q-mean=12.3 Q-max=45.6 | buffer=50000/100000
|
|
ppo: entropy=1.23 KL=0.008 adv-mean=0.001
|
|
```
|
|
|
|
### Hyperopt intra-trial (inline enhancement)
|
|
|
|
```
|
|
Hyperopt (dqn): trial 3/20 (epoch 12, loss 0.042) | best Sharpe 1.23 | 0 failures
|
|
```
|
|
|
|
## Existing Gap Fills
|
|
|
|
Wire up eval metrics (accuracy, precision, recall, F1) in DQN and PPO trainers. These proto fields already exist but are never populated for RL models.
|
|
|
|
## Testing
|
|
|
|
- Extend `training_metrics.rs` unit tests for new registrations + verbose gating
|
|
- Extend `service.rs` `group_into_sessions` tests with new metric → proto field mappings
|
|
- Full pipeline tested on next CI build with new training binaries
|
|
|
|
## Files Modified
|
|
|
|
| File | Change |
|
|
|---|---|
|
|
| `crates/common/src/metrics/training_metrics.rs` | +12 gauges, +5 verbose metrics, verbose gate |
|
|
| `services/monitoring_service/proto/monitoring.proto` | +12 proto fields |
|
|
| `services/monitoring_service/src/service.rs` | +12 match arms in group_into_sessions |
|
|
| `bin/fxt/src/commands/train/monitor.rs` | Enhanced display (grad norm col, RL section, hyperopt detail) |
|
|
| `crates/ml/src/trainers/dqn/trainer.rs` | Record Q-values, gradient norm, LR, epoch duration, eval metrics, buffer size |
|
|
| `crates/ml/src/trainers/ppo.rs` | Record entropy, KL, advantage, gradient norm, LR, epoch duration, eval metrics |
|
|
| `crates/ml/examples/train_baseline_supervised.rs` | Record gradient norm, LR, epoch duration |
|
|
| `crates/ml/examples/hyperopt_baseline_rl.rs` | Record trial_epoch, trial_best_loss, elapsed_seconds |
|
|
| `crates/ml/examples/hyperopt_baseline_supervised.rs` | Record trial_epoch, trial_best_loss, elapsed_seconds |
|