foxhunt/docs/agent7_overfitting_integration_analysis.md

Agent 7: Overfitting Detection Integration Analysis

Executive Summary

Objective: Integrate ValidationMetrics::is_overfitting() into DQN hyperopt early stopping to detect and prune overfitting trials.

Status: ✅ Ready for implementation

Key Finding: Current DQN hyperopt already tracks train_loss and val_loss but doesn't use the overfitting detection logic available in validation_metrics.rs.

---

1. Current State Analysis

1.1 Available Infrastructure

ValidationMetrics (ml/src/trainers/validation_metrics.rs)

The ValidationMetrics struct provides comprehensive overfitting detection:

pub struct ValidationMetrics {
    pub epoch: usize,
    pub train_loss: f32,
    pub val_loss: f32,
    pub q_value_mean: f32,
    pub q_value_std: f32,
    pub action_distribution: [f32; 3],
    pub policy_entropy: f32,
    pub win_rate: f32,
    pub sharpe_ratio: f32,
    pub gradient_norm: f32,
}

impl ValidationMetrics {
    /// Check if model is overfitting
    ///
    /// Signals:
    /// 1. Train loss decreasing while validation loss increasing (5 epoch trend)
    /// 2. Train/val loss ratio > 2.0 (memorization)
    pub fn is_overfitting(&self, history: &[Self]) -> bool;
}

Early Stop Criteria Enum (lines 204-226):

pub enum EarlyStopCriteria {
    Overfitting,  // Calls is_overfitting() internally
    ValidationLossIncrease { patience: usize },
    ActionCollapse { hold_threshold: f32, patience: usize },
    EntropyCollapse { threshold: f32, patience: usize },
    QValueExplosion { threshold: f32 },
    GradientExplosion { threshold: f32 },
    All,  // Checks all criteria
}

1.2 DQN Hyperopt Current Implementation

DQNTrainer (ml/src/hyperopt/adapters/dqn.rs)

Early Stopping Fields (lines 618-621):

pub struct DQNTrainer {
    early_stopping_plateau_window: usize,  // Default: 5
    early_stopping_min_epochs: usize,      // Default: 1000 (effectively disabled)
    // ... other fields
}

Hyperparameters Config (lines 1784-1788):

let hyperparams = DQNHyperparameters {
    early_stopping_enabled: true,
    plateau_window: self.early_stopping_plateau_window,
    min_epochs_before_stopping: self.early_stopping_min_epochs,
    // ... other fields
};

Current Metrics Tracked (DQNMetrics struct):

• train_loss: f64
• val_loss: f64
• avg_q_value: f64
• gradient_norm: f64
• q_value_std: f64
• action_distribution: [buy%, sell%, hold%]

Problem: These metrics exist but is_overfitting() is NOT called!

1.3 DQN Trainer Implementation

Validation Loss Computation (trainer.rs:932-1000)

async fn compute_validation_loss(&mut self) -> Result<f64> {
    // Samples up to 1000 validation examples
    // Computes MSE loss on validation set
    // Returns single scalar loss value
}

Problem: Only returns scalar f64, no ValidationMetrics object created.

Training Loop Tracking (trainer.rs:2037-2125)

// Compute validation loss
let val_loss = self.compute_validation_loss().await?;

// Track histories
self.loss_history.push(avg_loss);          // train_loss
self.q_value_history.push(avg_q_value);
self.val_loss_history.push(val_loss);

Fields Available:

• self.loss_history (train loss per epoch)
• self.val_loss_history (validation loss per epoch)
• self.q_value_history (mean Q-values per epoch)

---

2. Integration Strategy

2.1 Minimal Integration (Recommended)

Goal: Add overfitting detection WITHOUT major refactoring.

Step 1: Import ValidationMetrics

// At top of dqn.rs
use crate::trainers::validation_metrics::{ValidationMetrics, EarlyStopCriteria};

Step 2: Build ValidationMetrics History in Hyperopt

After each epoch in train_with_params(), construct ValidationMetrics:

// After training completes
let mut val_metrics_history = Vec::new();

for epoch in 0..epochs_completed {
    let vm = ValidationMetrics::new(
        epoch,
        train_losses[epoch] as f32,
        val_losses[epoch] as f32,
        q_values[epoch] as f32,
        q_value_stds[epoch] as f32,
        action_dists[epoch],  // [buy%, sell%, hold%]
        policy_entropies[epoch] as f32,
        win_rates[epoch] as f32,
        sharpe_ratios[epoch] as f32,
        gradient_norms[epoch] as f32,
    );
    val_metrics_history.push(vm);
}

Step 3: Check Overfitting at End of Trial

// After training loop, before returning metrics
if let Some(latest_vm) = val_metrics_history.last() {
    let criteria = EarlyStopCriteria::Overfitting;

    if let Some(reason) = criteria.should_stop(&latest_vm, &val_metrics_history) {
        tracing::warn!("⚠️  Trial {} PRUNED: {}", current_trial, reason);

        // Return heavily penalized metrics
        return Ok(DQNMetrics {
            train_loss: latest_vm.train_loss as f64,
            val_loss: latest_vm.val_loss as f64,
            avg_episode_reward: -1000.0,  // Prune trial
            gradient_norm: latest_vm.gradient_norm as f64,
            q_value_std: latest_vm.q_value_std as f64,
            // ... other fields
        });
    }
}

2.2 Comprehensive Integration (Future Enhancement)

Goal: Refactor DQN trainer to emit ValidationMetrics natively.

Step 1: Add ValidationMetrics to DQNTrainer

pub struct DQNTrainer {
    // ... existing fields
    validation_metrics_history: Vec<ValidationMetrics>,
}

Step 2: Build ValidationMetrics Each Epoch

// In train() method after each epoch
let vm = ValidationMetrics::new(
    epoch,
    train_loss as f32,
    val_loss as f32,
    avg_q_value as f32,
    q_value_std as f32,
    action_distribution,
    policy_entropy as f32,
    win_rate as f32,
    sharpe_ratio as f32,
    gradient_norm as f32,
);

self.validation_metrics_history.push(vm);

// Check all early stop criteria
let criteria = EarlyStopCriteria::All;
if let Some(reason) = criteria.should_stop(&vm, &self.validation_metrics_history) {
    tracing::warn!("Early stopping triggered: {}", reason);
    break;  // Stop training
}

Step 3: Return ValidationMetrics from Trainer

pub fn get_validation_metrics(&self) -> &[ValidationMetrics] {
    &self.validation_metrics_history
}

---

3. Proposed Code Changes

3.1 File: ml/src/hyperopt/adapters/dqn.rs

Change 1: Add Import (after line 56)

use crate::trainers::validation_metrics::{ValidationMetrics, EarlyStopCriteria};

Change 2: Track Metrics During Training (in train_with_params)

Location: After line 1900 (where trainer.train().await completes)

// Get metrics from completed training
let trainer_metrics = trainer.get_training_metrics();

// Build ValidationMetrics history for overfitting detection
let mut val_metrics_history = Vec::new();

let epochs_completed = trainer_metrics.loss_history.len();
for epoch in 0..epochs_completed {
    // Extract action distribution from trainer
    let action_dist = trainer.get_action_distribution_at_epoch(epoch)
        .unwrap_or([0.33, 0.33, 0.34]);  // Fallback to uniform

    let policy_entropy = calculate_entropy(&action_dist);

    let vm = ValidationMetrics::new(
        epoch,
        trainer_metrics.loss_history[epoch] as f32,
        trainer_metrics.val_loss_history.get(epoch).copied().unwrap_or(0.0) as f32,
        trainer_metrics.q_value_history.get(epoch).copied().unwrap_or(0.0) as f32,
        trainer_metrics.q_value_std_history.get(epoch).copied().unwrap_or(1.0) as f32,
        action_dist,
        policy_entropy as f32,
        0.5,  // Default win_rate (can be computed from backtest)
        0.0,  // Default sharpe_ratio (from backtest_metrics if available)
        trainer_metrics.gradient_norm_history.get(epoch).copied().unwrap_or(0.0) as f32,
    );

    val_metrics_history.push(vm);
}

Change 3: Apply Overfitting Detection (before returning DQNMetrics)

// Check for overfitting using ValidationMetrics
if let Some(latest_vm) = val_metrics_history.last() {
    let criteria = EarlyStopCriteria::Overfitting;

    if let Some(reason) = criteria.should_stop(&latest_vm, &val_metrics_history) {
        tracing::warn!(
            "⚠️  Trial {} PRUNED (overfitting): {}",
            current_trial,
            reason
        );

        // Log overfitting detection
        write_training_log_dqn(
            &self.training_paths.logs_dir(),
            &format!("Trial PRUNED (overfitting): {}", reason),
        ).ok();

        // Return penalized metrics to prune this trial
        return Ok(DQNMetrics {
            train_loss: latest_vm.train_loss as f64,
            val_loss: latest_vm.val_loss as f64,
            avg_q_value: latest_vm.q_value_mean as f64,
            final_epsilon: trainer.get_epsilon().await.unwrap_or(0.0),
            epochs_completed: epochs_completed as u32,
            avg_episode_reward: -1000.0,  // Heavy penalty to prune trial
            buy_action_pct: latest_vm.action_distribution[0] as f64,
            sell_action_pct: latest_vm.action_distribution[1] as f64,
            hold_action_pct: latest_vm.action_distribution[2] as f64,
            gradient_norm: latest_vm.gradient_norm as f64,
            q_value_std: latest_vm.q_value_std as f64,
            backtest_metrics: None,  // No backtest for pruned trials
        });
    }
}

Helper Function: Calculate Entropy

/// Calculate Shannon entropy of action distribution
fn calculate_entropy(distribution: &[f32; 3]) -> f32 {
    distribution
        .iter()
        .filter(|&&p| p > 1e-8)  // Avoid log(0)
        .map(|&p| -p * p.log2())
        .sum()
}

---

4. Data Flow Diagram

┌─────────────────────────────────────────────────────────────┐
│ DQNTrainer (hyperopt/adapters/dqn.rs)                       │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  train_with_params(params)                                  │
│      │                                                       │
│      ├─> Create DQNHyperparameters                          │
│      │                                                       │
│      ├─> trainer.train().await                              │
│      │       │                                               │
│      │       ├─> FOR each epoch:                            │
│      │       │     ├─> Compute train_loss                   │
│      │       │     ├─> Compute val_loss                     │
│      │       │     ├─> Track q_values, gradients            │
│      │       │     └─> Store in histories                   │
│      │       │                                               │
│      │       └─> Return training complete                   │
│      │                                                       │
│      ├─> GET trainer metrics                                │
│      │                                                       │
│      ├─> BUILD ValidationMetrics history ◄──┐               │
│      │     FOR each epoch:                  │               │
│      │       ValidationMetrics::new(        │               │
│      │         epoch,                        │               │
│      │         train_loss[epoch],            │               │
│      │         val_loss[epoch],              │               │
│      │         q_value[epoch],               │               │
│      │         q_std[epoch],                 │               │
│      │         action_dist[epoch],           │               │
│      │         entropy[epoch],               │               │
│      │         win_rate, sharpe, grad_norm   │               │
│      │       )                                │               │
│      │                                        │               │
│      ├─> CHECK overfitting ────────────────┘               │
│      │     EarlyStopCriteria::Overfitting                   │
│      │       .should_stop(latest, history)                  │
│      │                                                       │
│      │     IF overfitting detected:                         │
│      │       ├─> Log pruning reason                         │
│      │       └─> Return penalized metrics                   │
│      │                                                       │
│      └─> Return final metrics                               │
│                                                              │
└─────────────────────────────────────────────────────────────┘

---

5. Overfitting Detection Logic

5.1 Signals Detected

Signal 1: Train/Val Divergence (5-epoch trend)

• Train loss monotonically decreasing
• Validation loss monotonically increasing
• Action: Prune trial immediately

Signal 2: High Train/Val Ratio

• train_loss / val_loss > 2.0
• Indicates severe memorization
• Action: Prune trial immediately

5.2 Detection Flow

pub fn is_overfitting(&self, history: &[Self]) -> bool {
    if history.len() < 5 {
        return false;  // Need 5 epochs minimum
    }

    let recent = &history[history.len()-5..];

    // Signal 1: Divergence check
    let train_decreasing = recent.windows(2)
        .all(|w| w[1].train_loss < w[0].train_loss);
    let val_increasing = recent.windows(2)
        .all(|w| w[1].val_loss > w[0].val_loss);

    if train_decreasing && val_increasing {
        return true;  // OVERFITTING DETECTED
    }

    // Signal 2: Ratio check
    if self.val_loss > 0.0 && self.train_loss / self.val_loss > 2.0 {
        return true;  // OVERFITTING DETECTED
    }

    false
}

---

6. Testing Strategy

6.1 Unit Test: Overfitting Detection

#[test]
fn test_hyperopt_overfitting_pruning() {
    // Create mock training history with overfitting pattern
    let mut val_metrics = vec![
        ValidationMetrics::new(0, 2.0, 2.0, 1.0, 0.1, [0.3, 0.3, 0.4], 0.5, 0.6, 1.8, 0.5),
        ValidationMetrics::new(1, 1.8, 2.1, 1.0, 0.1, [0.3, 0.3, 0.4], 0.5, 0.6, 1.8, 0.5),
        ValidationMetrics::new(2, 1.6, 2.2, 1.0, 0.1, [0.3, 0.3, 0.4], 0.5, 0.6, 1.8, 0.5),
        ValidationMetrics::new(3, 1.4, 2.3, 1.0, 0.1, [0.3, 0.3, 0.4], 0.5, 0.6, 1.8, 0.5),
        ValidationMetrics::new(4, 1.2, 2.4, 1.0, 0.1, [0.3, 0.3, 0.4], 0.5, 0.6, 1.8, 0.5),
    ];

    let latest = val_metrics.last().unwrap();
    assert!(latest.is_overfitting(&val_metrics), "Should detect overfitting");
}

6.2 Integration Test: Hyperopt Trial Pruning

#[tokio::test]
async fn test_hyperopt_prunes_overfitting_trials() {
    let trainer = DQNTrainer::new("test_data/", 20).unwrap();

    // Create params that cause overfitting (high LR, low regularization)
    let params = DQNParams {
        learning_rate: 1e-3,  // Very high LR
        batch_size: 32,
        gamma: 0.99,
        buffer_size: 10_000,
        hold_penalty_weight: 0.1,  // Low penalty (overfits to HOLD)
        // ... other params
    };

    let metrics = trainer.train_with_params(params).unwrap();

    // Should return penalized objective for overfitting trial
    assert!(metrics.avg_episode_reward < -500.0, "Overfitting trial should be pruned");
}

---

7. Expected Impact

7.1 Benefits

1. Early Trial Pruning: Stop overfitting trials before wasting 500 epochs

   • Current: 500 epochs × 30 trials = 15,000 epochs total
   • With pruning: ~10-15 trials pruned at epoch 20 → 7,500 epochs saved (50% speedup)

2. Better Final Models: Hyperopt won't select overfitting trials as "best"

   • Current: Trial #35 (Sharpe 1.207 → 2.612) was overfitting
   • With detection: Would be pruned at epoch ~50-100

3. Clearer Logs: Explicit overfitting warnings in hyperopt logs

   • ⚠️ Trial 12 PRUNED (overfitting): train/val ratio = 2.34

7.2 Risks

1. False Positives: May prune trials with legitimate train/val differences

   • Mitigation: Use 5-epoch window to avoid transient spikes

2. Metrics Collection Overhead: Building ValidationMetrics per epoch

   • Impact: Negligible (just struct construction, no computation)

---

8. Verification Checklist

• Import ValidationMetrics and EarlyStopCriteria in dqn.rs
• Add ValidationMetrics history construction in train_with_params()
• Add overfitting check before returning DQNMetrics
• Add calculate_entropy() helper function
• Write unit test for overfitting detection
• Run cargo test --package ml validation_metrics (should pass)
• Run hyperopt trial with known overfitting params
• Verify pruning message in logs
• Confirm penalized objective (-1000.0) returned
• Update hyperopt documentation

---

9. Open Questions

1. Should we also check other EarlyStopCriteria?

   • QValueExplosion, GradientExplosion, ActionCollapse, EntropyCollapse
   • Recommendation: Add in Wave 8 (comprehensive early stopping)

2. What threshold for train/val ratio?

   • Current: 2.0 (hardcoded in validation_metrics.rs)
   • Recommendation: Keep default, expose as tunable later

3. Should we track ValidationMetrics in DQNTrainer natively?

   • Recommendation: Phase 2 refactoring (comprehensive integration)

---

10. Next Steps

1. Agent 7 Implementation (THIS TASK):

   • Minimal integration (Strategy 2.1)
   • Add overfitting detection to hyperopt
   • Test with known overfitting params

2. Agent 8 (Future):

   • Refactor DQN trainer to emit ValidationMetrics natively
   • Add comprehensive EarlyStopCriteria::All checking

3. Agent 9 (Future):

   • Add ValidationMetrics to PPO, TFT, MAMBA-2 hyperopt
   • Unified validation across all trainers

---

Appendix A: Key File Locations

ml/src/trainers/validation_metrics.rs   # ValidationMetrics struct, is_overfitting()
ml/src/hyperopt/adapters/dqn.rs         # DQNTrainer, train_with_params()
ml/src/trainers/dqn/trainer.rs          # DQNTrainer::train(), compute_validation_loss()
ml/src/trainers/dqn/config.rs           # DQNHyperparameters
ml/src/trainers/dqn/statistics.rs       # FeatureStatistics, QValueStats

Appendix B: Metrics Mapping

ValidationMetrics Field	DQN Trainer Source
epoch	Loop index
train_loss	self.loss_history[epoch]
val_loss	self.val_loss_history[epoch]
q_value_mean	self.q_value_history[epoch]
q_value_std	Compute from Q-value distribution
action_distribution	Track [buy%, sell%, hold%] per epoch
policy_entropy	Calculate from action_distribution
win_rate	From backtest_metrics (if available)
sharpe_ratio	From backtest_metrics (if available)
gradient_norm	Track during training loop

---

Author: Agent 7 (Hive-Mind Swarm) Date: 2025-11-27 Status: Ready for Implementation