# Agent 7: Overfitting Detection Integration Analysis

## Executive Summary

**Objective**: Integrate `ValidationMetrics::is_overfitting()` into DQN hyperopt early stopping to detect and prune overfitting trials.

**Status**: ✅ Ready for implementation

**Key Finding**: Current DQN hyperopt already tracks `train_loss` and `val_loss` but doesn't use the overfitting detection logic available in `validation_metrics.rs`.

---

## 1. Current State Analysis

### 1.1 Available Infrastructure

#### ValidationMetrics (ml/src/trainers/validation_metrics.rs)

The `ValidationMetrics` struct provides comprehensive overfitting detection:

```rust
pub struct ValidationMetrics {
    pub epoch: usize,
    pub train_loss: f32,
    pub val_loss: f32,
    pub q_value_mean: f32,
    pub q_value_std: f32,
    pub action_distribution: [f32; 3],
    pub policy_entropy: f32,
    pub win_rate: f32,
    pub sharpe_ratio: f32,
    pub gradient_norm: f32,
}

impl ValidationMetrics {
    /// Check if model is overfitting
    ///
    /// Signals:
    /// 1. Train loss decreasing while validation loss increasing (5 epoch trend)
    /// 2. Train/val loss ratio > 2.0 (memorization)
    pub fn is_overfitting(&self, history: &[Self]) -> bool;
}
```

**Early Stop Criteria Enum** (lines 204-226):
```rust
pub enum EarlyStopCriteria {
    Overfitting,  // Calls is_overfitting() internally
    ValidationLossIncrease { patience: usize },
    ActionCollapse { hold_threshold: f32, patience: usize },
    EntropyCollapse { threshold: f32, patience: usize },
    QValueExplosion { threshold: f32 },
    GradientExplosion { threshold: f32 },
    All,  // Checks all criteria
}
```

### 1.2 DQN Hyperopt Current Implementation

#### DQNTrainer (ml/src/hyperopt/adapters/dqn.rs)

**Early Stopping Fields** (lines 618-621):
```rust
pub struct DQNTrainer {
    early_stopping_plateau_window: usize,  // Default: 5
    early_stopping_min_epochs: usize,      // Default: 1000 (effectively disabled)
    // ... other fields
}
```

**Hyperparameters Config** (lines 1784-1788):
```rust
let hyperparams = DQNHyperparameters {
    early_stopping_enabled: true,
    plateau_window: self.early_stopping_plateau_window,
    min_epochs_before_stopping: self.early_stopping_min_epochs,
    // ... other fields
};
```

**Current Metrics Tracked** (DQNMetrics struct):
- `train_loss: f64`
- `val_loss: f64`
- `avg_q_value: f64`
- `gradient_norm: f64`
- `q_value_std: f64`
- `action_distribution: [buy%, sell%, hold%]`

**Problem**: These metrics exist but `is_overfitting()` is NOT called!

### 1.3 DQN Trainer Implementation

#### Validation Loss Computation (trainer.rs:932-1000)

```rust
async fn compute_validation_loss(&mut self) -> Result<f64> {
    // Samples up to 1000 validation examples
    // Computes MSE loss on validation set
    // Returns single scalar loss value
}
```

**Problem**: Only returns scalar `f64`, no ValidationMetrics object created.

#### Training Loop Tracking (trainer.rs:2037-2125)

```rust
// Compute validation loss
let val_loss = self.compute_validation_loss().await?;

// Track histories
self.loss_history.push(avg_loss);          // train_loss
self.q_value_history.push(avg_q_value);
self.val_loss_history.push(val_loss);
```

**Fields Available**:
- `self.loss_history` (train loss per epoch)
- `self.val_loss_history` (validation loss per epoch)
- `self.q_value_history` (mean Q-values per epoch)

---

## 2. Integration Strategy

### 2.1 Minimal Integration (Recommended)

**Goal**: Add overfitting detection WITHOUT major refactoring.

#### Step 1: Import ValidationMetrics

```rust
// At top of dqn.rs
use crate::trainers::validation_metrics::{ValidationMetrics, EarlyStopCriteria};
```

#### Step 2: Build ValidationMetrics History in Hyperopt

After each epoch in `train_with_params()`, construct ValidationMetrics:

```rust
// After training completes
let mut val_metrics_history = Vec::new();

for epoch in 0..epochs_completed {
    let vm = ValidationMetrics::new(
        epoch,
        train_losses[epoch] as f32,
        val_losses[epoch] as f32,
        q_values[epoch] as f32,
        q_value_stds[epoch] as f32,
        action_dists[epoch],  // [buy%, sell%, hold%]
        policy_entropies[epoch] as f32,
        win_rates[epoch] as f32,
        sharpe_ratios[epoch] as f32,
        gradient_norms[epoch] as f32,
    );
    val_metrics_history.push(vm);
}
```

#### Step 3: Check Overfitting at End of Trial

```rust
// After training loop, before returning metrics
if let Some(latest_vm) = val_metrics_history.last() {
    let criteria = EarlyStopCriteria::Overfitting;

    if let Some(reason) = criteria.should_stop(&latest_vm, &val_metrics_history) {
        tracing::warn!("⚠️  Trial {} PRUNED: {}", current_trial, reason);

        // Return heavily penalized metrics
        return Ok(DQNMetrics {
            train_loss: latest_vm.train_loss as f64,
            val_loss: latest_vm.val_loss as f64,
            avg_episode_reward: -1000.0,  // Prune trial
            gradient_norm: latest_vm.gradient_norm as f64,
            q_value_std: latest_vm.q_value_std as f64,
            // ... other fields
        });
    }
}
```

### 2.2 Comprehensive Integration (Future Enhancement)

**Goal**: Refactor DQN trainer to emit ValidationMetrics natively.

#### Step 1: Add ValidationMetrics to DQNTrainer

```rust
pub struct DQNTrainer {
    // ... existing fields
    validation_metrics_history: Vec<ValidationMetrics>,
}
```

#### Step 2: Build ValidationMetrics Each Epoch

```rust
// In train() method after each epoch
let vm = ValidationMetrics::new(
    epoch,
    train_loss as f32,
    val_loss as f32,
    avg_q_value as f32,
    q_value_std as f32,
    action_distribution,
    policy_entropy as f32,
    win_rate as f32,
    sharpe_ratio as f32,
    gradient_norm as f32,
);

self.validation_metrics_history.push(vm);

// Check all early stop criteria
let criteria = EarlyStopCriteria::All;
if let Some(reason) = criteria.should_stop(&vm, &self.validation_metrics_history) {
    tracing::warn!("Early stopping triggered: {}", reason);
    break;  // Stop training
}
```

#### Step 3: Return ValidationMetrics from Trainer

```rust
pub fn get_validation_metrics(&self) -> &[ValidationMetrics] {
    &self.validation_metrics_history
}
```

---

## 3. Proposed Code Changes

### 3.1 File: ml/src/hyperopt/adapters/dqn.rs

#### Change 1: Add Import (after line 56)

```rust
use crate::trainers::validation_metrics::{ValidationMetrics, EarlyStopCriteria};
```

#### Change 2: Track Metrics During Training (in train_with_params)

**Location**: After line 1900 (where trainer.train().await completes)

```rust
// Get metrics from completed training
let trainer_metrics = trainer.get_training_metrics();

// Build ValidationMetrics history for overfitting detection
let mut val_metrics_history = Vec::new();

let epochs_completed = trainer_metrics.loss_history.len();
for epoch in 0..epochs_completed {
    // Extract action distribution from trainer
    let action_dist = trainer.get_action_distribution_at_epoch(epoch)
        .unwrap_or([0.33, 0.33, 0.34]);  // Fallback to uniform

    let policy_entropy = calculate_entropy(&action_dist);

    let vm = ValidationMetrics::new(
        epoch,
        trainer_metrics.loss_history[epoch] as f32,
        trainer_metrics.val_loss_history.get(epoch).copied().unwrap_or(0.0) as f32,
        trainer_metrics.q_value_history.get(epoch).copied().unwrap_or(0.0) as f32,
        trainer_metrics.q_value_std_history.get(epoch).copied().unwrap_or(1.0) as f32,
        action_dist,
        policy_entropy as f32,
        0.5,  // Default win_rate (can be computed from backtest)
        0.0,  // Default sharpe_ratio (from backtest_metrics if available)
        trainer_metrics.gradient_norm_history.get(epoch).copied().unwrap_or(0.0) as f32,
    );

    val_metrics_history.push(vm);
}
```

#### Change 3: Apply Overfitting Detection (before returning DQNMetrics)

```rust
// Check for overfitting using ValidationMetrics
if let Some(latest_vm) = val_metrics_history.last() {
    let criteria = EarlyStopCriteria::Overfitting;

    if let Some(reason) = criteria.should_stop(&latest_vm, &val_metrics_history) {
        tracing::warn!(
            "⚠️  Trial {} PRUNED (overfitting): {}",
            current_trial,
            reason
        );

        // Log overfitting detection
        write_training_log_dqn(
            &self.training_paths.logs_dir(),
            &format!("Trial PRUNED (overfitting): {}", reason),
        ).ok();

        // Return penalized metrics to prune this trial
        return Ok(DQNMetrics {
            train_loss: latest_vm.train_loss as f64,
            val_loss: latest_vm.val_loss as f64,
            avg_q_value: latest_vm.q_value_mean as f64,
            final_epsilon: trainer.get_epsilon().await.unwrap_or(0.0),
            epochs_completed: epochs_completed as u32,
            avg_episode_reward: -1000.0,  // Heavy penalty to prune trial
            buy_action_pct: latest_vm.action_distribution[0] as f64,
            sell_action_pct: latest_vm.action_distribution[1] as f64,
            hold_action_pct: latest_vm.action_distribution[2] as f64,
            gradient_norm: latest_vm.gradient_norm as f64,
            q_value_std: latest_vm.q_value_std as f64,
            backtest_metrics: None,  // No backtest for pruned trials
        });
    }
}
```

#### Helper Function: Calculate Entropy

```rust
/// Calculate Shannon entropy of action distribution
fn calculate_entropy(distribution: &[f32; 3]) -> f32 {
    distribution
        .iter()
        .filter(|&&p| p > 1e-8)  // Avoid log(0)
        .map(|&p| -p * p.log2())
        .sum()
}
```

---

## 4. Data Flow Diagram

```
┌─────────────────────────────────────────────────────────────┐
│ DQNTrainer (hyperopt/adapters/dqn.rs)                       │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  train_with_params(params)                                  │
│      │                                                       │
│      ├─> Create DQNHyperparameters                          │
│      │                                                       │
│      ├─> trainer.train().await                              │
│      │       │                                               │
│      │       ├─> FOR each epoch:                            │
│      │       │     ├─> Compute train_loss                   │
│      │       │     ├─> Compute val_loss                     │
│      │       │     ├─> Track q_values, gradients            │
│      │       │     └─> Store in histories                   │
│      │       │                                               │
│      │       └─> Return training complete                   │
│      │                                                       │
│      ├─> GET trainer metrics                                │
│      │                                                       │
│      ├─> BUILD ValidationMetrics history ◄──┐               │
│      │     FOR each epoch:                  │               │
│      │       ValidationMetrics::new(        │               │
│      │         epoch,                        │               │
│      │         train_loss[epoch],            │               │
│      │         val_loss[epoch],              │               │
│      │         q_value[epoch],               │               │
│      │         q_std[epoch],                 │               │
│      │         action_dist[epoch],           │               │
│      │         entropy[epoch],               │               │
│      │         win_rate, sharpe, grad_norm   │               │
│      │       )                                │               │
│      │                                        │               │
│      ├─> CHECK overfitting ────────────────┘               │
│      │     EarlyStopCriteria::Overfitting                   │
│      │       .should_stop(latest, history)                  │
│      │                                                       │
│      │     IF overfitting detected:                         │
│      │       ├─> Log pruning reason                         │
│      │       └─> Return penalized metrics                   │
│      │                                                       │
│      └─> Return final metrics                               │
│                                                              │
└─────────────────────────────────────────────────────────────┘
```

---

## 5. Overfitting Detection Logic

### 5.1 Signals Detected

**Signal 1: Train/Val Divergence** (5-epoch trend)
- Train loss monotonically decreasing
- Validation loss monotonically increasing
- **Action**: Prune trial immediately

**Signal 2: High Train/Val Ratio**
- `train_loss / val_loss > 2.0`
- Indicates severe memorization
- **Action**: Prune trial immediately

### 5.2 Detection Flow

```rust
pub fn is_overfitting(&self, history: &[Self]) -> bool {
    if history.len() < 5 {
        return false;  // Need 5 epochs minimum
    }

    let recent = &history[history.len()-5..];

    // Signal 1: Divergence check
    let train_decreasing = recent.windows(2)
        .all(|w| w[1].train_loss < w[0].train_loss);
    let val_increasing = recent.windows(2)
        .all(|w| w[1].val_loss > w[0].val_loss);

    if train_decreasing && val_increasing {
        return true;  // OVERFITTING DETECTED
    }

    // Signal 2: Ratio check
    if self.val_loss > 0.0 && self.train_loss / self.val_loss > 2.0 {
        return true;  // OVERFITTING DETECTED
    }

    false
}
```

---

## 6. Testing Strategy

### 6.1 Unit Test: Overfitting Detection

```rust
#[test]
fn test_hyperopt_overfitting_pruning() {
    // Create mock training history with overfitting pattern
    let mut val_metrics = vec![
        ValidationMetrics::new(0, 2.0, 2.0, 1.0, 0.1, [0.3, 0.3, 0.4], 0.5, 0.6, 1.8, 0.5),
        ValidationMetrics::new(1, 1.8, 2.1, 1.0, 0.1, [0.3, 0.3, 0.4], 0.5, 0.6, 1.8, 0.5),
        ValidationMetrics::new(2, 1.6, 2.2, 1.0, 0.1, [0.3, 0.3, 0.4], 0.5, 0.6, 1.8, 0.5),
        ValidationMetrics::new(3, 1.4, 2.3, 1.0, 0.1, [0.3, 0.3, 0.4], 0.5, 0.6, 1.8, 0.5),
        ValidationMetrics::new(4, 1.2, 2.4, 1.0, 0.1, [0.3, 0.3, 0.4], 0.5, 0.6, 1.8, 0.5),
    ];

    let latest = val_metrics.last().unwrap();
    assert!(latest.is_overfitting(&val_metrics), "Should detect overfitting");
}
```

### 6.2 Integration Test: Hyperopt Trial Pruning

```rust
#[tokio::test]
async fn test_hyperopt_prunes_overfitting_trials() {
    let trainer = DQNTrainer::new("test_data/", 20).unwrap();

    // Create params that cause overfitting (high LR, low regularization)
    let params = DQNParams {
        learning_rate: 1e-3,  // Very high LR
        batch_size: 32,
        gamma: 0.99,
        buffer_size: 10_000,
        hold_penalty_weight: 0.1,  // Low penalty (overfits to HOLD)
        // ... other params
    };

    let metrics = trainer.train_with_params(params).unwrap();

    // Should return penalized objective for overfitting trial
    assert!(metrics.avg_episode_reward < -500.0, "Overfitting trial should be pruned");
}
```

---

## 7. Expected Impact

### 7.1 Benefits

1. **Early Trial Pruning**: Stop overfitting trials before wasting 500 epochs
   - Current: 500 epochs × 30 trials = 15,000 epochs total
   - With pruning: ~10-15 trials pruned at epoch 20 → 7,500 epochs saved (50% speedup)

2. **Better Final Models**: Hyperopt won't select overfitting trials as "best"
   - Current: Trial #35 (Sharpe 1.207 → 2.612) was overfitting
   - With detection: Would be pruned at epoch ~50-100

3. **Clearer Logs**: Explicit overfitting warnings in hyperopt logs
   - `⚠️  Trial 12 PRUNED (overfitting): train/val ratio = 2.34`

### 7.2 Risks

1. **False Positives**: May prune trials with legitimate train/val differences
   - Mitigation: Use 5-epoch window to avoid transient spikes

2. **Metrics Collection Overhead**: Building ValidationMetrics per epoch
   - Impact: Negligible (just struct construction, no computation)

---

## 8. Verification Checklist

- [ ] Import ValidationMetrics and EarlyStopCriteria in dqn.rs
- [ ] Add ValidationMetrics history construction in train_with_params()
- [ ] Add overfitting check before returning DQNMetrics
- [ ] Add calculate_entropy() helper function
- [ ] Write unit test for overfitting detection
- [ ] Run `cargo test --package ml validation_metrics` (should pass)
- [ ] Run hyperopt trial with known overfitting params
- [ ] Verify pruning message in logs
- [ ] Confirm penalized objective (-1000.0) returned
- [ ] Update hyperopt documentation

---

## 9. Open Questions

1. **Should we also check other EarlyStopCriteria?**
   - QValueExplosion, GradientExplosion, ActionCollapse, EntropyCollapse
   - Recommendation: Add in Wave 8 (comprehensive early stopping)

2. **What threshold for train/val ratio?**
   - Current: 2.0 (hardcoded in validation_metrics.rs)
   - Recommendation: Keep default, expose as tunable later

3. **Should we track ValidationMetrics in DQNTrainer natively?**
   - Recommendation: Phase 2 refactoring (comprehensive integration)

---

## 10. Next Steps

1. **Agent 7 Implementation** (THIS TASK):
   - Minimal integration (Strategy 2.1)
   - Add overfitting detection to hyperopt
   - Test with known overfitting params

2. **Agent 8** (Future):
   - Refactor DQN trainer to emit ValidationMetrics natively
   - Add comprehensive EarlyStopCriteria::All checking

3. **Agent 9** (Future):
   - Add ValidationMetrics to PPO, TFT, MAMBA-2 hyperopt
   - Unified validation across all trainers

---

## Appendix A: Key File Locations

```
ml/src/trainers/validation_metrics.rs   # ValidationMetrics struct, is_overfitting()
ml/src/hyperopt/adapters/dqn.rs         # DQNTrainer, train_with_params()
ml/src/trainers/dqn/trainer.rs          # DQNTrainer::train(), compute_validation_loss()
ml/src/trainers/dqn/config.rs           # DQNHyperparameters
ml/src/trainers/dqn/statistics.rs       # FeatureStatistics, QValueStats
```

## Appendix B: Metrics Mapping

| ValidationMetrics Field | DQN Trainer Source                     |
|-------------------------|----------------------------------------|
| `epoch`                 | Loop index                             |
| `train_loss`            | `self.loss_history[epoch]`             |
| `val_loss`              | `self.val_loss_history[epoch]`         |
| `q_value_mean`          | `self.q_value_history[epoch]`          |
| `q_value_std`           | Compute from Q-value distribution      |
| `action_distribution`   | Track [buy%, sell%, hold%] per epoch   |
| `policy_entropy`        | Calculate from action_distribution     |
| `win_rate`              | From backtest_metrics (if available)   |
| `sharpe_ratio`          | From backtest_metrics (if available)   |
| `gradient_norm`         | Track during training loop             |

---

**Author**: Agent 7 (Hive-Mind Swarm)
**Date**: 2025-11-27
**Status**: Ready for Implementation