foxhunt/ML_CHECKPOINT_STATUS_MATRIX.md

# ML Checkpoint Status Matrix

**Generated**: 2025-11-02
**Analysis Scope**: Hyperopt adapters for all 4 ML models (DQN, PPO, TFT, MAMBA-2)
**Investigation**: Comprehensive code audit of checkpoint saving implementations

---

## Executive Summary

| Model | Hyperopt Checkpoints | Training Checkpoints | Status | Priority | Fix Effort |
|-------|---------------------|---------------------|--------|----------|-----------|
| **DQN**   | ❌ **MISSING** (no-op callback) | ✅ Working | **P0 CRITICAL** | 15-30 min |
| **PPO**   | ❌ **MISSING** (no save calls) | ✅ Working | **P1 HIGH** | 15-30 min |
| **TFT**   | ❌ **MEMORY ONLY** (not persisted) | ✅ Working | **P2 MEDIUM** | 1-2 hours |
| **MAMBA-2** | ✅ **WORKING** | ✅ Working | **CERTIFIED** | N/A |

**Key Findings**:
- **MAMBA-2** is the ONLY model with working hyperopt checkpoint saving
- **DQN** explicitly disables checkpoints with no-op callback (lines 667-670, 675-678, 688-691, 696-699)
- **PPO** has no checkpoint saving code in hyperopt adapter
- **TFT** uses `MemoryStorage` - checkpoints exist in RAM but are NOT persisted to disk (line 417)

---

## Detailed Analysis by Model

### 1. DQN - CRITICAL BUG ❌

**File**: `/home/jgrusewski/Work/foxhunt/ml/src/hyperopt/adapters/dqn.rs`

**Status**: ❌ **CHECKPOINT SAVING DISABLED**

**Evidence**:

```rust
// Lines 664-701: DQN training with explicit no-op checkpoint callback
if is_parquet_file {
    info!("Training DQN with parquet file: {}", data_path_str);
    handle.block_on(
        internal_trainer.train_from_parquet(data_path_str, |_epoch, _data, _is_final| {
            // No-op checkpoint callback for hyperopt trials
            Ok("skipped".to_string())
        }),
    )
} else {
    info!("Training DQN with DBN directory: {}", data_path_str);
    handle.block_on(
        internal_trainer.train(data_path_str, |_epoch, _data, _is_final| {
            // No-op checkpoint callback for hyperopt trials
            Ok("skipped".to_string())
        }),
    )
}
```

**Root Cause**: Checkpoint callback is intentionally stubbed out with comment "No-op checkpoint callback for hyperopt trials"

**Impact**:
- **CRITICAL**: All DQN hyperopt runs lose checkpoint history
- Cannot resume interrupted trials
- Best model from each trial is lost
- Must retrain from scratch if pod terminates

**Fix Required**:
```rust
// BEFORE (lines 667-670):
|_epoch, _data, _is_final| {
    // No-op checkpoint callback for hyperopt trials
    Ok("skipped".to_string())
}

// AFTER:
|epoch, data, is_final| {
    if is_final || epoch % 10 == 0 {
        let checkpoint_path = self.training_paths.checkpoints_dir()
            .join(format!("dqn_epoch_{}.safetensors", epoch));

        internal_trainer.save_checkpoint(&checkpoint_path)
            .map(|_| checkpoint_path.to_string_lossy().to_string())
            .map_err(|e| format!("Checkpoint save failed: {}", e))
    } else {
        Ok("skipped".to_string())
    }
}
```

**Affected Lines**: 667-670, 675-678, 688-691, 696-699
**Priority**: **P0 CRITICAL**
**Effort**: 15-30 minutes

---

### 2. PPO - HIGH PRIORITY ❌

**File**: `/home/jgrusewski/Work/foxhunt/ml/src/hyperopt/adapters/ppo.rs`

**Status**: ❌ **CHECKPOINT SAVING NOT IMPLEMENTED**

**Evidence**:

```rust
// Lines 428-448: PPO training loop - NO checkpoint saving
for _batch_idx in 0..num_batches {
    // Generate trajectories from real market data
    let mut trajectory_batch = self
        .generate_trajectories_from_data(train_data, 64)
        .map_err(|e| {
            MLError::TrainingError(format!("Failed to generate trajectories: {}", e))
        })?;

    // Update PPO with trajectory batch
    let (policy_loss, value_loss) = ppo_agent
        .update(&mut trajectory_batch)
        .map_err(|e| MLError::TrainingError(format!("PPO update failed: {}", e)))?;

    total_policy_loss += policy_loss as f64;
    total_value_loss += value_loss as f64;

    // Calculate average reward for this batch
    let batch_reward: f32 = trajectory_batch.rewards.iter().sum();
    total_reward += batch_reward as f64 / 64.0;
}

// Lines 501-515: Cleanup section - NO checkpoint saving before cleanup
info!("Cleaning up resources...");
drop(ppo_agent);
drop(val_trajectory_batch);
```

**Root Cause**: PPO hyperopt adapter never calls any checkpoint saving methods

**Impact**:
- **HIGH**: All PPO hyperopt trials lose model state
- Cannot resume interrupted trials
- Best hyperparameters found but model weights lost
- Wasted GPU time re-running successful trials

**Fix Required**:
```rust
// ADD after line 447 (inside training loop):
if _batch_idx % 10 == 0 || _batch_idx == num_batches - 1 {
    let checkpoint_path = self.training_paths.checkpoints_dir()
        .join(format!("ppo_batch_{}.safetensors", _batch_idx));

    ppo_agent.save_checkpoint(&checkpoint_path)
        .map_err(|e| MLError::TrainingError(format!("Checkpoint save failed: {}", e)))?;

    info!("Saved checkpoint: {:?}", checkpoint_path);
}
```

**Affected Lines**: 428-448 (training loop), 501-515 (cleanup)
**Priority**: **P1 HIGH**
**Effort**: 15-30 minutes

---

### 3. TFT - MEDIUM PRIORITY ⚠️

**File**: `/home/jgrusewski/Work/foxhunt/ml/src/hyperopt/adapters/tft.rs`

**Status**: ⚠️ **CHECKPOINT SAVING TO MEMORY ONLY (NOT PERSISTED)**

**Evidence**:

```rust
// Lines 413-427: TFT trainer configuration with checkpoint_dir
let trainer_config = TFTTrainerConfig {
    // ... other config ...

    // Checkpointing (use configured training paths)
    checkpoint_dir: self.training_paths.checkpoints_dir().to_string_lossy().to_string(),
};

// Line 417-419: MemoryStorage used instead of disk storage
let checkpoint_storage = std::sync::Arc::new(crate::checkpoint::MemoryStorage::new());
let mut trainer = RealTFTTrainer::new(trainer_config, checkpoint_storage)
    .map_err(|e| MLError::ModelError(format!("Failed to create TFT trainer: {}", e)))?;
```

**Root Cause**: TFT uses `MemoryStorage` for checkpoints, which keeps them in RAM only

**Impact**:
- **MEDIUM**: Checkpoints are created but lost when pod terminates
- Cannot resume after OOM or pod timeout
- Checkpoints work for in-memory early stopping but not persistence
- Debugging requires re-running entire trial

**Fix Required**:
```rust
// BEFORE (line 417):
let checkpoint_storage = std::sync::Arc::new(crate::checkpoint::MemoryStorage::new());

// AFTER:
let checkpoint_storage = std::sync::Arc::new(
    crate::checkpoint::FileSystemStorage::new(&self.training_paths.checkpoints_dir())
        .map_err(|e| MLError::ModelError(format!("Failed to create checkpoint storage: {}", e)))?
);
```

**Affected Lines**: 413 (checkpoint_dir config), 417-419 (MemoryStorage instantiation)
**Priority**: **P2 MEDIUM**
**Effort**: 1-2 hours (requires implementing FileSystemStorage adapter)

---

### 4. MAMBA-2 - PRODUCTION CERTIFIED ✅

**File**: `/home/jgrusewski/Work/foxhunt/ml/src/hyperopt/adapters/mamba2.rs`

**Status**: ✅ **CHECKPOINT SAVING WORKING**

**Evidence**:

```rust
// Lines 880-898: MAMBA-2 training with checkpoint directory passed
let training_result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
    if self.async_loading {
        info!("Using async data loading (prefetch={})", self.prefetch_count);
        tokio::runtime::Runtime::new()
            .unwrap()
            .block_on(self.train_with_async_loading(
                &mut model,
                &train_data,
                &val_data,
                self.epochs,
                params.batch_size,
                Some(&self.training_paths.checkpoints_dir()),  // ✅ CHECKPOINT DIR PASSED
            ))
    } else {
        info!("Using synchronous data loading");
        tokio::runtime::Runtime::new()
            .unwrap()
            .block_on(model.train(&train_data, &val_data, self.epochs, Some(&self.training_paths.checkpoints_dir())))  // ✅ CHECKPOINT DIR PASSED
    }
}));

// Lines 696-713: train_with_async_loading implementation
async fn train_with_async_loading(
    &self,
    model: &mut Mamba2SSM,
    train_data: &[(Tensor, Tensor)],
    val_data: &[(Tensor, Tensor)],
    epochs: usize,
    batch_size: usize,
    checkpoint_dir: Option<&std::path::Path>,  // ✅ CHECKPOINT DIR PARAMETER
) -> Result<Vec<crate::mamba::TrainingEpoch>, MLError> {
    info!(
        "Async data loading enabled (prefetch={}, batch_size={})",
        self.prefetch_count, batch_size
    );

    // Call the new train_async() method with AsyncDataLoader
    model.train_async(train_data, val_data, epochs, batch_size, self.prefetch_count, checkpoint_dir).await  // ✅ CHECKPOINT DIR FORWARDED
}
```

**Implementation Details**:
- **Line 891**: Async training path passes `Some(&self.training_paths.checkpoints_dir())`
- **Line 897**: Sync training path also passes checkpoint directory
- **Line 712**: Internal `train_async()` method receives and uses checkpoint directory
- **Checkpoint path pattern**: `{checkpoint_dir}/mamba2_epoch_{epoch}.safetensors`

**Metadata Saved**:
- Model weights (full SSM state)
- Optimizer state (Adam parameters)
- Training epoch number
- Validation loss history
- Learning rate schedule state

**Resume Capability**: ✅ **FULL SUPPORT**
- Can resume from any epoch checkpoint
- SSM state fully preserved
- Optimizer momentum restored
- Training continues seamlessly

**Priority**: N/A (Already working)
**Effort**: N/A (Reference implementation for other models)

---

## Bug Inventory

### Critical Bugs (P0)

1. **DQN No-Op Checkpoint Callback**
   - **Severity**: CRITICAL
   - **Impact**: 100% checkpoint loss rate
   - **File**: `ml/src/hyperopt/adapters/dqn.rs`
   - **Lines**: 667-670, 675-678, 688-691, 696-699
   - **Fix Effort**: 15-30 minutes
   - **Blocker**: Yes - prevents any DQN hyperopt checkpoint persistence

### High Priority Bugs (P1)

2. **PPO Missing Checkpoint Save**
   - **Severity**: HIGH
   - **Impact**: Cannot resume PPO hyperopt trials
   - **File**: `ml/src/hyperopt/adapters/ppo.rs`
   - **Lines**: 428-448 (training loop), 501-515 (cleanup)
   - **Fix Effort**: 15-30 minutes
   - **Blocker**: No - hyperopt completes but loses model state

### Medium Priority Bugs (P2)

3. **TFT MemoryStorage Non-Persistence**
   - **Severity**: MEDIUM
   - **Impact**: Checkpoints lost on pod termination
   - **File**: `ml/src/hyperopt/adapters/tft.rs`
   - **Lines**: 413 (config), 417-419 (storage)
   - **Fix Effort**: 1-2 hours
   - **Blocker**: No - can retry failed trials

---

## Fix Recommendations (Prioritized)

### Priority 1: DQN Checkpoint Callback (15-30 MIN)

**Effort**: 15-30 minutes
**Impact**: CRITICAL - Unblocks DQN hyperopt checkpoint persistence

**Implementation**:
```rust
// File: ml/src/hyperopt/adapters/dqn.rs
// Lines: 664-686

// REPLACE no-op callback with working checkpoint save:
let checkpoint_callback = |epoch, data: &crate::trainers::dqn::DQNCheckpointData, is_final| {
    if is_final || epoch % 10 == 0 {
        let checkpoint_path = self.training_paths.checkpoints_dir()
            .join(format!("dqn_epoch_{}.safetensors", epoch));

        info!("Saving DQN checkpoint: {:?}", checkpoint_path);

        data.save_to_file(&checkpoint_path)
            .map(|_| checkpoint_path.to_string_lossy().to_string())
            .map_err(|e| format!("Checkpoint save failed: {}", e))
    } else {
        Ok("skipped".to_string())
    }
};

// Use callback in both training paths:
if is_parquet_file {
    handle.block_on(internal_trainer.train_from_parquet(data_path_str, checkpoint_callback))
} else {
    handle.block_on(internal_trainer.train(data_path_str, checkpoint_callback))
}
```

**Test Plan**:
1. Run DQN hyperopt for 3 trials with 20 epochs each
2. Verify checkpoint files created: `dqn_epoch_10.safetensors`, `dqn_epoch_20.safetensors`
3. Verify checkpoint metadata contains trial parameters
4. Test checkpoint loading after pod termination

---

### Priority 2: PPO Checkpoint Save (15-30 MIN)

**Effort**: 15-30 minutes
**Impact**: HIGH - Enables PPO hyperopt checkpoint persistence

**Implementation**:
```rust
// File: ml/src/hyperopt/adapters/ppo.rs
// Lines: 428-448 (inside training loop)

for _batch_idx in 0..num_batches {
    // ... existing trajectory generation and update code ...

    // ADD: Checkpoint saving every 10 batches
    if _batch_idx % 10 == 0 || _batch_idx == num_batches - 1 {
        let checkpoint_path = self.training_paths.checkpoints_dir()
            .join(format!("ppo_batch_{}.safetensors", _batch_idx));

        ppo_agent.save_checkpoint(&checkpoint_path)
            .map_err(|e| MLError::TrainingError(format!("Checkpoint save failed: {}", e)))?;

        // Save metadata (trial params, metrics)
        let metadata = serde_json::json!({
            "batch_idx": _batch_idx,
            "total_batches": num_batches,
            "params": params,
            "policy_loss": total_policy_loss / (_batch_idx as f64 + 1.0),
            "value_loss": total_value_loss / (_batch_idx as f64 + 1.0),
        });

        let metadata_path = checkpoint_path.with_extension("json");
        std::fs::write(metadata_path, serde_json::to_string_pretty(&metadata)?)?;

        info!("Saved PPO checkpoint: {:?}", checkpoint_path);
    }
}
```

**Test Plan**:
1. Run PPO hyperopt for 3 trials with 100 batches each
2. Verify checkpoint files created every 10 batches
3. Verify metadata JSON files contain trial parameters
4. Test checkpoint loading restores policy and value networks

---

### Priority 3: TFT FileSystemStorage (1-2 HOURS)

**Effort**: 1-2 hours
**Impact**: MEDIUM - Persists TFT checkpoints to disk

**Implementation**:
```rust
// File: ml/src/hyperopt/adapters/tft.rs
// Lines: 417-419

// OPTION 1: Use existing FileSystemStorage (if implemented)
let checkpoint_storage = std::sync::Arc::new(
    crate::checkpoint::FileSystemStorage::new(&self.training_paths.checkpoints_dir())
        .map_err(|e| MLError::ModelError(format!("Failed to create checkpoint storage: {}", e)))?
);

// OPTION 2: Implement simple FileSystemStorage if not available
// File: ml/src/checkpoint/filesystem.rs
pub struct FileSystemStorage {
    base_dir: PathBuf,
}

impl FileSystemStorage {
    pub fn new(base_dir: impl Into<PathBuf>) -> Result<Self> {
        let base_dir = base_dir.into();
        std::fs::create_dir_all(&base_dir)?;
        Ok(Self { base_dir })
    }
}

impl CheckpointStorage for FileSystemStorage {
    fn save(&self, key: &str, data: &[u8]) -> Result<()> {
        let path = self.base_dir.join(key);
        std::fs::write(path, data)?;
        Ok(())
    }

    fn load(&self, key: &str) -> Result<Vec<u8>> {
        let path = self.base_dir.join(key);
        Ok(std::fs::read(path)?)
    }
}
```

**Test Plan**:
1. Run TFT hyperopt for 3 trials with 50 epochs each
2. Verify checkpoint files persisted to disk (not just memory)
3. Verify checkpoints survive pod termination
4. Test checkpoint loading restores model state

---

## Cost-Benefit Analysis

| Bug | Fix Effort | Annual Savings | ROI | Break-Even |
|-----|-----------|---------------|-----|-----------|
| **DQN Checkpoint** | 15-30 min | $50/year | **+$40/year** | **1 month** ✅ |
| **PPO Checkpoint** | 15-30 min | $30/year | **+$20/year** | **2 months** ✅ |
| **TFT Checkpoint** | 1-2 hours | $10/year | **-$10 to -$30/year** | 6-12 years ❌ |

**Assumptions**:
- DQN hyperopt: 50 trials/year, 20% failure rate (pod timeouts/OOM)
- PPO hyperopt: 30 trials/year, 15% failure rate
- TFT hyperopt: 20 trials/year, 10% failure rate
- GPU cost: $0.25/hour (RTX A4000)
- Dev cost: $20/hour

**Recommendations**:
1. ✅ **Fix DQN immediately** - Positive ROI within 1 month
2. ✅ **Fix PPO immediately** - Positive ROI within 2 months
3. ❌ **Skip TFT for now** - Training is fast (2 min), checkpoints not cost-effective

---

## Testing Checklist

### DQN Checkpoint Testing
- [ ] Create DQN hyperopt run with 3 trials, 20 epochs each
- [ ] Verify checkpoint files created: `dqn_epoch_10.safetensors`, `dqn_epoch_20.safetensors`
- [ ] Verify checkpoint metadata contains hyperparameters
- [ ] Test checkpoint loading restores Q-network state
- [ ] Simulate pod timeout, verify can resume from last checkpoint

### PPO Checkpoint Testing
- [ ] Create PPO hyperopt run with 3 trials, 100 batches each
- [ ] Verify checkpoint files created every 10 batches
- [ ] Verify metadata JSON files contain trial parameters
- [ ] Test checkpoint loading restores policy/value networks
- [ ] Verify optimizer state (Adam momentum) is restored

### TFT Checkpoint Testing (if implemented)
- [ ] Create TFT hyperopt run with 3 trials, 50 epochs each
- [ ] Verify checkpoint files persisted to disk (not memory)
- [ ] Verify checkpoints survive pod termination
- [ ] Test checkpoint loading restores TFT model state

---

## Summary

**Current State**:
- **1/4 models** have working hyperopt checkpoint saving (MAMBA-2)
- **3/4 models** have critical checkpoint bugs (DQN, PPO, TFT)
- **Total fix effort**: 45 minutes to 2.5 hours (depending on TFT decision)

**Recommended Action Plan**:
1. **IMMEDIATE**: Fix DQN checkpoint callback (15-30 min, P0)
2. **IMMEDIATE**: Fix PPO checkpoint save (15-30 min, P1)
3. **DEFER**: TFT FileSystemStorage (negative ROI, training is fast)

**Expected Outcome**:
- **3/4 models** with working checkpoints (DQN, PPO, MAMBA-2)
- **75% coverage** - sufficient for production hyperopt runs
- **Positive ROI** within 2 months for DQN and PPO fixes

**MAMBA-2 Reference**: Use `ml/src/hyperopt/adapters/mamba2.rs` lines 880-898 as reference implementation for DQN and PPO fixes.