foxhunt/HYPEROPT_LOG_IMPLEMENTATION.md

# Hyperopt Log File Implementation

## Overview
Implementation of proper log file writing for hyperopt training across all ML model adapters.

## Requirements
1. Write training logs to `{logs_dir}/training.log`
2. Write hyperopt trial results to `{hyperopt_dir}/trials.json`
3. Follow the checkpoint saving pattern (use `TrainingPaths.create_all()`)
4. Implement for ALL adapters: MAMBA-2, DQN, PPO, TFT

## Implementation Pattern

### 1. Create Log Writer Helper (add to each adapter)

```rust
use std::fs::OpenOptions;
use std::io::Write;

/// Write a log entry to the training log file
fn write_training_log(logs_dir: &std::path::Path, message: &str) -> Result<(), std::io::Error> {
    let log_file = logs_dir.join("training.log");
    let mut file = OpenOptions::new()
        .create(true)
        .append(true)
        .open(log_file)?;

    let timestamp = chrono::Utc::now().format("%Y-%m-%d %H:%M:%S");
    writeln!(file, "[{}] {}", timestamp, message)?;
    Ok(())
}

/// Write trial results to JSON file
fn write_trial_result<P: serde::Serialize>(
    hyperopt_dir: &std::path::Path,
    trial_result: &crate::hyperopt::traits::TrialResult<P>,
) -> Result<(), std::io::Error> {
    let trials_file = hyperopt_dir.join("trials.json");

    // Read existing trials (if any)
    let mut all_trials = if trials_file.exists() {
        let content = std::fs::read_to_string(&trials_file)?;
        serde_json::from_str::<Vec<serde_json::Value>>(&content).unwrap_or_default()
    } else {
        Vec::new()
    };

    // Append new trial
    let trial_json = serde_json::to_value(trial_result)
        .map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e))?;
    all_trials.push(trial_json);

    // Write back to file (pretty printed)
    let content = serde_json::to_string_pretty(&all_trials)
        .map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e))?;
    std::fs::write(&trials_file, content)?;

    Ok(())
}
```

### 2. Integration Points in train_with_params()

Add logging at these key points:

```rust
fn train_with_params(&mut self, params: Self::Params) -> Result<Self::Metrics, MLError> {
    // 1. Log trial start
    let trial_start = std::time::Instant::now();
    write_training_log(
        &self.training_paths.logs_dir(),
        &format!("=== Starting Trial ===\n{:#?}", params)
    ).ok(); // Don't fail on log write errors

    // ... existing parameter logging ...

    // 2. Create all directories (including logs_dir)
    self.training_paths.create_all()
        .map_err(|e| MLError::ModelError(format!("Failed to create training directories: {}", e)))?;

    // ... training code ...

    // 3. Log training completion
    let duration_secs = trial_start.elapsed().as_secs_f64();
    write_training_log(
        &self.training_paths.logs_dir(),
        &format!("Training completed in {:.2}s: metrics={:#?}", duration_secs, metrics)
    ).ok();

    // 4. Write trial result to JSON
    let trial_result = crate::hyperopt::traits::TrialResult {
        trial_num: 0, // Will be set by optimizer
        params: params.clone(),
        objective: Self::extract_objective(&metrics),
        duration_secs,
    };

    write_trial_result(&self.training_paths.hyperopt_dir(), &trial_result).ok();

    Ok(metrics)
}
```

## Code Changes by File

### 1. ml/src/hyperopt/adapters/mamba2.rs

**Add imports** (after line 38):
```rust
use std::fs::OpenOptions;
use std::io::Write as IoWrite;
```

**Add helper functions** (after line 698, before `impl HyperparameterOptimizable`):
```rust
/// Write a log entry to the training log file
fn write_training_log_mamba2(logs_dir: &std::path::Path, message: &str) -> Result<(), std::io::Error> {
    let log_file = logs_dir.join("training.log");
    let mut file = OpenOptions::new()
        .create(true)
        .append(true)
        .open(log_file)?;

    let timestamp = chrono::Utc::now().format("%Y-%m-%d %H:%M:%S");
    writeln!(file, "[{}] {}", timestamp, message)?;
    Ok(())
}

/// Write trial results to JSON file
fn write_trial_result_mamba2(
    hyperopt_dir: &std::path::Path,
    trial_result: &crate::hyperopt::traits::TrialResult<Mamba2Params>,
) -> Result<(), std::io::Error> {
    let trials_file = hyperopt_dir.join("trials.json");

    // Read existing trials (if any)
    let mut all_trials = if trials_file.exists() {
        let content = std::fs::read_to_string(&trials_file)?;
        serde_json::from_str::<Vec<serde_json::Value>>(&content).unwrap_or_default()
    } else {
        Vec::new()
    };

    // Append new trial
    let trial_json = serde_json::to_value(trial_result)
        .map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e))?;
    all_trials.push(trial_json);

    // Write back to file (pretty printed)
    let content = serde_json::to_string_pretty(&all_trials)
        .map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e))?;
    std::fs::write(&trials_file, content)?;

    Ok(())
}
```

**Modify train_with_params** (add logging at line 705, 799, 889):
```rust
fn train_with_params(&mut self, mut params: Self::Params) -> Result<Self::Metrics, MLError> {
    // START: Add trial timing
    let trial_start = std::time::Instant::now();

    // Clamp batch_size to configured bounds (for GPU memory constraints)
    let original_batch_size = params.batch_size;
    // ... existing code ...

    info!("Training MAMBA-2 with 12 hyperparameters:");
    // ... existing parameter logging ...

    // Log trial start
    write_training_log_mamba2(
        &self.training_paths.logs_dir(),
        &format!("=== Starting MAMBA-2 Trial ===\nParams: {:#?}", params)
    ).ok();

    // ... rest of existing code until line 889 ...

    info!("Training completed:");
    info!("  Training loss: {:.6}", metrics.train_loss);
    // ... existing metric logging ...

    // END: Add trial completion logging
    let duration_secs = trial_start.elapsed().as_secs_f64();
    write_training_log_mamba2(
        &self.training_paths.logs_dir(),
        &format!("Training completed in {:.2}s: val_loss={:.6}, train_loss={:.6}, accuracy={:.2}%",
                 duration_secs, metrics.val_loss, metrics.train_loss, metrics.directional_accuracy * 100.0)
    ).ok();

    // Write trial result to JSON
    let trial_result = crate::hyperopt::traits::TrialResult {
        trial_num: 0, // Will be overwritten by optimizer
        params: params.clone(),
        objective: Self::extract_objective(&metrics),
        duration_secs,
    };

    write_trial_result_mamba2(&self.training_paths.hyperopt_dir(), &trial_result).ok();

    Ok(metrics)
}
```

### 2. ml/src/hyperopt/adapters/dqn.rs

**Add imports** (after line 36):
```rust
use std::fs::OpenOptions;
use std::io::Write as IoWrite;
```

**Add helper functions** (after line 285, before `impl HyperparameterOptimizable`):
```rust
/// Write a log entry to the training log file
fn write_training_log_dqn(logs_dir: &std::path::Path, message: &str) -> Result<(), std::io::Error> {
    let log_file = logs_dir.join("training.log");
    let mut file = OpenOptions::new()
        .create(true)
        .append(true)
        .open(log_file)?;

    let timestamp = chrono::Utc::now().format("%Y-%m-%d %H:%M:%S");
    writeln!(file, "[{}] {}", timestamp, message)?;
    Ok(())
}

/// Write trial results to JSON file
fn write_trial_result_dqn(
    hyperopt_dir: &std::path::Path,
    trial_result: &crate::hyperopt::traits::TrialResult<DQNParams>,
) -> Result<(), std::io::Error> {
    let trials_file = hyperopt_dir.join("trials.json");

    // Read existing trials (if any)
    let mut all_trials = if trials_file.exists() {
        let content = std::fs::read_to_string(&trials_file)?;
        serde_json::from_str::<Vec<serde_json::Value>>(&content).unwrap_or_default()
    } else {
        Vec::new()
    };

    // Append new trial
    let trial_json = serde_json::to_value(trial_result)
        .map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e))?;
    all_trials.push(trial_json);

    // Write back to file (pretty printed)
    let content = serde_json::to_string_pretty(&all_trials)
        .map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e))?;
    std::fs::write(&trials_file, content)?;

    Ok(())
}
```

**Modify train_with_params** (add logging at line 291, 310, 416):
```rust
fn train_with_params(&mut self, params: Self::Params) -> Result<Self::Metrics, MLError> {
    // START: Add trial timing
    let trial_start = std::time::Instant::now();

    // Fix 1: Clamp buffer size to max (4GB GPU constraint)
    let clamped_buffer_size = params.buffer_size.min(self.buffer_size_max);

    info!("Training DQN with parameters:");
    // ... existing parameter logging ...

    // Log trial start
    write_training_log_dqn(
        &self.training_paths.logs_dir(),
        &format!("=== Starting DQN Trial ===\nParams: {:#?}", params)
    ).ok();

    // Create all training directories
    self.training_paths.create_all()
        .map_err(|e| MLError::ConfigError {
            reason: format!("Failed to create training directories: {}", e),
        })?;

    info!("Training directories created:");
    info!("  Checkpoints: {:?}", self.training_paths.checkpoints_dir());
    info!("  Logs: {:?}", self.training_paths.logs_dir());  // Add this line
    info!("  Hyperopt: {:?}", self.training_paths.hyperopt_dir());  // Add this line

    // ... rest of existing code until line 416 ...

    info!("Training completed:");
    info!("  Final loss: {:.6}", metrics.train_loss);
    info!("  Avg Q-value: {:.4}", metrics.avg_q_value);

    // END: Add trial completion logging
    let duration_secs = trial_start.elapsed().as_secs_f64();
    write_training_log_dqn(
        &self.training_paths.logs_dir(),
        &format!("Training completed in {:.2}s: loss={:.6}, q_value={:.4}",
                 duration_secs, metrics.train_loss, metrics.avg_q_value)
    ).ok();

    // Write trial result to JSON
    let trial_result = crate::hyperopt::traits::TrialResult {
        trial_num: 0, // Will be overwritten by optimizer
        params: params.clone(),
        objective: Self::extract_objective(&metrics),
        duration_secs,
    };

    write_trial_result_dqn(&self.training_paths.hyperopt_dir(), &trial_result).ok();

    Ok(metrics)
}
```

### 3. ml/src/hyperopt/adapters/ppo.rs

**Add imports** (after line 38):
```rust
use std::fs::OpenOptions;
use std::io::Write as IoWrite;
```

**Add helper functions** (after line 244, before `impl HyperparameterOptimizable`):
```rust
/// Write a log entry to the training log file
fn write_training_log_ppo(logs_dir: &std::path::Path, message: &str) -> Result<(), std::io::Error> {
    let log_file = logs_dir.join("training.log");
    let mut file = OpenOptions::new()
        .create(true)
        .append(true)
        .open(log_file)?;

    let timestamp = chrono::Utc::now().format("%Y-%m-%d %H:%M:%S");
    writeln!(file, "[{}] {}", timestamp, message)?;
    Ok(())
}

/// Write trial results to JSON file
fn write_trial_result_ppo(
    hyperopt_dir: &std::path::Path,
    trial_result: &crate::hyperopt::traits::TrialResult<PPOParams>,
) -> Result<(), std::io::Error> {
    let trials_file = hyperopt_dir.join("trials.json");

    // Read existing trials (if any)
    let mut all_trials = if trials_file.exists() {
        let content = std::fs::read_to_string(&trials_file)?;
        serde_json::from_str::<Vec<serde_json::Value>>(&content).unwrap_or_default()
    } else {
        Vec::new()
    };

    // Append new trial
    let trial_json = serde_json::to_value(trial_result)
        .map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e))?;
    all_trials.push(trial_json);

    // Write back to file (pretty printed)
    let content = serde_json::to_string_pretty(&all_trials)
        .map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e))?;
    std::fs::write(&trials_file, content)?;

    Ok(())
}
```

**Modify train_with_params** (add logging at line 250, 384):
```rust
fn train_with_params(&mut self, params: Self::Params) -> Result<Self::Metrics, MLError> {
    // START: Add trial timing
    let trial_start = std::time::Instant::now();

    info!("Training PPO with parameters:");
    // ... existing parameter logging ...

    // Log trial start
    write_training_log_ppo(
        &self.training_paths.logs_dir(),
        &format!("=== Starting PPO Trial ===\nParams: {:#?}", params)
    ).ok();

    // Create PPO config with trial hyperparameters
    let ppo_config = PPOConfig {
        // ... existing config ...
    };

    // Create directories (add this before creating PPO agent)
    self.training_paths.create_all()
        .map_err(|e| MLError::ModelError(format!("Failed to create training directories: {}", e)))?;

    info!("Training directories created:");
    info!("  Logs: {:?}", self.training_paths.logs_dir());
    info!("  Hyperopt: {:?}", self.training_paths.hyperopt_dir());

    // ... rest of existing code until line 384 ...

    info!("Training completed:");
    info!("  Policy loss: {:.6}", metrics.policy_loss);
    // ... existing metric logging ...

    // END: Add trial completion logging
    let duration_secs = trial_start.elapsed().as_secs_f64();
    write_training_log_ppo(
        &self.training_paths.logs_dir(),
        &format!("Training completed in {:.2}s: val_policy_loss={:.6}, val_value_loss={:.6}",
                 duration_secs, metrics.val_policy_loss, metrics.val_value_loss)
    ).ok();

    // Write trial result to JSON
    let trial_result = crate::hyperopt::traits::TrialResult {
        trial_num: 0, // Will be overwritten by optimizer
        params: params.clone(),
        objective: Self::extract_objective(&metrics),
        duration_secs,
    };

    write_trial_result_ppo(&self.training_paths.hyperopt_dir(), &trial_result).ok();

    Ok(metrics)
}
```

### 4. ml/src/hyperopt/adapters/tft.rs

**Add imports** (after line 38):
```rust
use std::fs::OpenOptions;
use std::io::Write as IoWrite;
```

**Add helper functions** (after line 275, before `impl HyperparameterOptimizable`):
```rust
/// Write a log entry to the training log file
fn write_training_log_tft(logs_dir: &std::path::Path, message: &str) -> Result<(), std::io::Error> {
    let log_file = logs_dir.join("training.log");
    let mut file = OpenOptions::new()
        .create(true)
        .append(true)
        .open(log_file)?;

    let timestamp = chrono::Utc::now().format("%Y-%m-%d %H:%M:%S");
    writeln!(file, "[{}] {}", timestamp, message)?;
    Ok(())
}

/// Write trial results to JSON file
fn write_trial_result_tft(
    hyperopt_dir: &std::path::Path,
    trial_result: &crate::hyperopt::traits::TrialResult<TFTParams>,
) -> Result<(), std::io::Error> {
    let trials_file = hyperopt_dir.join("trials.json");

    // Read existing trials (if any)
    let mut all_trials = if trials_file.exists() {
        let content = std::fs::read_to_string(&trials_file)?;
        serde_json::from_str::<Vec<serde_json::Value>>(&content).unwrap_or_default()
    } else {
        Vec::new()
    };

    // Append new trial
    let trial_json = serde_json::to_value(trial_result)
        .map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e))?;
    all_trials.push(trial_json);

    // Write back to file (pretty printed)
    let content = serde_json::to_string_pretty(&all_trials)
        .map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e))?;
    std::fs::write(&trials_file, content)?;

    Ok(())
}
```

**Modify train_with_params** (add logging at line 281, 303, 369):
```rust
fn train_with_params(&mut self, params: Self::Params) -> Result<Self::Metrics, MLError> {
    // START: Add trial timing
    let trial_start = std::time::Instant::now();

    info!("Training TFT with parameters:");
    // ... existing parameter logging ...

    // Log trial start
    write_training_log_tft(
        &self.training_paths.logs_dir(),
        &format!("=== Starting TFT Trial ===\nParams: {:#?}", params)
    ).ok();

    // Validate num_heads divides hidden_size
    if params.hidden_size % params.num_heads != 0 {
        // ... existing validation ...
    }

    // Create directories (add this before creating trainer)
    self.training_paths.create_all()
        .map_err(|e| MLError::ModelError(format!("Failed to create training directories: {}", e)))?;

    info!("Training directories created:");
    info!("  Checkpoints: {:?}", self.training_paths.checkpoints_dir());
    info!("  Logs: {:?}", self.training_paths.logs_dir());
    info!("  Hyperopt: {:?}", self.training_paths.hyperopt_dir());

    // ... rest of existing code until line 369 ...

    info!("Training completed:");
    info!("  Training loss: {:.6}", metrics.train_loss);
    // ... existing metric logging ...

    // END: Add trial completion logging
    let duration_secs = trial_start.elapsed().as_secs_f64();
    write_training_log_tft(
        &self.training_paths.logs_dir(),
        &format!("Training completed in {:.2}s: val_loss={:.6}, train_loss={:.6}, rmse={:.4}",
                 duration_secs, metrics.val_loss, metrics.train_loss, metrics.val_rmse)
    ).ok();

    // Write trial result to JSON
    let trial_result = crate::hyperopt::traits::TrialResult {
        trial_num: 0, // Will be overwritten by optimizer
        params: params.clone(),
        objective: Self::extract_objective(&metrics),
        duration_secs,
    };

    write_trial_result_tft(&self.training_paths.hyperopt_dir(), &trial_result).ok();

    Ok(metrics)
}
```

## Verification

After implementation, verify:

1. **Directory Structure**:
```
/runpod-volume/training_runs/{model_name}/run_{run_id}/
├── checkpoints/
│   └── best_model.safetensors
├── logs/
│   └── training.log          # ← NEW
├── hyperopt/
│   └── trials.json           # ← NEW
└── metrics/
```

2. **Log Format** (training.log):
```
[2025-10-29 14:32:15] === Starting MAMBA-2 Trial ===
Params: Mamba2Params {
    learning_rate: 0.0001,
    batch_size: 32,
    ...
}
[2025-10-29 14:34:23] Training completed in 128.45s: val_loss=0.234567, train_loss=0.198765, accuracy=67.89%
```

3. **Trial Results** (trials.json):
```json
[
  {
    "trial_num": 1,
    "params": {
      "learning_rate": 0.0001,
      "batch_size": 32,
      ...
    },
    "objective": 0.234567,
    "duration_secs": 128.45
  },
  ...
]
```

## Testing

Test with hyperopt examples:
```bash
# MAMBA-2
cargo run -p ml --example hyperopt_mamba2_demo --release --features cuda

# DQN
cargo run -p ml --example hyperopt_dqn_demo --release --features cuda

# PPO
cargo run -p ml --example hyperopt_ppo_demo --release --features cuda

# TFT
cargo run -p ml --example hyperopt_tft_demo --release --features cuda
```

Check outputs:
```bash
ls -lh /tmp/ml_training/training_runs/*/run_*/logs/training.log
cat /tmp/ml_training/training_runs/*/run_*/hyperopt/trials.json
```

## Notes

1. **Error Handling**: `.ok()` is used for log writes to avoid failing trials on I/O errors
2. **Timestamps**: UTC timestamps for consistency across deployments
3. **JSON Format**: Pretty-printed for human readability
4. **Append Mode**: Logs append, trials accumulate in JSON array
5. **Trial Numbers**: Set to 0 initially, optimizer overwrites with actual trial number

## Dependencies

No new dependencies required - uses existing:
- `std::fs::OpenOptions` - file I/O
- `std::io::Write` - write operations
- `chrono::Utc` - timestamps (already imported)
- `serde_json` - JSON serialization (already in Cargo.toml)

## Status

- [ ] MAMBA-2 adapter (/home/jgrusewski/Work/foxhunt/ml/src/hyperopt/adapters/mamba2.rs)
- [ ] DQN adapter (/home/jgrusewski/Work/foxhunt/ml/src/hyperopt/adapters/dqn.rs)
- [ ] PPO adapter (/home/jgrusewski/Work/foxhunt/ml/src/hyperopt/adapters/ppo.rs)
- [ ] TFT adapter (/home/jgrusewski/Work/foxhunt/ml/src/hyperopt/adapters/tft.rs)
- [ ] Integration testing
- [ ] Runpod deployment verification