foxhunt/PPO_COMPREHENSIVE_TUNING_GUIDE.md

# PPO Comprehensive Hyperparameter Tuning - Execution Guide

**Mission**: Run 50-trial Optuna hyperparameter optimization for PPO model
**Objective**: Maximize `0.7 * Sharpe Ratio + 0.3 * Explained Variance`
**Baseline**: Epoch 380 (explained_var=0.4469, EXCELLENT performance)
**Expected Duration**: 8-12 hours
**Date**: 2025-10-14

---

## Executive Summary

This guide provides a complete workflow for running comprehensive PPO hyperparameter tuning with:

- **50 trials** with intelligent sampling (TPE algorithm)
- **Early stopping** at epoch 50 (vs 500-epoch baseline)
- **Composite objective**: 0.7 * Sharpe + 0.3 * Explained Variance
- **4-symbol validation**: 6E.FUT, ZN.FUT, ES.FUT, NQ.FUT
- **MedianPruner**: 30-40% time savings via early trial termination
- **GPU acceleration**: RTX 3050 Ti with batch size <= 230

### Key Deliverables

| File | Purpose |
|------|---------|
| `tuning_config_ppo_comprehensive.yaml` | PPO search space configuration |
| `run_ppo_comprehensive_tuning.sh` | End-to-end execution script |
| `hyperparameter_tuner_ppo_enhanced.py` | Enhanced Python tuner with composite objective |
| `PPO_COMPREHENSIVE_TUNING_GUIDE.md` | This guide |

---

## Prerequisites

### 1. System Requirements

- **GPU**: NVIDIA RTX 3050 Ti (4GB VRAM) with CUDA 12.1
- **Memory**: 16GB RAM minimum
- **Storage**: 50GB free space for checkpoints/logs
- **OS**: Linux (validated on Ubuntu)

### 2. Data Files Required

All 4 symbol data files must be present:

```bash
test_data/6E.FUT_2024-01-02.dbn.zst   # Euro FX (1,661 bars)
test_data/ZN.FUT_2024-01-02.dbn.zst   # Treasury (28,935 bars)
test_data/ES.FUT_2024-01-02.dbn.zst   # S&P 500 (1,674 bars)
test_data/NQ.FUT_2024-01-02.dbn.zst   # Nasdaq (available)
```

### 3. Services Running

```bash
# Start required services
docker-compose up -d postgres redis vault

# Start ML Training Service
cargo run -p ml_training_service --release &

# Verify services
grpc_health_probe -addr=localhost:50054  # ML Training Service
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c '\dt'
```

### 4. Baseline Checkpoint

The baseline checkpoint from Agent 79 analysis must exist:

```bash
ml/trained_models/production/ppo_real_data/ppo_actor_epoch_380.safetensors
```

**Baseline Performance**:
- Explained Variance: **0.4469** (only 0.0531 from optimal 0.5)
- Training Phase: Late-stage refinement (epoch 350-500)
- Status: **EXCELLENT** (within 5% of theoretical optimal)

---

## Search Space Configuration

### Hyperparameters to Optimize (6 parameters)

| Hyperparameter | Search Space | Choices | Rationale |
|----------------|--------------|---------|-----------|
| **Learning Rate** | [0.0001, 0.0003, 0.001] | 3 | Critical for convergence speed |
| **Batch Size** | [32, 64, 128, 256] | 4 | GPU memory vs sample efficiency |
| **Gamma** | [0.95, 0.99] | 2 | Discount factor for returns |
| **GAE Lambda** | [0.9, 0.95, 0.98] | 3 | Advantage estimation smoothing |
| **Clip Epsilon** | [0.1, 0.2, 0.3] | 3 | PPO policy update constraint |
| **Entropy Coef** | [0.001, 0.01, 0.1] | 3 | Exploration vs exploitation |

**Total Combinations**: 3 × 4 × 2 × 3 × 3 × 3 = **648 possible configurations**
**Sampling Strategy**: TPE (Tree-structured Parzen Estimator) - intelligent sampling
**Trials**: 50 (covers ~7.7% of search space with intelligent prioritization)

### Fixed Parameters (Agent 32 Fix)

These are NOT optimized (fixed based on Agent 32 policy collapse fix):

```yaml
policy_hidden_dims: [128, 64]
value_hidden_dims: [128, 64]
value_loss_coef: 1.0        # Prioritize value learning
rollout_steps: 2048
minibatch_size: 64
num_ppo_epochs: 10
max_grad_norm: 0.5
```

---

## Objective Function

### Composite Objective Formula

```
Objective = 0.7 × Sharpe Ratio + 0.3 × Explained Variance
```

### Rationale

| Metric | Weight | Rationale |
|--------|--------|-----------|
| **Sharpe Ratio** | 70% | Primary metric for risk-adjusted returns (production goal) |
| **Explained Variance** | 30% | Critical for PPO value network accuracy (convergence indicator) |

### Why Composite?

1. **Sharpe Ratio Alone**: May converge poorly (value network struggles)
2. **Explained Variance Alone**: May not maximize trading performance
3. **Combined**: Balances trading performance + model convergence quality

### Example Calculation

```python
# Trial 1 Results
sharpe_ratio = 1.5
explained_variance = 0.42

objective = 0.7 * 1.5 + 0.3 * 0.42 = 1.05 + 0.126 = 1.176

# Trial 2 Results
sharpe_ratio = 1.8
explained_variance = 0.38

objective = 0.7 * 1.8 + 0.3 * 0.38 = 1.26 + 0.114 = 1.374

# Trial 2 wins despite lower explained variance!
```

---

## Execution Steps

### Quick Start (Recommended)

```bash
cd /home/jgrusewski/Work/foxhunt

# Run comprehensive tuning (8-12 hours)
./run_ppo_comprehensive_tuning.sh
```

This script handles:
- ✅ Prerequisites validation
- ✅ Service health checks
- ✅ Data file verification
- ✅ GPU detection
- ✅ Job submission via TLI
- ✅ Progress monitoring
- ✅ Results retrieval
- ✅ Summary report generation

### Manual Execution (Advanced)

If you prefer manual control:

#### Step 1: Verify Prerequisites

```bash
# Check GPU
nvidia-smi

# Check data files
ls -lh test_data/*.dbn.zst

# Check baseline
ls -lh ml/trained_models/production/ppo_real_data/ppo_actor_epoch_380.safetensors

# Check services
grpc_health_probe -addr=localhost:50054
```

#### Step 2: Start Tuning Job

```bash
# Via TLI (recommended)
cargo run -p tli -- tune start \
  --model PPO \
  --trials 50 \
  --config tuning_config_ppo_comprehensive.yaml

# Capture job ID
export JOB_ID="<uuid-from-output>"
```

#### Step 3: Monitor Progress

```bash
# Real-time monitoring
cargo run -p tli -- tune status --job-id $JOB_ID --watch

# Check every 5 minutes
watch -n 300 "cargo run -p tli -- tune status --job-id $JOB_ID"
```

#### Step 4: Retrieve Best Hyperparameters

```bash
# After completion (8-12 hours)
cargo run -p tli -- tune best --job-id $JOB_ID

# Stop if needed (graceful shutdown)
cargo run -p tli -- tune stop --job-id $JOB_ID
```

---

## Expected Timeline

### Duration Breakdown

| Phase | Duration | Description |
|-------|----------|-------------|
| **Setup** | 5 min | Prerequisites check, service validation |
| **Trial 1-5** | 60-90 min | No pruning (baseline establishment) |
| **Trial 6-50** | 6-10 hours | MedianPruner active (30-40% time savings) |
| **Analysis** | 10 min | Best hyperparameters retrieval |
| **Total** | **8-12 hours** | Full 50-trial optimization |

### Per-Trial Estimates

- **Successful Trial**: 10-15 minutes (50 epochs @ 12-18 sec/epoch)
- **Pruned Trial**: 3-5 minutes (stopped early by MedianPruner)
- **Pruning Rate**: 30-40% of trials (15-20 trials pruned)
- **Average**: ~10 minutes per trial effective

### Progress Checkpoints

Monitor these milestones during execution:

| Trials | Expected Time | Status Check |
|--------|--------------|--------------|
| 5 | 1 hour | Baseline established, MedianPruner activating |
| 10 | 2 hours | Pruning active, best trial identified |
| 25 | 5 hours | Halfway point, convergence patterns visible |
| 40 | 8 hours | Late-stage refinement, diminishing returns |
| 50 | 10 hours | Completion, final analysis |

---

## Output Files

### Generated Files

```
ml/trained_models/tuning/ppo_comprehensive/
├── job_id.txt                      # Job UUID for reference
├── tuning_execution.log            # Full execution log
├── best_hyperparameters.txt        # Best hyperparameters (plaintext)
├── TUNING_SUMMARY_REPORT.md        # Comprehensive summary
├── checkpoints/                    # Checkpoints from trials
│   ├── trial_0_epoch_50.safetensors
│   ├── trial_1_epoch_50.safetensors
│   └── ...
├── plots/                          # Optuna visualization plots
│   ├── optimization_history.png
│   ├── param_importances.png
│   ├── parallel_coordinate.png
│   └── contour_plot.png
└── logs/
    ├── trial_0.log
    ├── trial_1.log
    └── ...
```

### Key Output Files

1. **best_hyperparameters.txt**: Use this for production training
2. **TUNING_SUMMARY_REPORT.md**: Share with team for analysis
3. **tuning_execution.log**: Debugging and audit trail

---

## Interpreting Results

### Success Criteria

| Metric | Target | Status |
|--------|--------|--------|
| Composite Objective | > baseline | Check best_hyperparameters.txt |
| Sharpe Ratio | > 1.5 | Risk-adjusted returns improvement |
| Explained Variance | > 0.45 | Value network convergence quality |
| Combined Improvement | > 5% | Overall performance gain |

### Baseline Comparison

**Baseline (Epoch 380)**:
- Explained Variance: 0.4469
- Sharpe Ratio: TBD (measure during tuning)
- Training: 500 epochs (5.6 minutes)

**Target (Tuned Model)**:
- Explained Variance: > 0.47 (5% improvement)
- Sharpe Ratio: > baseline (risk-adjusted gains)
- Training: 50 epochs (early stopping efficiency)

### Value Network Convergence Analysis

Monitor these indicators in the summary report:

1. **Explained Variance Trajectory**: Should approach 0.47-0.50
2. **Policy Stability**: KL divergence < 0.0002 (healthy)
3. **Loss Dynamics**: Policy loss stable, value loss decreasing
4. **Entropy Decay**: Gradual reduction (exploration → exploitation)

---

## Troubleshooting

### Common Issues

#### 1. GPU Out of Memory

**Symptom**: Trial fails with CUDA OOM error
**Solution**: Batch size auto-limited to 230 for RTX 3050 Ti

```bash
# Check GPU memory
nvidia-smi

# If OOM persists, reduce batch size in config
# Edit tuning_config_ppo_comprehensive.yaml:
batch_size:
  choices: [32, 64, 128]  # Remove 256
```

#### 2. Service Not Running

**Symptom**: gRPC connection refused
**Solution**: Restart ML Training Service

```bash
# Check status
grpc_health_probe -addr=localhost:50054

# Restart if needed
pkill -f ml_training_service
cargo run -p ml_training_service --release &
```

#### 3. Missing Data Files

**Symptom**: "Data file not found" error
**Solution**: Download missing symbols

```bash
# Check data files
ls test_data/*.dbn.zst

# If missing, download from Databento
# See: ML_DATA_DOWNLOAD_GUIDE.md
```

#### 4. Slow Progress

**Symptom**: Trials taking >20 minutes each
**Solution**: Check GPU utilization

```bash
# Monitor GPU usage
watch -n 1 nvidia-smi

# If GPU idle (<10% utilization):
# 1. Check CUDA_HOME environment
# 2. Verify candle-core GPU compilation
# 3. Check use_gpu flag in config
```

#### 5. Trial Failures

**Symptom**: Multiple trials failing with training errors
**Solution**: Check logs for patterns

```bash
# View trial logs
tail -f ml/trained_models/tuning/ppo_comprehensive/logs/trial_*.log

# Common causes:
# - NaN in loss (reduce learning rate)
# - Policy collapse (increase entropy_coef)
# - Gradient explosion (reduce learning rate)
```

---

## Next Steps After Tuning

### 1. Validate Best Hyperparameters

```bash
# Run 500-epoch production training with best hyperparameters
cargo run -p ml --example train_ppo_production \
  --learning-rate <best_lr> \
  --batch-size <best_batch> \
  --gamma <best_gamma> \
  --gae-lambda <best_lambda> \
  --clip-epsilon <best_clip> \
  --entropy-coef <best_entropy>
```

### 2. Checkpoint Analysis

```bash
# Analyze checkpoints from production training
cargo run -p ml --example analyze_ppo_checkpoints \
  --checkpoint-dir ml/trained_models/production/ppo_tuned/
```

### 3. Cross-Symbol Backtesting

```bash
# Test trained model on all 4 symbols
cargo run -p backtesting_service --example comprehensive_backtest \
  --model-path ml/trained_models/production/ppo_tuned/ppo_final_epoch500.safetensors \
  --symbols 6E.FUT,ZN.FUT,ES.FUT,NQ.FUT
```

### 4. Production Deployment

After validation:

```bash
# Deploy to inference service
cp ml/trained_models/production/ppo_tuned/ppo_final_epoch500.safetensors \
   services/inference/models/ppo_production_v2.safetensors

# Update model registry
cargo run -p ml --example model_registry_api register \
  --model-path models/ppo_production_v2.safetensors \
  --version 2.0 \
  --description "PPO with optimized hyperparameters (50-trial tuning)"
```

---

## Performance Expectations

### Improvement Targets

Based on similar hyperparameter tuning studies:

| Metric | Baseline | Target | Improvement |
|--------|----------|--------|-------------|
| Explained Variance | 0.4469 | 0.47+ | +5-10% |
| Sharpe Ratio | TBD | +10-20% | Risk-adjusted gains |
| Value Network Loss | 200-220 | 150-180 | Better convergence |
| Policy Stability | Good | Excellent | Lower KL divergence |

### Realistic Outcomes

- **Best Case**: 15-20% improvement in composite objective
- **Expected Case**: 5-10% improvement (still significant)
- **Worst Case**: Marginal improvement (baseline already excellent)

**Note**: The baseline (Epoch 380) is already **EXCELLENT** (explained_var=0.4469, only 0.0531 from optimal 0.5). Significant improvements may be limited, but any gain is valuable for production trading.

---

## References

### Related Documentation

- **Baseline Analysis**: `PPO_CHECKPOINT_ANALYSIS_REPORT.md`
- **Agent 32 Fix**: `AGENT32_PPO_FIX_SUMMARY.md`
- **Convergence Analysis**: `CONVERGENCE_ANALYSIS_REPORT.md`
- **ML Training Roadmap**: `ML_TRAINING_ROADMAP.md`
- **Optuna Integration**: `OPTUNA_TUNING_INTEGRATION_REPORT.md`

### Key Findings from Agent 79

- **Epoch 380**: Best checkpoint (explained_var=0.4469)
- **Top 10 Checkpoints**: All in "Balanced" risk profile (0.40-0.45)
- **Training Quality**: 100% policy update rate, zero NaN occurrences
- **Value Network**: Improved 33.2% from epoch 10 to 500
- **Policy Stability**: KL divergence mean 0.000138 (healthy)

---

## FAQ

### Q1: Why 50 trials instead of 100+?

**A**: With 648 possible combinations and TPE intelligent sampling, 50 trials covers critical regions of the search space efficiently. MedianPruner further optimizes by stopping unpromising trials early.

### Q2: Why early stopping at epoch 50 vs 500?

**A**: For hyperparameter search, we prioritize breadth (50 trials) over depth (500 epochs per trial). Early stopping at epoch 50 still reveals which hyperparameter sets converge best. Once we find optimal hyperparameters, we run full 500-epoch production training.

### Q3: Can I pause/resume tuning?

**A**: Yes! Optuna JournalStorage persists state to MinIO. If interrupted:

```bash
# Resume with same job ID
cargo run -p tli -- tune start \
  --model PPO \
  --trials 50 \
  --config tuning_config_ppo_comprehensive.yaml \
  --resume $JOB_ID
```

### Q4: How do I know if tuning is working?

**A**: Monitor these indicators:

1. **Trial Progress**: Should complete 4-6 trials/hour
2. **Pruning**: 30-40% of trials pruned (MedianPruner active)
3. **Best Value Improving**: Composite objective increasing over trials
4. **No Repeated Failures**: <5% trial failure rate acceptable

### Q5: What if results are worse than baseline?

**A**: This is unlikely but possible. If composite objective < baseline:

1. **Verify Baseline Measurement**: Re-run baseline with same validation data
2. **Check Objective Weights**: Try 0.8 sharpe + 0.2 explained_var
3. **Expand Search Space**: Add more learning rate options
4. **Increase Trials**: Run 100 trials for better coverage

---

## Support

For issues or questions:

1. **Check Logs**: `ml/trained_models/tuning/ppo_comprehensive/tuning_execution.log`
2. **Review Error Messages**: Trial-specific logs in `logs/trial_*.log`
3. **Verify Services**: `grpc_health_probe -addr=localhost:50054`
4. **GPU Status**: `nvidia-smi`

---

**Mission Status**: ⚙️ **READY TO EXECUTE**
**Estimated Completion**: 8-12 hours from start
**Next Action**: Run `./run_ppo_comprehensive_tuning.sh`

**Good luck! 🚀**