foxhunt/docs/archive/ml_models/PPO_COMPREHENSIVE_TUNING_GUIDE.md

PPO Comprehensive Hyperparameter Tuning - Execution Guide

Mission: Run 50-trial Optuna hyperparameter optimization for PPO model Objective: Maximize 0.7 * Sharpe Ratio + 0.3 * Explained Variance Baseline: Epoch 380 (explained_var=0.4469, EXCELLENT performance) Expected Duration: 8-12 hours Date: 2025-10-14

---

Executive Summary

This guide provides a complete workflow for running comprehensive PPO hyperparameter tuning with:

• 50 trials with intelligent sampling (TPE algorithm)
• Early stopping at epoch 50 (vs 500-epoch baseline)
• Composite objective: 0.7 * Sharpe + 0.3 * Explained Variance
• 4-symbol validation: 6E.FUT, ZN.FUT, ES.FUT, NQ.FUT
• MedianPruner: 30-40% time savings via early trial termination
• GPU acceleration: RTX 3050 Ti with batch size <= 230

Key Deliverables

File	Purpose
tuning_config_ppo_comprehensive.yaml	PPO search space configuration
run_ppo_comprehensive_tuning.sh	End-to-end execution script
hyperparameter_tuner_ppo_enhanced.py	Enhanced Python tuner with composite objective
PPO_COMPREHENSIVE_TUNING_GUIDE.md	This guide

---

Prerequisites

1. System Requirements

• GPU: NVIDIA RTX 3050 Ti (4GB VRAM) with CUDA 12.1
• Memory: 16GB RAM minimum
• Storage: 50GB free space for checkpoints/logs
• OS: Linux (validated on Ubuntu)

2. Data Files Required

All 4 symbol data files must be present:

test_data/6E.FUT_2024-01-02.dbn.zst   # Euro FX (1,661 bars)
test_data/ZN.FUT_2024-01-02.dbn.zst   # Treasury (28,935 bars)
test_data/ES.FUT_2024-01-02.dbn.zst   # S&P 500 (1,674 bars)
test_data/NQ.FUT_2024-01-02.dbn.zst   # Nasdaq (available)

3. Services Running

# Start required services
docker-compose up -d postgres redis vault

# Start ML Training Service
cargo run -p ml_training_service --release &

# Verify services
grpc_health_probe -addr=localhost:50054  # ML Training Service
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c '\dt'

4. Baseline Checkpoint

The baseline checkpoint from Agent 79 analysis must exist:

ml/trained_models/production/ppo_real_data/ppo_actor_epoch_380.safetensors

Baseline Performance:

• Explained Variance: 0.4469 (only 0.0531 from optimal 0.5)
• Training Phase: Late-stage refinement (epoch 350-500)
• Status: EXCELLENT (within 5% of theoretical optimal)

---

Search Space Configuration

Hyperparameters to Optimize (6 parameters)

Hyperparameter	Search Space	Choices	Rationale
Learning Rate	[0.0001, 0.0003, 0.001]	3	Critical for convergence speed
Batch Size	[32, 64, 128, 256]	4	GPU memory vs sample efficiency
Gamma	[0.95, 0.99]	2	Discount factor for returns
GAE Lambda	[0.9, 0.95, 0.98]	3	Advantage estimation smoothing
Clip Epsilon	[0.1, 0.2, 0.3]	3	PPO policy update constraint
Entropy Coef	[0.001, 0.01, 0.1]	3	Exploration vs exploitation

Total Combinations: 3 × 4 × 2 × 3 × 3 × 3 = 648 possible configurations Sampling Strategy: TPE (Tree-structured Parzen Estimator) - intelligent sampling Trials: 50 (covers ~7.7% of search space with intelligent prioritization)

Fixed Parameters (Agent 32 Fix)

These are NOT optimized (fixed based on Agent 32 policy collapse fix):

policy_hidden_dims: [128, 64]
value_hidden_dims: [128, 64]
value_loss_coef: 1.0        # Prioritize value learning
rollout_steps: 2048
minibatch_size: 64
num_ppo_epochs: 10
max_grad_norm: 0.5

---

Objective Function

Composite Objective Formula

Objective = 0.7 × Sharpe Ratio + 0.3 × Explained Variance

Rationale

Metric	Weight	Rationale
Sharpe Ratio	70%	Primary metric for risk-adjusted returns (production goal)
Explained Variance	30%	Critical for PPO value network accuracy (convergence indicator)

Why Composite?

1. Sharpe Ratio Alone: May converge poorly (value network struggles)
2. Explained Variance Alone: May not maximize trading performance
3. Combined: Balances trading performance + model convergence quality

Example Calculation

# Trial 1 Results
sharpe_ratio = 1.5
explained_variance = 0.42

objective = 0.7 * 1.5 + 0.3 * 0.42 = 1.05 + 0.126 = 1.176

# Trial 2 Results
sharpe_ratio = 1.8
explained_variance = 0.38

objective = 0.7 * 1.8 + 0.3 * 0.38 = 1.26 + 0.114 = 1.374

# Trial 2 wins despite lower explained variance!

---

Execution Steps

Quick Start (Recommended)

cd /home/jgrusewski/Work/foxhunt

# Run comprehensive tuning (8-12 hours)
./run_ppo_comprehensive_tuning.sh

This script handles:

• ✅ Prerequisites validation
• ✅ Service health checks
• ✅ Data file verification
• ✅ GPU detection
• ✅ Job submission via TLI
• ✅ Progress monitoring
• ✅ Results retrieval
• ✅ Summary report generation

Manual Execution (Advanced)

If you prefer manual control:

Step 1: Verify Prerequisites

# Check GPU
nvidia-smi

# Check data files
ls -lh test_data/*.dbn.zst

# Check baseline
ls -lh ml/trained_models/production/ppo_real_data/ppo_actor_epoch_380.safetensors

# Check services
grpc_health_probe -addr=localhost:50054

Step 2: Start Tuning Job

# Via TLI (recommended)
cargo run -p tli -- tune start \
  --model PPO \
  --trials 50 \
  --config tuning_config_ppo_comprehensive.yaml

# Capture job ID
export JOB_ID="<uuid-from-output>"

Step 3: Monitor Progress

# Real-time monitoring
cargo run -p tli -- tune status --job-id $JOB_ID --watch

# Check every 5 minutes
watch -n 300 "cargo run -p tli -- tune status --job-id $JOB_ID"

Step 4: Retrieve Best Hyperparameters

# After completion (8-12 hours)
cargo run -p tli -- tune best --job-id $JOB_ID

# Stop if needed (graceful shutdown)
cargo run -p tli -- tune stop --job-id $JOB_ID

---

Expected Timeline

Duration Breakdown

Phase	Duration	Description
Setup	5 min	Prerequisites check, service validation
Trial 1-5	60-90 min	No pruning (baseline establishment)
Trial 6-50	6-10 hours	MedianPruner active (30-40% time savings)
Analysis	10 min	Best hyperparameters retrieval
Total	8-12 hours	Full 50-trial optimization

Per-Trial Estimates

• Successful Trial: 10-15 minutes (50 epochs @ 12-18 sec/epoch)
• Pruned Trial: 3-5 minutes (stopped early by MedianPruner)
• Pruning Rate: 30-40% of trials (15-20 trials pruned)
• Average: ~10 minutes per trial effective

Progress Checkpoints

Monitor these milestones during execution:

Trials	Expected Time	Status Check
5	1 hour	Baseline established, MedianPruner activating
10	2 hours	Pruning active, best trial identified
25	5 hours	Halfway point, convergence patterns visible
40	8 hours	Late-stage refinement, diminishing returns
50	10 hours	Completion, final analysis

---

Output Files

Generated Files

ml/trained_models/tuning/ppo_comprehensive/
├── job_id.txt                      # Job UUID for reference
├── tuning_execution.log            # Full execution log
├── best_hyperparameters.txt        # Best hyperparameters (plaintext)
├── TUNING_SUMMARY_REPORT.md        # Comprehensive summary
├── checkpoints/                    # Checkpoints from trials
│   ├── trial_0_epoch_50.safetensors
│   ├── trial_1_epoch_50.safetensors
│   └── ...
├── plots/                          # Optuna visualization plots
│   ├── optimization_history.png
│   ├── param_importances.png
│   ├── parallel_coordinate.png
│   └── contour_plot.png
└── logs/
    ├── trial_0.log
    ├── trial_1.log
    └── ...

Key Output Files

1. best_hyperparameters.txt: Use this for production training
2. TUNING_SUMMARY_REPORT.md: Share with team for analysis
3. tuning_execution.log: Debugging and audit trail

---

Interpreting Results

Success Criteria

Metric	Target	Status
Composite Objective	> baseline	Check best_hyperparameters.txt
Sharpe Ratio	> 1.5	Risk-adjusted returns improvement
Explained Variance	> 0.45	Value network convergence quality
Combined Improvement	> 5%	Overall performance gain

Baseline Comparison

Baseline (Epoch 380):

• Explained Variance: 0.4469
• Sharpe Ratio: TBD (measure during tuning)
• Training: 500 epochs (5.6 minutes)

Target (Tuned Model):

• Explained Variance: > 0.47 (5% improvement)
• Sharpe Ratio: > baseline (risk-adjusted gains)
• Training: 50 epochs (early stopping efficiency)

Value Network Convergence Analysis

Monitor these indicators in the summary report:

1. Explained Variance Trajectory: Should approach 0.47-0.50
2. Policy Stability: KL divergence < 0.0002 (healthy)
3. Loss Dynamics: Policy loss stable, value loss decreasing
4. Entropy Decay: Gradual reduction (exploration → exploitation)

---

Troubleshooting

Common Issues

1. GPU Out of Memory

Symptom: Trial fails with CUDA OOM error Solution: Batch size auto-limited to 230 for RTX 3050 Ti

# Check GPU memory
nvidia-smi

# If OOM persists, reduce batch size in config
# Edit tuning_config_ppo_comprehensive.yaml:
batch_size:
  choices: [32, 64, 128]  # Remove 256

2. Service Not Running

Symptom: gRPC connection refused Solution: Restart ML Training Service

# Check status
grpc_health_probe -addr=localhost:50054

# Restart if needed
pkill -f ml_training_service
cargo run -p ml_training_service --release &

3. Missing Data Files

Symptom: "Data file not found" error Solution: Download missing symbols

# Check data files
ls test_data/*.dbn.zst

# If missing, download from Databento
# See: ML_DATA_DOWNLOAD_GUIDE.md

4. Slow Progress

Symptom: Trials taking >20 minutes each Solution: Check GPU utilization

# Monitor GPU usage
watch -n 1 nvidia-smi

# If GPU idle (<10% utilization):
# 1. Check CUDA_HOME environment
# 2. Verify candle-core GPU compilation
# 3. Check use_gpu flag in config

5. Trial Failures

Symptom: Multiple trials failing with training errors Solution: Check logs for patterns

# View trial logs
tail -f ml/trained_models/tuning/ppo_comprehensive/logs/trial_*.log

# Common causes:
# - NaN in loss (reduce learning rate)
# - Policy collapse (increase entropy_coef)
# - Gradient explosion (reduce learning rate)

---

Next Steps After Tuning

1. Validate Best Hyperparameters

# Run 500-epoch production training with best hyperparameters
cargo run -p ml --example train_ppo_production \
  --learning-rate <best_lr> \
  --batch-size <best_batch> \
  --gamma <best_gamma> \
  --gae-lambda <best_lambda> \
  --clip-epsilon <best_clip> \
  --entropy-coef <best_entropy>

2. Checkpoint Analysis

# Analyze checkpoints from production training
cargo run -p ml --example analyze_ppo_checkpoints \
  --checkpoint-dir ml/trained_models/production/ppo_tuned/

3. Cross-Symbol Backtesting

# Test trained model on all 4 symbols
cargo run -p backtesting_service --example comprehensive_backtest \
  --model-path ml/trained_models/production/ppo_tuned/ppo_final_epoch500.safetensors \
  --symbols 6E.FUT,ZN.FUT,ES.FUT,NQ.FUT

4. Production Deployment

After validation:

# Deploy to inference service
cp ml/trained_models/production/ppo_tuned/ppo_final_epoch500.safetensors \
   services/inference/models/ppo_production_v2.safetensors

# Update model registry
cargo run -p ml --example model_registry_api register \
  --model-path models/ppo_production_v2.safetensors \
  --version 2.0 \
  --description "PPO with optimized hyperparameters (50-trial tuning)"

---

Performance Expectations

Improvement Targets

Based on similar hyperparameter tuning studies:

Metric	Baseline	Target	Improvement
Explained Variance	0.4469	0.47+	+5-10%
Sharpe Ratio	TBD	+10-20%	Risk-adjusted gains
Value Network Loss	200-220	150-180	Better convergence
Policy Stability	Good	Excellent	Lower KL divergence

Realistic Outcomes

• Best Case: 15-20% improvement in composite objective
• Expected Case: 5-10% improvement (still significant)
• Worst Case: Marginal improvement (baseline already excellent)

Note: The baseline (Epoch 380) is already EXCELLENT (explained_var=0.4469, only 0.0531 from optimal 0.5). Significant improvements may be limited, but any gain is valuable for production trading.

---

References

Related Documentation

• Baseline Analysis: PPO_CHECKPOINT_ANALYSIS_REPORT.md
• Agent 32 Fix: AGENT32_PPO_FIX_SUMMARY.md
• Convergence Analysis: CONVERGENCE_ANALYSIS_REPORT.md
• ML Training Roadmap: ML_TRAINING_ROADMAP.md
• Optuna Integration: OPTUNA_TUNING_INTEGRATION_REPORT.md

Key Findings from Agent 79

• Epoch 380: Best checkpoint (explained_var=0.4469)
• Top 10 Checkpoints: All in "Balanced" risk profile (0.40-0.45)
• Training Quality: 100% policy update rate, zero NaN occurrences
• Value Network: Improved 33.2% from epoch 10 to 500
• Policy Stability: KL divergence mean 0.000138 (healthy)

---

FAQ

Q1: Why 50 trials instead of 100+?

A: With 648 possible combinations and TPE intelligent sampling, 50 trials covers critical regions of the search space efficiently. MedianPruner further optimizes by stopping unpromising trials early.

Q2: Why early stopping at epoch 50 vs 500?

A: For hyperparameter search, we prioritize breadth (50 trials) over depth (500 epochs per trial). Early stopping at epoch 50 still reveals which hyperparameter sets converge best. Once we find optimal hyperparameters, we run full 500-epoch production training.

Q3: Can I pause/resume tuning?

A: Yes! Optuna JournalStorage persists state to MinIO. If interrupted:

# Resume with same job ID
cargo run -p tli -- tune start \
  --model PPO \
  --trials 50 \
  --config tuning_config_ppo_comprehensive.yaml \
  --resume $JOB_ID

Q4: How do I know if tuning is working?

A: Monitor these indicators:

1. Trial Progress: Should complete 4-6 trials/hour
2. Pruning: 30-40% of trials pruned (MedianPruner active)
3. Best Value Improving: Composite objective increasing over trials
4. No Repeated Failures: <5% trial failure rate acceptable

Q5: What if results are worse than baseline?

A: This is unlikely but possible. If composite objective < baseline:

1. Verify Baseline Measurement: Re-run baseline with same validation data
2. Check Objective Weights: Try 0.8 sharpe + 0.2 explained_var
3. Expand Search Space: Add more learning rate options
4. Increase Trials: Run 100 trials for better coverage

---

Support

For issues or questions:

1. Check Logs: ml/trained_models/tuning/ppo_comprehensive/tuning_execution.log
2. Review Error Messages: Trial-specific logs in logs/trial_*.log
3. Verify Services: grpc_health_probe -addr=localhost:50054
4. GPU Status: nvidia-smi

---

Mission Status: ⚙️ READY TO EXECUTE Estimated Completion: 8-12 hours from start Next Action: Run ./run_ppo_comprehensive_tuning.sh

Good luck! 🚀