foxhunt/PPO_PRODUCTION_DEPLOYMENT_NOTES.md

PPO Production Training Deployment Notes

Date: 2025-11-01 Task: Deploy PPO production training with best hyperparameters from hyperopt

Hyperparameters from Hyperopt

Best hyperparameters found:

• policy_learning_rate: 1.0e-06
• value_learning_rate: 0.001
• clip_epsilon: 0.1126
• value_loss_coeff: 0.5
• entropy_coeff: 0.006142

Implementation Limitation

The current train_ppo_parquet binary uses PpoHyperparameters struct which has:

• Single learning_rate field (NOT separate policy_lr and value_lr)
• Hardcoded values for clip_epsilon (0.2), vf_coef (0.5), ent_coef (0.01)

The hyperopt PPO adapter uses low-level PPOConfig which supports separate learning rates.

Deployment Approach

Given the time constraint and "REUSE existing infrastructure" principle, we:

1. Updated Dockerfile.foxhunt-build to include train_ppo_parquet binary

   • Added --example train_ppo_parquet to cargo build
   • Added binary copy and strip commands
   • Total changes: 3 lines in build stage, 2 lines in runtime stage

2. Created deploy_ppo_production.sh deployment script

   • Uses train_ppo_parquet with closest approximation of optimal hyperparameters
   • Learning rate: 0.001 (using value_lr from hyperopt, more critical for PPO stability)
   • Epochs: 10000 (production run)
   • Batch size: 64 (suitable for RTX A4000 16GB VRAM)
   • Early stopping: DISABLED (to complete full training)

3. Hyperparameter Approximation

   • ✅ Learning rate: 0.001 (exact match to value_lr)
   • ❌ clip_epsilon: 0.2 (hardcoded, optimal was 0.1126)
   • ✅ value_loss_coeff: 0.5 (exact match to optimal)
   • ❌ entropy_coeff: 0.01 (hardcoded, optimal was 0.006142)
   • ❌ Policy LR: Not separately controllable (optimal was 1e-6)

Training Configuration

train_ppo_parquet \
  --parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet \
  --epochs 10000 \
  --learning-rate 0.001 \
  --batch-size 64 \
  --output-dir /runpod-volume/ml_training/ppo_production \
  --no-early-stopping

Deployment Details

• GPU: RTX A4000 (16GB VRAM, $0.25/hr)
• Data: ES_FUT_180d.parquet (180 days of E-mini S&P 500 futures)
• Expected Duration: 30-90 minutes
• Expected Cost: $0.12-$0.38
• Checkpoint Directory: /runpod-volume/ml_training/ppo_production
• Checkpoint Interval: Every 10 epochs (automatic in trainer)

Docker Build

Building enhanced image: jgrusewski/foxhunt-hyperopt:latest

• Base image: nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04
• GLIBC: 2.35 (Ubuntu 22.04)
• Binaries included:
  • hyperopt_mamba2_demo
  • hyperopt_dqn_demo
  • hyperopt_ppo_demo
  • hyperopt_tft_demo
  • train_dqn
  • train_ppo_parquet (NEW)

Future Enhancement Recommendation

For exact hyperparameter control, create train_ppo_production.rs that:

• Uses low-level PPOConfig (same as hyperopt adapter)
• Accepts all hyperparameters via CLI:

  --policy-lr 1e-6
  --value-lr 0.001
  --clip-epsilon 0.1126
  --value-loss-coeff 0.5
  --entropy-coeff 0.006142
  

• Reuses parquet loading and 225-feature extraction from train_ppo_parquet

Estimated effort: 2-4 hours (new binary + Docker rebuild + testing)

Files Modified

1. /home/jgrusewski/Work/foxhunt/Dockerfile.foxhunt-build - Added train_ppo_parquet binary
2. /home/jgrusewski/Work/foxhunt/deploy_ppo_production.sh - Deployment script

Execution Steps

1. Build Docker image (IN PROGRESS):

   DOCKER_BUILDKIT=1 docker build -f Dockerfile.foxhunt-build -t jgrusewski/foxhunt-hyperopt:latest .
   

2. Deploy to Runpod:

   ./deploy_ppo_production.sh
   

3. Monitor training:

   python3 scripts/python/runpod/monitor_logs.py <pod_id>
   

4. Retrieve results from S3:

   aws s3 ls s3://se3zdnb5o4/models/ppo_production/ --profile runpod --recursive
   

Success Criteria

• Training completes 10,000 epochs
• Policy loss decreasing over time
• Value loss stabilizing
• Explained variance > 0.4
• Final checkpoint saved: ppo_checkpoint_epoch_10000.safetensors
• No NaN/Inf errors in training metrics

Notes

• This deployment uses approximated hyperparameters due to binary limitations
• Performance will likely be good but not optimal
• For optimal performance, rebuild with train_ppo_production binary supporting all hyperparameters