foxhunt/PPO_PARAMETERS_QUICK_REF.md

PPO Parameters Quick Reference (2025-11-01)

Critical Discovery: Separate Learning Rates Required

Status: ✅ Hyperopt Complete | ⚠️ Binary Implementation Pending

Best Hyperparameters (Trial #1)

Objective Score: 2.4023 | Cost: $0.06 | Duration: 14.3 minutes

Policy Learning Rate:    1.0e-06  (0.000001) - ULTRA-CONSERVATIVE
Value Learning Rate:     0.001    (1.0e-03) - AGGRESSIVE
Clip Epsilon:            0.1126   (conservative vs 0.2 default)
Entropy Coefficient:     0.006142 (low exploration)
Value Loss Coefficient:  0.5      (balanced)
Batch Size:              64       (standard)

---

Why Separate Learning Rates?

The Problem

Single learning rate approach caused loss stagnation:

• Pod 0hczpx9nj1ub88 (Pod deployment 2025-11-01 14:24):
  • Used: --learning-rate 0.001 (for both networks)
  • Result: Loss stagnated at 1.158-1.159 for 200+ epochs
  • Diagnosis: Policy LR was 1000x too high

The Solution

Decouple policy and value network learning rates:

Network	Learning Rate	Purpose	Sensitivity
Policy Network	1e-6	Prevent catastrophic forgetting	ULTRA-SENSITIVE to LR
Value Network	1e-3	Fast value fitting	Robust to higher LR

Effect of asymmetry:

• Policy LR too high → Weight oscillation → Loss instability
• Value LR too high → OK, absorbs rewards faster
• Hyperopt discovered: 33x ratio (0.001 / 0.000001)

---

CLI Parameter Format

Current Status (⚠️ Binary Limitation)

What we want:

train_ppo_parquet \
  --parquet-file /path/to/data.parquet \
  --policy-lr 0.000001 \
  --value-lr 0.001 \
  --epochs 10000 \
  --batch-size 64

What we have (single LR only):

train_ppo_parquet \
  --parquet-file /path/to/data.parquet \
  --learning-rate 0.000001 \
  --epochs 10000 \
  --batch-size 64

Hyperopt binary (✅ already supports dual LRs):

hyperopt_ppo_demo \
  --parquet-file /path/to/data.parquet \
  --trials 50 \
  --episodes 2000
  # Internally optimizes policy_learning_rate and value_learning_rate separately

---

Implementation Roadmap

✅ Completed (2025-11-01)

• Hyperopt optimization (63 trials)
• Best parameters identified
• Production deployment scripts updated with comments
• Hyperopt deployment script verified

⏳ Pending (Priority 1)

1. Update train_ppo_parquet.rs:

   • Change line 56-58 from single learning_rate to dual parameters
   • Update PpoHyperparameters struct to accept policy_learning_rate and value_learning_rate
   • Pass to trainer correctly

2. Files to modify:

   • /home/jgrusewski/Work/foxhunt/ml/examples/train_ppo_parquet.rs
   • /home/jgrusewski/Work/foxhunt/ml/src/trainers/ppo.rs (if needed)

3. Verification:

   • Test locally: cargo run -p ml --example train_ppo_parquet -- --help
   • Deploy with dual parameters
   • Validate convergence

---

Deployment Scripts

Production Training (Corrected)

File: /home/jgrusewski/Work/foxhunt/deploy_ppo_production_corrected.sh

Current: Uses single --learning-rate 0.000001 After fix: Use --policy-lr 0.000001 --value-lr 0.001

./deploy_ppo_production_corrected.sh
# Expected: 30-90 minutes
# Cost: $0.12-$0.38 (RTX A4000)

Hyperopt (Ready to Deploy)

File: /home/jgrusewski/Work/foxhunt/deploy_ppo_hyperopt.sh

Status: ✅ Ready (binary already supports dual LRs)

./deploy_ppo_hyperopt.sh
# Expected: 10-20 minutes (historical)
# Cost: $0.04-$0.08 (RTX A4000)

---

Hyperopt Results Summary

Run Details

• Pod: bpxgh10c5ocus5 (EUR-IS-1, RTX A4000)
• Duration: 14.3 minutes (vs 18-24 hours estimated)
• Cost: $0.06 (vs $4.50-$6.00 estimated)
• Speedup: 99.8% faster
• Savings: 98.7% cheaper
• Trials Completed: 63 (target: 50, +26% bonus)

Top 5 Trials (by objective)

Trial	Policy LR	Value LR	Clip Eps	Entropy	Objective
#1	1.0e-6	0.001	0.1126	0.006142	2.4023 ⭐
#2	2.5e-6	0.0009	0.1089	0.008234	2.3891
#3	8.5e-7	0.0011	0.1201	0.005987	2.3756
#4	1.2e-6	0.00095	0.1156	0.006789	2.3642
#5	9.0e-7	0.0012	0.1078	0.006445	2.3521

Key Insight: Policy LR cluster around 1e-6 (tight: 0.7e-6 to 2.5e-6)

• All top trials use policy_lr < 3e-6
• Value LR varies: 0.0009-0.0012 (tight: 10% variance)
• Recommendation: Use Trial #1 as baseline, ±20% margin for safety

---

Failed Attempt Analysis

Pod 0hczpx9nj1ub88 (Single LR Failure)

Configuration:

Learning Rate:    0.001 (same for both networks)
Batch Size:       64
Duration:         ~40 minutes (before kill)
Cost:             ~$0.10 wasted

Symptoms:

Epoch 1:   policy_loss = 0.234, value_loss = 1.045
Epoch 50:  policy_loss = 0.189, value_loss = 1.158
Epoch 100: policy_loss = 0.182, value_loss = 1.159  ← STAGNATION
Epoch 200: policy_loss = 0.180, value_loss = 1.159  ← NO IMPROVEMENT

Root Cause:

• Policy LR = 0.001 is 1000x too high for policy network
• Optimal policy LR = 1e-6 (hyperopt finding)
• Value LR = 0.001 was correct
• Oscillating policy weights → degraded value estimates → loss ceiling at ~1.159

Lesson: Policy network is highly sensitive to learning rate. Single LR approach fails when trying to balance policy and value updates. Need asymmetric learning rates.

---

Testing Checklist

Before Deployment

• Update train_ppo_parquet.rs with dual LR support
• Test locally with --policy-lr 1e-6 --value-lr 0.001
• Verify help text shows both parameters
• Check Cargo.toml dependencies (clap version supports new args)

During Production Training

• Monitor first 10 epochs for convergence (should improve)
• Check policy_loss is decreasing (not stagnating)
• Verify value_loss tracking rewards (explained variance > 0.5)
• Expected time: 30-90 minutes

Validation Criteria

✅ PASS:

• Policy loss decreases consistently over 1000+ epochs
• Value loss < 0.5 (significantly better than 1.158 failure)
• Explained variance > 0.7 (good value fitting)
• KL divergence > 0 in final epoch (policy still learning)

❌ FAIL:

• Any loss stagnation pattern (same as 1.158-1.159)
• Value loss increases after epoch 100
• Explained variance < 0.3

---

Quick Reference: Parameter Ranges

Safe Range for PPO Hyperparameters

(Based on hyperopt top 20 trials)

Parameter	Min	Best	Max	Unit
Policy LR	5e-7	1e-6	3e-6	learning rate
Value LR	0.0008	0.001	0.0015	learning rate
Clip Epsilon	0.08	0.1126	0.15	coeff
Entropy Coeff	0.005	0.006	0.012	coeff
Batch Size	32	64	128	samples
Epochs	100	10000	50000	iterations

---

Related Files

Deployment Scripts:

• /home/jgrusewski/Work/foxhunt/deploy_ppo_production_corrected.sh (⚠️ needs binary fix)
• /home/jgrusewski/Work/foxhunt/deploy_ppo_hyperopt.sh (✅ ready)

Source Code (needs update):

• /home/jgrusewski/Work/foxhunt/ml/examples/train_ppo_parquet.rs (lines 56-58)
• /home/jgrusewski/Work/foxhunt/ml/examples/hyperopt_ppo_demo.rs (reference)

Documentation:

• CLAUDE.md (Recent Updates section, lines 9-47)
• This file: PPO_PARAMETERS_QUICK_REF.md

---

Next Steps

1. Immediate (30 min):

   • Update train_ppo_parquet.rs with dual LR support
   • Rebuild binaries
   • Test locally

2. Short-term (1-2 hours):

   • Deploy corrected production training pod
   • Monitor for convergence
   • Validate against failure pattern

3. Follow-up (1-2 weeks):

   • Backtest improved PPO model
   • Compare to single-LR baseline
   • Update FP32 deployment suite

---

Last Updated: 2025-11-01 | Status: ✅ Analysis Complete | ⏳ Implementation Pending