2df1ea92e1
BREAKING CHANGES: - Removed orphaned dqn.rs monolithic trainer (4,975 lines) - Removed orphaned dqn_ensemble.rs module (816 lines) - Removed orphaned tft.rs and tft_complete_int8_integration_test.rs - TFT trainer split into modular directory structure DQN Module Refactoring: - Split trainers/dqn.rs into modular structure (config.rs, statistics.rs, trainer.rs) - Fixed hyperopt 39D search space (continuous params only) - Boolean flags (use_dueling, use_double_dqn, use_per, use_noisy_nets) are now FIXED architectural decisions - use_distributional defaults to false (Candle BUG #36 - scatter_add gradient issues) Clean Module Structure: - ml/src/trainers/dqn/ directory with proper mod.rs exports - ml/src/trainers/tft/ directory with config.rs, types.rs, model.rs, trainer.rs, tests.rs - All P0 features validated: TD-error clamping, batch diversity, LR scheduler, priority staleness Documentation: - Added comprehensive docs in docs/codebase-cleanup/ - ADR-001 for DQN refactoring decisions - Rainbow DQN component matrix and quick reference guides Build Status: Compiles with zero errors 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
57 lines
2.4 KiB
Bash
Executable File
57 lines
2.4 KiB
Bash
Executable File
#!/bin/bash
|
|
set -e
|
|
|
|
echo "========================================="
|
|
echo "PPO Production Training (CORRECTED - Hyperopt LR)"
|
|
echo "========================================="
|
|
echo ""
|
|
|
|
# Set PYTHONPATH
|
|
export PYTHONPATH="/home/jgrusewski/Work/foxhunt:$PYTHONPATH"
|
|
|
|
# Activate venv
|
|
source .venv/bin/activate
|
|
|
|
# Generate timestamp for output directory
|
|
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
|
|
|
|
echo "Configuration:"
|
|
echo " Policy learning rate: 0.000001 (1e-6 from hyperopt, ultra-conservative)"
|
|
echo " Value learning rate: 0.001 (aggressive, from hyperopt best trial)"
|
|
echo " Batch size: 64"
|
|
echo " Epochs: 10000"
|
|
echo " Early stopping: DISABLED"
|
|
echo " Output: /runpod-volume/ml_training/ppo_production_${TIMESTAMP}"
|
|
echo ""
|
|
|
|
# Deploy PPO production training with CORRECTED dual learning rates from hyperopt
|
|
# ✅ DUAL LEARNING RATES IMPLEMENTED (2025-11-01)
|
|
# The binary now supports --policy-lr and --value-lr flags separately
|
|
python3 scripts/runpod_deploy.py \
|
|
--gpu-type "RTX A4000" \
|
|
--image "jgrusewski/foxhunt-hyperopt:latest" \
|
|
--command "train_ppo_parquet --parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet --epochs 10000 --policy-lr 0.000001 --value-lr 0.001 --batch-size 64 --output-dir /runpod-volume/ml_training/ppo_production_${TIMESTAMP} --no-early-stopping"
|
|
|
|
echo ""
|
|
echo "✅ PPO production training deployment initiated (DUAL LEARNING RATES)"
|
|
echo "Monitor logs: python3 scripts/python/runpod/monitor_logs.py <pod_id>"
|
|
echo "Expected duration: 30-90 minutes"
|
|
echo "Expected cost: \$0.12-\$0.38 @ \$0.25/hr (RTX A4000)"
|
|
echo ""
|
|
echo "DUAL LEARNING RATES APPLIED (Hyperopt Best Trial #1, obj=2.4023):"
|
|
echo " • Policy LR: 0.000001 (1e-6, ultra-conservative - 1000x smaller)"
|
|
echo " • Value LR: 0.001 (aggressive, 3.3x larger than policy)"
|
|
echo " • Clip epsilon: 0.1126 (conservative vs 0.2 default)"
|
|
echo " • Entropy coeff: 0.006142 (low exploration)"
|
|
echo ""
|
|
echo "Why dual LRs matter:"
|
|
echo " • Policy network: Slow updates to prevent catastrophic forgetting"
|
|
echo " • Value network: Fast updates to match actual returns"
|
|
echo " • Single LR (0.001) caused loss stagnation at 1.158-1.159 in Pod 0hczpx9nj1ub88"
|
|
echo " • Asymmetric 1000x ratio is CRITICAL for PPO convergence"
|
|
echo ""
|
|
echo "Status: ✅ READY FOR DEPLOYMENT"
|
|
echo " • Binary supports --policy-lr and --value-lr flags"
|
|
echo " • Implementation verified in train_ppo_parquet.rs (lines 57-63)"
|
|
echo " • Dual optimizers initialized correctly (ppo/ppo.rs lines 698-732)"
|