foxhunt/docs/guides/QUICK_START_TRAINING.md

Quick Start: ML Model Training

Time to Complete: 30-60 minutes (initial setup) + 4-6 weeks (training) Prerequisites: Docker, RTX 3050 Ti GPU, 16GB RAM Goal: Train your first ML model (DQN) with real market data

---

Step 1: Environment Setup (5 minutes)

Start Infrastructure

cd /home/jgrusewski/Work/foxhunt
docker-compose up -d

Verify Services

docker-compose ps
# Should show: postgres, redis, vault, prometheus, grafana all healthy

Run Database Migrations

cargo sqlx migrate run

---

Step 2: GPU Validation (2 minutes)

Check GPU

nvidia-smi
# Should show: RTX 3050 Ti, 4GB VRAM available

Verify CUDA

nvcc --version
# Should show: CUDA 11.8 or higher

---

Step 3: Run GPU Benchmark (30-60 minutes)

Purpose: Determine if local training (4-6 weeks) or cloud GPU ($250/week) is optimal

cargo run -p ml --example gpu_training_benchmark --release

Output: JSON report with recommendation

• local_gpu: Train on RTX 3050 Ti (4-6 weeks)
• cloud_gpu: Rent A100 GPU (1-2 weeks, $250/week)
• either: User choice based on cost analysis

---

Step 4: Download Market Data (10 minutes)

Option A: Use Existing Test Data (Quick Start)

ls test_data/
# Available: ES.FUT (1,674 bars), ZN.FUT (28,935 bars), 6E.FUT (29,937 bars)

Option B: Download 90-Day Data (Recommended for Production)

# Cost: ~$2, Size: ~180,000 bars
# Symbols: ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT
# Follow: /home/jgrusewski/Work/foxhunt/90_DAY_DATA_EXPANSION_PLAN.md

---

Step 5: Train Your First Model (DQN)

Start Training (Local GPU)

# Terminal 1: Start ML Training Service
cargo run -p ml_training_service

# Terminal 2: Start API Gateway
cargo run -p api_gateway

# Terminal 3: Login with TLI
tli login --username admin --password <password>

# Start DQN Training
tli train start --model DQN --symbol ES.FUT --epochs 100

Monitor Progress

# Watch training in real-time
tli train status --job-id <uuid> --watch

# Streaming progress updates
# Epoch 1/100: Loss 0.5234, Reward 120.5, ETA 4h 23m
# Epoch 2/100: Loss 0.4891, Reward 135.2, ETA 4h 18m
# ...

Expected Timeline (RTX 3050 Ti)

• Epoch Duration: ~2-5 minutes per epoch
• 100 Epochs: 3-8 hours (depends on batch size)
• Full Training: 2-3 days for optimal convergence

---

Step 6: Checkpoint Analysis

List Checkpoints

tli checkpoints list --model DQN

Quick Analysis

cargo run -p ml --example quick_checkpoint_analysis --release

Deep Dive Analysis

cargo run -p ml --example analyze_dqn_checkpoints --release

Output:

• Top 10 checkpoints ranked by Sharpe ratio
• Explained variance trajectory
• Convergence analysis

---

Step 7: Select Best Checkpoint

Use Framework

# See: /home/jgrusewski/Work/foxhunt/docs/CHECKPOINT_SELECTION_FRAMEWORK.md

# Criteria:
# 1. Sharpe Ratio > 1.5 (risk-adjusted returns)
# 2. Win Rate > 55% (prediction accuracy)
# 3. Max Drawdown < 15% (risk control)
# 4. Explained Variance > 0.7 (model fit)

Load Best Checkpoint

tli checkpoints load --checkpoint-id <best-checkpoint-uuid>

---

Step 8: Backtest Strategy

Run Backtest

tli backtest run \
  --strategy dqn_strategy \
  --symbol ES.FUT \
  --start 2024-01-01 \
  --end 2024-12-31 \
  --checkpoint-id <best-checkpoint-uuid>

Review Results

tli backtest results --backtest-id <uuid>

# Expected Output:
# Sharpe Ratio: 1.85
# Win Rate: 58.3%
# Max Drawdown: 12.4%
# Total PnL: $125,450
# Number of Trades: 1,247

---

Step 9: Paper Trading (Safe Live Testing)

Deploy Paper Trading

# See: /home/jgrusewski/Work/foxhunt/PAPER_TRADING_DEPLOYMENT_PLAN.md

# 1. Configure paper trading account
# 2. Deploy DQN model with best checkpoint
# 3. Monitor for 2-4 weeks
# 4. Validate Sharpe ratio > 1.5 in live conditions

---

Step 10: Production Deployment

Prerequisites

• ✅ Paper trading validated (2-4 weeks)
• ✅ Sharpe ratio > 1.5 in live conditions
• ✅ Max drawdown < 15%
• ✅ Risk limits configured
• ✅ Security audit complete

Deploy to Production

# See: /home/jgrusewski/Work/foxhunt/docs/PRODUCTION_DEPLOYMENT_RUNBOOK_V3.md

# 1. Blue-green deployment
# 2. Canary release (1% traffic)
# 3. Monitor for 48 hours
# 4. Gradual rollout to 100%

---

Troubleshooting

GPU Out of Memory

# Reduce batch size in training config
# Default: 64 → Try: 32 or 16

Training Too Slow

# Check GPU utilization
nvidia-smi -l 1

# If <80% utilization: Increase batch size
# If >95% utilization: Optimal (expected)

Checkpoint Not Found

# List all checkpoints
tli checkpoints list --model DQN

# Verify checkpoint directory
ls -lh ~/.foxhunt/checkpoints/DQN/

Poor Backtest Results (Sharpe < 1.0)

# Options:
# 1. Train longer (200-500 epochs)
# 2. Hyperparameter tuning (see tuning guide)
# 3. Try different model (PPO, MAMBA-2)
# 4. Add more training data (90 days recommended)

---

Next Steps

Train Additional Models

# PPO (2-3 days)
tli train start --model PPO --symbol ES.FUT --epochs 100

# MAMBA-2 (3-4 days, requires more VRAM)
tli train start --model MAMBA2 --symbol ES.FUT --epochs 100

# TFT (5-7 days, largest model)
tli train start --model TFT --symbol ES.FUT --epochs 100

Hyperparameter Tuning

# Optimize DQN hyperparameters (4-8 hours, 50 trials)
tli tune start --model DQN --trials 50 --watch

# See: /home/jgrusewski/Work/foxhunt/TUNING_QUICKSTART_GUIDE.md

Ensemble Models

# Combine multiple models for better performance
# See: /home/jgrusewski/Work/foxhunt/ENSEMBLE_IMPLEMENTATION_GUIDE.md

# Expected: Sharpe ratio 2.0-2.5 with ensemble (vs 1.5-2.0 single model)

---

Key Resources

Essential Documentation

• ML Infrastructure Guide - Master index
• GPU Benchmark Guide - GPU performance testing
• Checkpoint Selection Framework - How to choose best model
• Agent 78: DQN Training Success - Real example

Training Guides

• ML Training Roadmap - 4-6 week plan
• DQN Training Report - DQN specifics
• PPO Training Guide - PPO training
• Feature Engineering Report - 16 features + 10 indicators

---

Success Metrics

Training Success

• ✅ Training completes without OOM errors
• ✅ Loss decreasing over epochs
• ✅ Explained variance > 0.7
• ✅ Checkpoints saved every 10 epochs

Model Quality

• ✅ Sharpe ratio > 1.5
• ✅ Win rate > 55%
• ✅ Max drawdown < 15%
• ✅ Consistent performance across validation periods

Production Readiness

• ✅ Paper trading validates backtest results
• ✅ Sharpe ratio > 1.5 in live conditions
• ✅ Risk limits enforced
• ✅ Monitoring and alerting operational

---

Estimated Total Time:

• Setup: 30-60 minutes
• GPU Benchmark: 30-60 minutes
• DQN Training: 2-3 days
• Backtest + Analysis: 1-2 hours
• Paper Trading: 2-4 weeks
• Production Deployment: 1-2 days

Total: ~5-7 weeks from zero to production

Next Guide: Quick Start: Hyperparameter Tuning