foxhunt/docs/guides/QUICK_START_TRAINING.md

# Quick Start: ML Model Training

**Time to Complete**: 30-60 minutes (initial setup) + 4-6 weeks (training)
**Prerequisites**: Docker, RTX 3050 Ti GPU, 16GB RAM
**Goal**: Train your first ML model (DQN) with real market data

---

## Step 1: Environment Setup (5 minutes)

### Start Infrastructure
```bash
cd /home/jgrusewski/Work/foxhunt
docker-compose up -d
```

### Verify Services
```bash
docker-compose ps
# Should show: postgres, redis, vault, prometheus, grafana all healthy
```

### Run Database Migrations
```bash
cargo sqlx migrate run
```

---

## Step 2: GPU Validation (2 minutes)

### Check GPU
```bash
nvidia-smi
# Should show: RTX 3050 Ti, 4GB VRAM available
```

### Verify CUDA
```bash
nvcc --version
# Should show: CUDA 11.8 or higher
```

---

## Step 3: Run GPU Benchmark (30-60 minutes)

**Purpose**: Determine if local training (4-6 weeks) or cloud GPU ($250/week) is optimal

```bash
cargo run -p ml --example gpu_training_benchmark --release
```

**Output**: JSON report with recommendation
- `local_gpu`: Train on RTX 3050 Ti (4-6 weeks)
- `cloud_gpu`: Rent A100 GPU (1-2 weeks, $250/week)
- `either`: User choice based on cost analysis

---

## Step 4: Download Market Data (10 minutes)

### Option A: Use Existing Test Data (Quick Start)
```bash
ls test_data/
# Available: ES.FUT (1,674 bars), ZN.FUT (28,935 bars), 6E.FUT (29,937 bars)
```

### Option B: Download 90-Day Data (Recommended for Production)
```bash
# Cost: ~$2, Size: ~180,000 bars
# Symbols: ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT
# Follow: /home/jgrusewski/Work/foxhunt/90_DAY_DATA_EXPANSION_PLAN.md
```

---

## Step 5: Train Your First Model (DQN)

### Start Training (Local GPU)
```bash
# Terminal 1: Start ML Training Service
cargo run -p ml_training_service

# Terminal 2: Start API Gateway
cargo run -p api_gateway

# Terminal 3: Login with TLI
tli login --username admin --password <password>

# Start DQN Training
tli train start --model DQN --symbol ES.FUT --epochs 100
```

### Monitor Progress
```bash
# Watch training in real-time
tli train status --job-id <uuid> --watch

# Streaming progress updates
# Epoch 1/100: Loss 0.5234, Reward 120.5, ETA 4h 23m
# Epoch 2/100: Loss 0.4891, Reward 135.2, ETA 4h 18m
# ...
```

### Expected Timeline (RTX 3050 Ti)
- **Epoch Duration**: ~2-5 minutes per epoch
- **100 Epochs**: 3-8 hours (depends on batch size)
- **Full Training**: 2-3 days for optimal convergence

---

## Step 6: Checkpoint Analysis

### List Checkpoints
```bash
tli checkpoints list --model DQN
```

### Quick Analysis
```bash
cargo run -p ml --example quick_checkpoint_analysis --release
```

### Deep Dive Analysis
```bash
cargo run -p ml --example analyze_dqn_checkpoints --release
```

**Output**:
- Top 10 checkpoints ranked by Sharpe ratio
- Explained variance trajectory
- Convergence analysis

---

## Step 7: Select Best Checkpoint

### Use Framework
```bash
# See: /home/jgrusewski/Work/foxhunt/docs/CHECKPOINT_SELECTION_FRAMEWORK.md

# Criteria:
# 1. Sharpe Ratio > 1.5 (risk-adjusted returns)
# 2. Win Rate > 55% (prediction accuracy)
# 3. Max Drawdown < 15% (risk control)
# 4. Explained Variance > 0.7 (model fit)
```

### Load Best Checkpoint
```bash
tli checkpoints load --checkpoint-id <best-checkpoint-uuid>
```

---

## Step 8: Backtest Strategy

### Run Backtest
```bash
tli backtest run \
  --strategy dqn_strategy \
  --symbol ES.FUT \
  --start 2024-01-01 \
  --end 2024-12-31 \
  --checkpoint-id <best-checkpoint-uuid>
```

### Review Results
```bash
tli backtest results --backtest-id <uuid>

# Expected Output:
# Sharpe Ratio: 1.85
# Win Rate: 58.3%
# Max Drawdown: 12.4%
# Total PnL: $125,450
# Number of Trades: 1,247
```

---

## Step 9: Paper Trading (Safe Live Testing)

### Deploy Paper Trading
```bash
# See: /home/jgrusewski/Work/foxhunt/PAPER_TRADING_DEPLOYMENT_PLAN.md

# 1. Configure paper trading account
# 2. Deploy DQN model with best checkpoint
# 3. Monitor for 2-4 weeks
# 4. Validate Sharpe ratio > 1.5 in live conditions
```

---

## Step 10: Production Deployment

### Prerequisites
- ✅ Paper trading validated (2-4 weeks)
- ✅ Sharpe ratio > 1.5 in live conditions
- ✅ Max drawdown < 15%
- ✅ Risk limits configured
- ✅ Security audit complete

### Deploy to Production
```bash
# See: /home/jgrusewski/Work/foxhunt/docs/PRODUCTION_DEPLOYMENT_RUNBOOK_V3.md

# 1. Blue-green deployment
# 2. Canary release (1% traffic)
# 3. Monitor for 48 hours
# 4. Gradual rollout to 100%
```

---

## Troubleshooting

### GPU Out of Memory
```bash
# Reduce batch size in training config
# Default: 64 → Try: 32 or 16
```

### Training Too Slow
```bash
# Check GPU utilization
nvidia-smi -l 1

# If <80% utilization: Increase batch size
# If >95% utilization: Optimal (expected)
```

### Checkpoint Not Found
```bash
# List all checkpoints
tli checkpoints list --model DQN

# Verify checkpoint directory
ls -lh ~/.foxhunt/checkpoints/DQN/
```

### Poor Backtest Results (Sharpe < 1.0)
```bash
# Options:
# 1. Train longer (200-500 epochs)
# 2. Hyperparameter tuning (see tuning guide)
# 3. Try different model (PPO, MAMBA-2)
# 4. Add more training data (90 days recommended)
```

---

## Next Steps

### Train Additional Models
```bash
# PPO (2-3 days)
tli train start --model PPO --symbol ES.FUT --epochs 100

# MAMBA-2 (3-4 days, requires more VRAM)
tli train start --model MAMBA2 --symbol ES.FUT --epochs 100

# TFT (5-7 days, largest model)
tli train start --model TFT --symbol ES.FUT --epochs 100
```

### Hyperparameter Tuning
```bash
# Optimize DQN hyperparameters (4-8 hours, 50 trials)
tli tune start --model DQN --trials 50 --watch

# See: /home/jgrusewski/Work/foxhunt/TUNING_QUICKSTART_GUIDE.md
```

### Ensemble Models
```bash
# Combine multiple models for better performance
# See: /home/jgrusewski/Work/foxhunt/ENSEMBLE_IMPLEMENTATION_GUIDE.md

# Expected: Sharpe ratio 2.0-2.5 with ensemble (vs 1.5-2.0 single model)
```

---

## Key Resources

### Essential Documentation
- **[ML Infrastructure Guide](/home/jgrusewski/Work/foxhunt/docs/ML_INFRASTRUCTURE_GUIDE.md)** - Master index
- **[GPU Benchmark Guide](/home/jgrusewski/Work/foxhunt/ml/docs/GPU_BENCHMARK_GUIDE.md)** - GPU performance testing
- **[Checkpoint Selection Framework](/home/jgrusewski/Work/foxhunt/docs/CHECKPOINT_SELECTION_FRAMEWORK.md)** - How to choose best model
- **[Agent 78: DQN Training Success](/home/jgrusewski/Work/foxhunt/AGENT_78_DQN_PRODUCTION_TRAINING_SUCCESS.md)** - Real example

### Training Guides
- **[ML Training Roadmap](/home/jgrusewski/Work/foxhunt/ML_TRAINING_ROADMAP.md)** - 4-6 week plan
- **[DQN Training Report](/home/jgrusewski/Work/foxhunt/AGENT_25_DQN_TRAINING_REPORT.md)** - DQN specifics
- **[PPO Training Guide](/home/jgrusewski/Work/foxhunt/AGENT32_PPO_FIX_SUMMARY.md)** - PPO training
- **[Feature Engineering Report](/home/jgrusewski/Work/foxhunt/FEATURE_ENGINEERING_ENHANCEMENT_REPORT.md)** - 16 features + 10 indicators

---

## Success Metrics

### Training Success
- ✅ Training completes without OOM errors
- ✅ Loss decreasing over epochs
- ✅ Explained variance > 0.7
- ✅ Checkpoints saved every 10 epochs

### Model Quality
- ✅ Sharpe ratio > 1.5
- ✅ Win rate > 55%
- ✅ Max drawdown < 15%
- ✅ Consistent performance across validation periods

### Production Readiness
- ✅ Paper trading validates backtest results
- ✅ Sharpe ratio > 1.5 in live conditions
- ✅ Risk limits enforced
- ✅ Monitoring and alerting operational

---

**Estimated Total Time**:
- Setup: 30-60 minutes
- GPU Benchmark: 30-60 minutes
- DQN Training: 2-3 days
- Backtest + Analysis: 1-2 hours
- Paper Trading: 2-4 weeks
- Production Deployment: 1-2 days

**Total**: ~5-7 weeks from zero to production

**Next Guide**: [Quick Start: Hyperparameter Tuning](/home/jgrusewski/Work/foxhunt/docs/guides/QUICK_START_TUNING.md)