foxhunt/docs/RETRAINING_QUICKSTART.md

# Model Retraining Quick Start Guide

**For**: ML Engineers, Trading Operations
**Time to First Run**: 15 minutes
**Full Retraining Duration**: 2-3 days (RTX 3050 Ti)

---

## Prerequisites (5 minutes)

```bash
# 1. Navigate to project
cd /home/jgrusewski/Work/foxhunt

# 2. Start infrastructure
docker-compose up -d

# 3. Verify GPU
nvidia-smi  # Should show RTX 3050 Ti

# 4. Check data
ls test_data/real/databento/ml_training/*.dbn | wc -l
# Should show 80-100 files (90 days × 4 symbols)
```

---

## Quick Test (10 minutes)

### Dry Run (Validation Only)

```bash
cargo run -p ml --example retrain_all_models --release -- --dry-run
```

**Expected Output**:
```
🔍 DRY RUN MODE - No training will be performed
📋 Validating prerequisites...
✅ Prerequisites validated
📊 Data range prepared:
  • Start: 2024-07-15
  • End: 2024-10-14
  • Symbols: ["ES.FUT", "NQ.FUT", "ZN.FUT", "6E.FUT"]
  • Total bars: 180000
✅ Dry run validation complete - pipeline ready for execution
```

If this passes, you're ready to retrain!

---

## Full Quarterly Retraining (2-3 days)

### Option A: Rust Binary (Manual)

```bash
cargo run -p ml --example retrain_all_models --release --features cuda -- \
  --models DQN,PPO,MAMBA2,TFT \
  --latest-days 90 \
  --min-sharpe 1.5 \
  --min-win-rate 0.55 \
  --version-tag 2024Q4_v1
```

### Option B: Shell Script (Recommended)

```bash
./scripts/quarterly_retrain.sh
```

**With Slack/Email notifications**:
```bash
export SLACK_WEBHOOK="https://hooks.slack.com/services/YOUR/WEBHOOK"
export EMAIL_RECIPIENTS="ml-team@foxhunt.ai"
./scripts/quarterly_retrain.sh
```

---

## Monitor Progress

### View Logs

```bash
# Real-time log
tail -f logs/retraining_2024Q4_v1_*.log

# Search for errors
grep ERROR logs/retraining_2024Q4_v1_*.log

# Check progress
grep "Epoch.*/" logs/retraining_2024Q4_v1_*.log
```

### Expected Timeline

| Model | Duration | Progress Indicator |
|-------|----------|-------------------|
| DQN | 6-8 hours | `Epoch 1/200: loss=...` |
| PPO | 8-12 hours | `Epoch 1/200: loss=...` |
| MAMBA-2 | 20-30 hours | `Epoch 1/150: loss=...` |
| TFT | 10-15 hours | `Epoch 1/100: loss=...` |
| **Total** | **44-65 hours** | |

---

## Review Results

### Summary Report

```bash
# View JSON summary
cat ml/trained_models/quarterly/2024Q4/retraining_summary_2024Q4_v1.json | jq

# Check quality gates
jq '.results[] | {model: .model_type, passed: .quality_gate_passed, sharpe: .validation_metrics.sharpe_ratio}' \
   ml/trained_models/quarterly/2024Q4/retraining_summary_2024Q4_v1.json
```

**Example Output**:
```json
{
  "model": "DQN",
  "passed": true,
  "sharpe": 1.8
}
{
  "model": "PPO",
  "passed": true,
  "sharpe": 1.65
}
{
  "model": "MAMBA2",
  "passed": true,
  "sharpe": 1.9
}
{
  "model": "TFT",
  "passed": false,
  "sharpe": 1.42
}
```

### Quality Gate Pass/Fail

✅ **PASSED**: Sharpe ≥1.5, Win Rate ≥55%, Drawdown ≤25%
❌ **FAILED**: Does not meet one or more thresholds

---

## Next Steps After Training

### If Quality Gates Passed ✅

```bash
# 1. Deploy to staging
kubectl apply -f k8s/staging/ml-deployment-2024Q4.yaml

# 2. Monitor staging (7-10 days)
# - Grafana: http://localhost:3000/d/staging-ml
# - Check Sharpe ratio, win rate, latency

# 3. Production canary (10% traffic, 48 hours)
tli deploy canary --models DQN:2024Q4_v1 --traffic-percentage 10

# 4. Gradual rollout (25% → 50% → 100%)
tli deploy canary --traffic-percentage 25  # 48 hours
tli deploy canary --traffic-percentage 50  # 72 hours
tli model promote --model-type DQN --version 2024Q4_v1  # 100%
```

### If Quality Gates Failed ❌

```bash
# 1. Review failure reasons
jq '.results[] | select(.quality_gate_passed == false) | .quality_gate_failures' \
   ml/trained_models/quarterly/2024Q4/retraining_summary_2024Q4_v1.json

# Common issues:
# - "Sharpe ratio 1.42 < 1.50" → Needs hyperparameter tuning
# - "Win rate 52% < 55%" → May need more/better data
# - "Max drawdown 28% > 25%" → Model too aggressive

# 2. Investigate root cause
cargo run -p ml --example comprehensive_model_backtest -- \
  --checkpoint ml/trained_models/quarterly/2024Q4/tft_2024Q4_v1_final.safetensors

# 3. Corrective action
# Option A: Retrain with adjusted hyperparameters
# Option B: Run Optuna tuning (8-12 hours)
# Option C: Acquire more/better training data
```

---

## Automated Scheduling

### Install Cron Job

```bash
# Install (requires root)
sudo ./scripts/install_cron.sh

# Choose systemd timer (recommended)
sudo systemctl enable foxhunt-retrain.timer
sudo systemctl start foxhunt-retrain.timer

# Verify
sudo systemctl list-timers foxhunt-retrain.timer
```

**Schedule**: First Sunday of Jan/Apr/Jul/Oct at 2 AM

### Manual Trigger

```bash
# Test scheduling without waiting
sudo -u foxhunt ./scripts/quarterly_retrain.sh --dry-run

# Full run
sudo -u foxhunt ./scripts/quarterly_retrain.sh
```

---

## Troubleshooting

### GPU Out of Memory

**Symptoms**: `CUDA out of memory` error

**Solutions**:
```bash
# Option 1: Reduce batch size (in best_hyperparameters.yaml)
# dqn.batch_size: 128 → 64

# Option 2: Train sequentially (not parallel)
# Don't use --parallel flag

# Option 3: Use CPU (slower)
export CUDA_VISIBLE_DEVICES=""
cargo run -p ml --example retrain_all_models --release
```

### Data Loading Fails

**Symptoms**: `No DBN files found`

**Solutions**:
```bash
# Verify data
ls -lh test_data/real/databento/ml_training/*.dbn

# Re-download if needed
cargo run -p ml --example download_training_data --release -- \
  --symbols ES.FUT,NQ.FUT,ZN.FUT,6E.FUT \
  --start-date 2024-07-01 \
  --end-date 2024-10-01
```

### Training Not Converging

**Symptoms**: Loss stays high (>10.0), accuracy <50%

**Solutions**:
```bash
# Option 1: Increase learning rate
# Edit best_hyperparameters.yaml
# dqn.learning_rate: 0.0001 → 0.0005

# Option 2: More epochs
# dqn.epochs: 200 → 300

# Option 3: Run hyperparameter tuning
tli tune start --model DQN --trials 50
```

---

## Key Files

| File | Purpose |
|------|---------|
| `ml/examples/retrain_all_models.rs` | Main pipeline |
| `scripts/quarterly_retrain.sh` | Automation script |
| `ml/config/best_hyperparameters.yaml` | Hyperparameters |
| `docs/MODEL_RETRAINING_SOP.md` | Full documentation |
| `logs/retraining_*.log` | Training logs |
| `ml/trained_models/quarterly/` | Output checkpoints |

---

## Emergency Rollback

If production model fails after deployment:

```bash
# Immediate rollback to previous version
tli model rollback --model-type DQN --to-version 2024Q3_v1

# Verify
grpc_health_probe -addr=localhost:50054

# Document incident
tli model archive --model-type DQN --version 2024Q4_v1 \
  --reason "Failed production: Sharpe 0.4 < 1.5"
```

---

## Support

| Issue | Contact |
|-------|---------|
| Training failures | ML Engineering Lead |
| Infrastructure issues | DevOps Team |
| Quality gate failures | Trading Operations |
| Emergency rollback | On-call Engineer |

**Full Documentation**: `/home/jgrusewski/Work/foxhunt/docs/MODEL_RETRAINING_SOP.md`

---

**Last Updated**: 2025-10-14
**Next Review**: After first quarterly retraining (Jan 2025)