foxhunt/docs/RETRAINING_QUICKSTART.md

Model Retraining Quick Start Guide

For: ML Engineers, Trading Operations Time to First Run: 15 minutes Full Retraining Duration: 2-3 days (RTX 3050 Ti)

---

Prerequisites (5 minutes)

# 1. Navigate to project
cd /home/jgrusewski/Work/foxhunt

# 2. Start infrastructure
docker-compose up -d

# 3. Verify GPU
nvidia-smi  # Should show RTX 3050 Ti

# 4. Check data
ls test_data/real/databento/ml_training/*.dbn | wc -l
# Should show 80-100 files (90 days × 4 symbols)

---

Quick Test (10 minutes)

Dry Run (Validation Only)

cargo run -p ml --example retrain_all_models --release -- --dry-run

Expected Output:

🔍 DRY RUN MODE - No training will be performed
📋 Validating prerequisites...
✅ Prerequisites validated
📊 Data range prepared:
  • Start: 2024-07-15
  • End: 2024-10-14
  • Symbols: ["ES.FUT", "NQ.FUT", "ZN.FUT", "6E.FUT"]
  • Total bars: 180000
✅ Dry run validation complete - pipeline ready for execution

If this passes, you're ready to retrain!

---

Full Quarterly Retraining (2-3 days)

Option A: Rust Binary (Manual)

cargo run -p ml --example retrain_all_models --release --features cuda -- \
  --models DQN,PPO,MAMBA2,TFT \
  --latest-days 90 \
  --min-sharpe 1.5 \
  --min-win-rate 0.55 \
  --version-tag 2024Q4_v1

Option B: Shell Script (Recommended)

./scripts/quarterly_retrain.sh

With Slack/Email notifications:

export SLACK_WEBHOOK="https://hooks.slack.com/services/YOUR/WEBHOOK"
export EMAIL_RECIPIENTS="ml-team@foxhunt.ai"
./scripts/quarterly_retrain.sh

---

Monitor Progress

View Logs

# Real-time log
tail -f logs/retraining_2024Q4_v1_*.log

# Search for errors
grep ERROR logs/retraining_2024Q4_v1_*.log

# Check progress
grep "Epoch.*/" logs/retraining_2024Q4_v1_*.log

Expected Timeline

Model	Duration	Progress Indicator
DQN	6-8 hours	Epoch 1/200: loss=...
PPO	8-12 hours	Epoch 1/200: loss=...
MAMBA-2	20-30 hours	Epoch 1/150: loss=...
TFT	10-15 hours	Epoch 1/100: loss=...
Total	44-65 hours

---

Review Results

Summary Report

# View JSON summary
cat ml/trained_models/quarterly/2024Q4/retraining_summary_2024Q4_v1.json | jq

# Check quality gates
jq '.results[] | {model: .model_type, passed: .quality_gate_passed, sharpe: .validation_metrics.sharpe_ratio}' \
   ml/trained_models/quarterly/2024Q4/retraining_summary_2024Q4_v1.json

Example Output:

{
  "model": "DQN",
  "passed": true,
  "sharpe": 1.8
}
{
  "model": "PPO",
  "passed": true,
  "sharpe": 1.65
}
{
  "model": "MAMBA2",
  "passed": true,
  "sharpe": 1.9
}
{
  "model": "TFT",
  "passed": false,
  "sharpe": 1.42
}

Quality Gate Pass/Fail

✅ PASSED: Sharpe ≥1.5, Win Rate ≥55%, Drawdown ≤25% ❌ FAILED: Does not meet one or more thresholds

---

Next Steps After Training

If Quality Gates Passed ✅

# 1. Deploy to staging
kubectl apply -f k8s/staging/ml-deployment-2024Q4.yaml

# 2. Monitor staging (7-10 days)
# - Grafana: http://localhost:3000/d/staging-ml
# - Check Sharpe ratio, win rate, latency

# 3. Production canary (10% traffic, 48 hours)
tli deploy canary --models DQN:2024Q4_v1 --traffic-percentage 10

# 4. Gradual rollout (25% → 50% → 100%)
tli deploy canary --traffic-percentage 25  # 48 hours
tli deploy canary --traffic-percentage 50  # 72 hours
tli model promote --model-type DQN --version 2024Q4_v1  # 100%

If Quality Gates Failed ❌

# 1. Review failure reasons
jq '.results[] | select(.quality_gate_passed == false) | .quality_gate_failures' \
   ml/trained_models/quarterly/2024Q4/retraining_summary_2024Q4_v1.json

# Common issues:
# - "Sharpe ratio 1.42 < 1.50" → Needs hyperparameter tuning
# - "Win rate 52% < 55%" → May need more/better data
# - "Max drawdown 28% > 25%" → Model too aggressive

# 2. Investigate root cause
cargo run -p ml --example comprehensive_model_backtest -- \
  --checkpoint ml/trained_models/quarterly/2024Q4/tft_2024Q4_v1_final.safetensors

# 3. Corrective action
# Option A: Retrain with adjusted hyperparameters
# Option B: Run Optuna tuning (8-12 hours)
# Option C: Acquire more/better training data

---

Automated Scheduling

Install Cron Job

# Install (requires root)
sudo ./scripts/install_cron.sh

# Choose systemd timer (recommended)
sudo systemctl enable foxhunt-retrain.timer
sudo systemctl start foxhunt-retrain.timer

# Verify
sudo systemctl list-timers foxhunt-retrain.timer

Schedule: First Sunday of Jan/Apr/Jul/Oct at 2 AM

Manual Trigger

# Test scheduling without waiting
sudo -u foxhunt ./scripts/quarterly_retrain.sh --dry-run

# Full run
sudo -u foxhunt ./scripts/quarterly_retrain.sh

---

Troubleshooting

GPU Out of Memory

Symptoms: CUDA out of memory error

Solutions:

# Option 1: Reduce batch size (in best_hyperparameters.yaml)
# dqn.batch_size: 128 → 64

# Option 2: Train sequentially (not parallel)
# Don't use --parallel flag

# Option 3: Use CPU (slower)
export CUDA_VISIBLE_DEVICES=""
cargo run -p ml --example retrain_all_models --release

Data Loading Fails

Symptoms: No DBN files found

Solutions:

# Verify data
ls -lh test_data/real/databento/ml_training/*.dbn

# Re-download if needed
cargo run -p ml --example download_training_data --release -- \
  --symbols ES.FUT,NQ.FUT,ZN.FUT,6E.FUT \
  --start-date 2024-07-01 \
  --end-date 2024-10-01

Training Not Converging

Symptoms: Loss stays high (>10.0), accuracy <50%

Solutions:

# Option 1: Increase learning rate
# Edit best_hyperparameters.yaml
# dqn.learning_rate: 0.0001 → 0.0005

# Option 2: More epochs
# dqn.epochs: 200 → 300

# Option 3: Run hyperparameter tuning
tli tune start --model DQN --trials 50

---

Key Files

File	Purpose
ml/examples/retrain_all_models.rs	Main pipeline
scripts/quarterly_retrain.sh	Automation script
ml/config/best_hyperparameters.yaml	Hyperparameters
docs/MODEL_RETRAINING_SOP.md	Full documentation
logs/retraining_*.log	Training logs
ml/trained_models/quarterly/	Output checkpoints

---

Emergency Rollback

If production model fails after deployment:

# Immediate rollback to previous version
tli model rollback --model-type DQN --to-version 2024Q3_v1

# Verify
grpc_health_probe -addr=localhost:50054

# Document incident
tli model archive --model-type DQN --version 2024Q4_v1 \
  --reason "Failed production: Sharpe 0.4 < 1.5"

---

Support

Issue	Contact
Training failures	ML Engineering Lead
Infrastructure issues	DevOps Team
Quality gate failures	Trading Operations
Emergency rollback	On-call Engineer

Full Documentation: /home/jgrusewski/Work/foxhunt/docs/MODEL_RETRAINING_SOP.md

---

Last Updated: 2025-10-14 Next Review: After first quarterly retraining (Jan 2025)