foxhunt/ML_TRAINING_QUICK_START.md

ML Training Quick Start Guide

Date: 2025-10-20 Status: ✅ SYSTEM READY - Execute immediately Timeline: 8-17 hours to production (NOT 4-6 weeks)

---

TL;DR

Models are already trained (Oct 20, 2025). VALIDATE FIRST before retraining.

Run this RIGHT NOW:

cd /home/jgrusewski/Work/foxhunt

cargo run -p backtesting_service --example backtest_mamba2 --release -- \
  --model-path ml/checkpoints/mamba2_dbn/best_model_epoch_10.safetensors \
  --data-path test_data/real/databento/ml_training \
  --symbol ES.FUT \
  --start-date 2024-03-01 \
  --end-date 2024-03-31 \
  --output-path backtests/mamba2_validation.json

Expected time: 1 hour Success criteria: Sharpe ≥1.5, Win Rate ≥55%, Drawdown ≤20%

---

System Status

✅ Infrastructure (100% Ready)

• GPU: RTX 3050 Ti idle (0% util, 48°C, 4GB VRAM free)
• Docker: 11/11 services healthy (Postgres, Redis, Vault, etc.)
• ML Training Service: Compiles successfully (port 50054)

✅ Training Data (Sufficient)

• 360 DBN files (16MB, ~180K-200K bars)
• 4 symbols: ES.FUT, NQ.FUT, 6E.FUT, ZN.FUT
• Quality: EXCELLENT (0 OHLCV violations)

✅ Feature Pipeline (225 Features)

• Wave C: 201 features (technical, microstructure, statistical)
• Wave D: 24 features (CUSUM, ADX, transitions, adaptive)
• Performance: 5.10μs/bar (196x faster than 1ms target)
• Tests: 99.4% pass rate (2,062/2,074)

✅ Model Checkpoints (ALREADY TRAINED!)

• MAMBA-2: best_model_epoch_10.safetensors (842KB, Oct 20, 2025)
• DQN: dqn_final_epoch100.safetensors (155KB, Oct 20, 2025)
• PPO: ppo_actor_epoch_20.safetensors (147KB, Oct 20, 2025)
• TFT: tft_225_epoch_0.safetensors (30MB, Oct 20, 2025)

---

Action Plan

Phase 1: Validate Existing Models (4 hours) ← START HERE

Run backtests to see if models already meet production targets

# MAMBA-2 (1h)
cargo run -p backtesting_service --example backtest_mamba2 --release -- \
  --model-path ml/checkpoints/mamba2_dbn/best_model_epoch_10.safetensors \
  --data-path test_data/real/databento/ml_training \
  --symbol ES.FUT --start-date 2024-03-01 --end-date 2024-03-31 \
  --output-path backtests/mamba2_validation.json

# DQN (1h)
cargo run -p backtesting_service --example backtest_dqn --release -- \
  --model-path ml/trained_models/dqn_final_epoch100.safetensors \
  --data-path test_data/real/databento/ml_training \
  --symbol ES.FUT --start-date 2024-03-01 --end-date 2024-03-31 \
  --output-path backtests/dqn_validation.json

# PPO (1h)
cargo run -p backtesting_service --example backtest_ppo --release -- \
  --actor-path ml/trained_models/ppo_actor_epoch_20.safetensors \
  --critic-path ml/trained_models/ppo_critic_epoch_20.safetensors \
  --data-path test_data/real/databento/ml_training \
  --symbol ES.FUT --start-date 2024-03-01 --end-date 2024-03-31 \
  --output-path backtests/ppo_validation.json

# Analyze (1h)
cargo run -p ml --example compare_backtest_results --release -- \
  --mamba2 backtests/mamba2_validation.json \
  --dqn backtests/dqn_validation.json \
  --ppo backtests/ppo_validation.json \
  --output backtests/model_comparison_report.md

Targets:

• Wave C: Sharpe ≥1.5, Win Rate ≥55%, Drawdown ≤20%
• Wave D: Sharpe ≥2.0, Win Rate ≥60%, Drawdown ≤15%

Decision:

• ✅ ALL PASS → Skip retraining, deploy immediately (Phase 5)
• ⚠️ SOME FAIL → Retrain only failing models (Phase 3)
• ❌ ALL FAIL → Full retraining (Phase 2 + 3)

---

Phase 2: Fix Warmup Bug (2 hours) ← IF RETRAINING

Only if Phase 1 shows models need retraining

File: /home/jgrusewski/Work/foxhunt/common/src/ml_strategy.rs

Add explicit warmup check in extract_features():

if bars.len() <= WARMUP_PERIOD {
    return Err(CommonError::invalid_input(
        format!("Insufficient data: {} bars, need >{}", bars.len(), WARMUP_PERIOD)
    ));
}

Validate:

cargo run -p ml --example validate_225_features_runtime --release
cargo test -p ml --lib feature_extraction --release

---

Phase 3: Retrain Models (4-8 hours) ← ONLY IF NEEDED

Automated pipeline for all models:

cargo run -p ml --example retrain_all_models --release -- \
  --models MAMBA2,DQN,PPO,TFT \
  --data-dir test_data/real/databento/ml_training \
  --output-dir ml/trained_models/quarterly_$(date +%Y%m%d) \
  --latest-days 90 \
  --min-sharpe 1.5 \
  --min-win-rate 0.55

Or individual models:

# MAMBA-2 (1.7-3.3h)
cargo run -p ml --example train_mamba2_dbn --release -- \
  --data-dir test_data/real/databento/ml_training \
  --epochs 50 --batch-size 32 --learning-rate 1e-4

# DQN (25-33min)
cargo run -p ml --example train_dqn --release -- \
  --data-dir test_data/real/databento/ml_training \
  --episodes 100 --batch-size 64

# PPO (6-8min)
cargo run -p ml --example train_ppo_extended --release -- \
  --data-dir test_data/real/databento/ml_training \
  --epochs 50 --batch-size 128

# TFT (1.5-2.5h)
cargo run -p ml --example train_tft_dbn --release -- \
  --data-dir test_data/real/databento/ml_training \
  --epochs 30 --batch-size 64

---

Phase 4: Validate Retrained (2 hours) ← AFTER RETRAINING

cargo test -p backtesting_service wave_d_backtest --release -- --nocapture

Targets: Sharpe ≥2.0, Win Rate ≥60%, Drawdown ≤15%

---

Phase 5: Production Deployment (4 hours) ← FINAL STEP

# 1. Apply database migration (regime detection tables)
cargo sqlx migrate run

# 2. Deploy checkpoints
mkdir -p ml/trained_models/production_$(date +%Y%m%d)
cp ml/checkpoints/mamba2_dbn/best_model_epoch_10.safetensors \
   ml/trained_models/production_$(date +%Y%m%d)/mamba2.safetensors
# ... (copy DQN, PPO, TFT)

ln -sfn production_$(date +%Y%m%d) ml/trained_models/production

# 3. Configure Grafana dashboards
curl -X POST http://admin:foxhunt123@localhost:3000/api/dashboards/import \
  -H "Content-Type: application/json" \
  -d @grafana/dashboards/wave_d_regime_detection.json

# 4. Start paper trading
tli trade ml regime --symbol ES.FUT
tli trade ml start-predictions --interval 30 --symbols ES.FUT,NQ.FUT

# 5. Enable Prometheus alerts
cp prometheus/alerts/wave_d_regime_detection.yml /etc/prometheus/alerts/
curl -X POST http://localhost:9090/-/reload

---

Timeline Summary

Scenario	Steps	Time	Cost
Best Case (models pass)	Phase 1 → Phase 5	8h (2 days)	$0
Likely Case (some fail)	Phase 1 → 2 → 3 → 4 → 5	16-18h (3 days)	$0
Worst Case (all fail)	Phase 1 → 2 → 3 → 4 → 5	17h (4 days)	$0

Original estimate (CLAUDE.md): 4-6 weeks (180-240 hours) Revised estimate: 8-17 hours (96-97% time savings)

---

Key Decisions

Decision	Choice	Rationale
Service vs. Examples?	Use ML examples	Faster iteration, easier debugging
Local vs. Cloud GPU?	Use local RTX 3050 Ti	$0 cost, available 24/7, 0 setup time
Validate vs. Retrain?	Validate first (Phase 1)	Models already trained today, may pass
Existing vs. More Data?	Use existing 360 files	Sufficient (180K bars), EXCELLENT quality

---

Blockers: NONE

✅ GPU ready (0% util, 48°C) ✅ Docker healthy (11/11 services) ✅ Data present (360 files, 16MB) ✅ Features operational (225 total, 5.10μs/bar) ✅ Checkpoints exist (trained Oct 20, 2025) ✅ Training scripts ready (26 examples)

Execute Phase 1 NOW.

---

Success Criteria

Wave C Targets (Minimum)

• Sharpe Ratio: ≥1.5
• Win Rate: ≥55%
• Max Drawdown: ≤20%

Wave D Targets (Goal)

• Sharpe Ratio: ≥2.0
• Win Rate: ≥60%
• Max Drawdown: ≤15%
• Regime Transitions: 5-10/day
• Position Sizing: 0.2x-1.5x range
• Stop-Loss: 1.5x-4.0x ATR

---

Next Command (Run NOW)

cd /home/jgrusewski/Work/foxhunt

cargo run -p backtesting_service --example backtest_mamba2 --release -- \
  --model-path ml/checkpoints/mamba2_dbn/best_model_epoch_10.safetensors \
  --data-path test_data/real/databento/ml_training \
  --symbol ES.FUT \
  --start-date 2024-03-01 \
  --end-date 2024-03-31 \
  --output-path backtests/mamba2_validation.json

Expected: 1 hour, JSON output with Sharpe/Win Rate/Drawdown

---

Documentation

• Full Investigation: AGENT_INVESTIGATION_05_ACTIONABLE_ROADMAP.md (21KB)
• Synthesis Report: INVESTIGATION_SYNTHESIS_COMPLETE.md (29KB)
• This Quick Start: ML_TRAINING_QUICK_START.md (you are here)
• System Status: CLAUDE.md (official system documentation)

---

Ready to execute. No blockers. Start Phase 1 validation immediately.