foxhunt/ML_TRAINING_QUICK_START.md

# ML Training Quick Start Guide

**Date**: 2025-10-20
**Status**: ✅ SYSTEM READY - Execute immediately
**Timeline**: 8-17 hours to production (NOT 4-6 weeks)

---

## TL;DR

**Models are already trained (Oct 20, 2025). VALIDATE FIRST before retraining.**

Run this RIGHT NOW:
```bash
cd /home/jgrusewski/Work/foxhunt

cargo run -p backtesting_service --example backtest_mamba2 --release -- \
  --model-path ml/checkpoints/mamba2_dbn/best_model_epoch_10.safetensors \
  --data-path test_data/real/databento/ml_training \
  --symbol ES.FUT \
  --start-date 2024-03-01 \
  --end-date 2024-03-31 \
  --output-path backtests/mamba2_validation.json
```

**Expected time**: 1 hour
**Success criteria**: Sharpe ≥1.5, Win Rate ≥55%, Drawdown ≤20%

---

## System Status

### ✅ Infrastructure (100% Ready)
- GPU: RTX 3050 Ti idle (0% util, 48°C, 4GB VRAM free)
- Docker: 11/11 services healthy (Postgres, Redis, Vault, etc.)
- ML Training Service: Compiles successfully (port 50054)

### ✅ Training Data (Sufficient)
- 360 DBN files (16MB, ~180K-200K bars)
- 4 symbols: ES.FUT, NQ.FUT, 6E.FUT, ZN.FUT
- Quality: EXCELLENT (0 OHLCV violations)

### ✅ Feature Pipeline (225 Features)
- Wave C: 201 features (technical, microstructure, statistical)
- Wave D: 24 features (CUSUM, ADX, transitions, adaptive)
- Performance: 5.10μs/bar (196x faster than 1ms target)
- Tests: 99.4% pass rate (2,062/2,074)

### ✅ Model Checkpoints (ALREADY TRAINED!)
- MAMBA-2: `best_model_epoch_10.safetensors` (842KB, Oct 20, 2025)
- DQN: `dqn_final_epoch100.safetensors` (155KB, Oct 20, 2025)
- PPO: `ppo_actor_epoch_20.safetensors` (147KB, Oct 20, 2025)
- TFT: `tft_225_epoch_0.safetensors` (30MB, Oct 20, 2025)

---

## Action Plan

### Phase 1: Validate Existing Models (4 hours) ← START HERE
**Run backtests to see if models already meet production targets**

```bash
# MAMBA-2 (1h)
cargo run -p backtesting_service --example backtest_mamba2 --release -- \
  --model-path ml/checkpoints/mamba2_dbn/best_model_epoch_10.safetensors \
  --data-path test_data/real/databento/ml_training \
  --symbol ES.FUT --start-date 2024-03-01 --end-date 2024-03-31 \
  --output-path backtests/mamba2_validation.json

# DQN (1h)
cargo run -p backtesting_service --example backtest_dqn --release -- \
  --model-path ml/trained_models/dqn_final_epoch100.safetensors \
  --data-path test_data/real/databento/ml_training \
  --symbol ES.FUT --start-date 2024-03-01 --end-date 2024-03-31 \
  --output-path backtests/dqn_validation.json

# PPO (1h)
cargo run -p backtesting_service --example backtest_ppo --release -- \
  --actor-path ml/trained_models/ppo_actor_epoch_20.safetensors \
  --critic-path ml/trained_models/ppo_critic_epoch_20.safetensors \
  --data-path test_data/real/databento/ml_training \
  --symbol ES.FUT --start-date 2024-03-01 --end-date 2024-03-31 \
  --output-path backtests/ppo_validation.json

# Analyze (1h)
cargo run -p ml --example compare_backtest_results --release -- \
  --mamba2 backtests/mamba2_validation.json \
  --dqn backtests/dqn_validation.json \
  --ppo backtests/ppo_validation.json \
  --output backtests/model_comparison_report.md
```

**Targets**:
- Wave C: Sharpe ≥1.5, Win Rate ≥55%, Drawdown ≤20%
- Wave D: Sharpe ≥2.0, Win Rate ≥60%, Drawdown ≤15%

**Decision**:
- ✅ **ALL PASS** → Skip retraining, deploy immediately (Phase 5)
- ⚠️ **SOME FAIL** → Retrain only failing models (Phase 3)
- ❌ **ALL FAIL** → Full retraining (Phase 2 + 3)

---

### Phase 2: Fix Warmup Bug (2 hours) ← IF RETRAINING
**Only if Phase 1 shows models need retraining**

**File**: `/home/jgrusewski/Work/foxhunt/common/src/ml_strategy.rs`

Add explicit warmup check in `extract_features()`:
```rust
if bars.len() <= WARMUP_PERIOD {
    return Err(CommonError::invalid_input(
        format!("Insufficient data: {} bars, need >{}", bars.len(), WARMUP_PERIOD)
    ));
}
```

**Validate**:
```bash
cargo run -p ml --example validate_225_features_runtime --release
cargo test -p ml --lib feature_extraction --release
```

---

### Phase 3: Retrain Models (4-8 hours) ← ONLY IF NEEDED
**Automated pipeline for all models**:
```bash
cargo run -p ml --example retrain_all_models --release -- \
  --models MAMBA2,DQN,PPO,TFT \
  --data-dir test_data/real/databento/ml_training \
  --output-dir ml/trained_models/quarterly_$(date +%Y%m%d) \
  --latest-days 90 \
  --min-sharpe 1.5 \
  --min-win-rate 0.55
```

**Or individual models**:
```bash
# MAMBA-2 (1.7-3.3h)
cargo run -p ml --example train_mamba2_dbn --release -- \
  --data-dir test_data/real/databento/ml_training \
  --epochs 50 --batch-size 32 --learning-rate 1e-4

# DQN (25-33min)
cargo run -p ml --example train_dqn --release -- \
  --data-dir test_data/real/databento/ml_training \
  --episodes 100 --batch-size 64

# PPO (6-8min)
cargo run -p ml --example train_ppo_extended --release -- \
  --data-dir test_data/real/databento/ml_training \
  --epochs 50 --batch-size 128

# TFT (1.5-2.5h)
cargo run -p ml --example train_tft_dbn --release -- \
  --data-dir test_data/real/databento/ml_training \
  --epochs 30 --batch-size 64
```

---

### Phase 4: Validate Retrained (2 hours) ← AFTER RETRAINING
```bash
cargo test -p backtesting_service wave_d_backtest --release -- --nocapture
```

**Targets**: Sharpe ≥2.0, Win Rate ≥60%, Drawdown ≤15%

---

### Phase 5: Production Deployment (4 hours) ← FINAL STEP
```bash
# 1. Apply database migration (regime detection tables)
cargo sqlx migrate run

# 2. Deploy checkpoints
mkdir -p ml/trained_models/production_$(date +%Y%m%d)
cp ml/checkpoints/mamba2_dbn/best_model_epoch_10.safetensors \
   ml/trained_models/production_$(date +%Y%m%d)/mamba2.safetensors
# ... (copy DQN, PPO, TFT)

ln -sfn production_$(date +%Y%m%d) ml/trained_models/production

# 3. Configure Grafana dashboards
curl -X POST http://admin:foxhunt123@localhost:3000/api/dashboards/import \
  -H "Content-Type: application/json" \
  -d @grafana/dashboards/wave_d_regime_detection.json

# 4. Start paper trading
tli trade ml regime --symbol ES.FUT
tli trade ml start-predictions --interval 30 --symbols ES.FUT,NQ.FUT

# 5. Enable Prometheus alerts
cp prometheus/alerts/wave_d_regime_detection.yml /etc/prometheus/alerts/
curl -X POST http://localhost:9090/-/reload
```

---

## Timeline Summary

| Scenario | Steps | Time | Cost |
|---|---|---|---|
| **Best Case** (models pass) | Phase 1 → Phase 5 | 8h (2 days) | $0 |
| **Likely Case** (some fail) | Phase 1 → 2 → 3 → 4 → 5 | 16-18h (3 days) | $0 |
| **Worst Case** (all fail) | Phase 1 → 2 → 3 → 4 → 5 | 17h (4 days) | $0 |

**Original estimate** (CLAUDE.md): 4-6 weeks (180-240 hours)
**Revised estimate**: 8-17 hours (96-97% time savings)

---

## Key Decisions

| Decision | Choice | Rationale |
|---|---|---|
| **Service vs. Examples?** | Use ML examples | Faster iteration, easier debugging |
| **Local vs. Cloud GPU?** | Use local RTX 3050 Ti | $0 cost, available 24/7, 0 setup time |
| **Validate vs. Retrain?** | Validate first (Phase 1) | Models already trained today, may pass |
| **Existing vs. More Data?** | Use existing 360 files | Sufficient (180K bars), EXCELLENT quality |

---

## Blockers: NONE

✅ GPU ready (0% util, 48°C)
✅ Docker healthy (11/11 services)
✅ Data present (360 files, 16MB)
✅ Features operational (225 total, 5.10μs/bar)
✅ Checkpoints exist (trained Oct 20, 2025)
✅ Training scripts ready (26 examples)

**Execute Phase 1 NOW.**

---

## Success Criteria

### Wave C Targets (Minimum)
- Sharpe Ratio: ≥1.5
- Win Rate: ≥55%
- Max Drawdown: ≤20%

### Wave D Targets (Goal)
- Sharpe Ratio: ≥2.0
- Win Rate: ≥60%
- Max Drawdown: ≤15%
- Regime Transitions: 5-10/day
- Position Sizing: 0.2x-1.5x range
- Stop-Loss: 1.5x-4.0x ATR

---

## Next Command (Run NOW)

```bash
cd /home/jgrusewski/Work/foxhunt

cargo run -p backtesting_service --example backtest_mamba2 --release -- \
  --model-path ml/checkpoints/mamba2_dbn/best_model_epoch_10.safetensors \
  --data-path test_data/real/databento/ml_training \
  --symbol ES.FUT \
  --start-date 2024-03-01 \
  --end-date 2024-03-31 \
  --output-path backtests/mamba2_validation.json
```

**Expected**: 1 hour, JSON output with Sharpe/Win Rate/Drawdown

---

## Documentation

- **Full Investigation**: `AGENT_INVESTIGATION_05_ACTIONABLE_ROADMAP.md` (21KB)
- **Synthesis Report**: `INVESTIGATION_SYNTHESIS_COMPLETE.md` (29KB)
- **This Quick Start**: `ML_TRAINING_QUICK_START.md` (you are here)
- **System Status**: `CLAUDE.md` (official system documentation)

---

**Ready to execute. No blockers. Start Phase 1 validation immediately.**