# Initial ML Model Training Plan - Small Scale Performance Testing

**Date**: 2025-10-20
**Purpose**: Train models with minimal data to get actual performance numbers
**Scope**: Small-scale, fast iteration testing
**Duration**: ~2-4 hours total

---

## Executive Summary

We have **100% test pass rate** and a production-ready system. Before committing to full-scale training (4-6 weeks, $2-$4 in data costs), we'll do a small-scale training run using **existing test data** to validate:

1. **Training pipeline works end-to-end**
2. **225-feature dimension is operational**
3. **Regime-adaptive strategies integrate correctly**
4. **Baseline performance metrics**

---

## Phase 1: Use Existing Test Data (0 cost, 30 minutes)

### Available Test Data

We already have real DBN test data in `test_data/`:
- ES.FUT (E-mini S&P 500)
- NQ.FUT (E-mini NASDAQ)
- CL.FUT (Crude Oil)

### Quick Training Run

```bash
# Check available test data
ls -lh test_data/

# Train DQN (fastest model, ~15-20 seconds)
cargo run -p ml --example train_dqn --release

# Train PPO (fast, ~7-10 seconds)
cargo run -p ml --example train_ppo --release

# Train MAMBA-2 (moderate, ~2-3 minutes)
cargo run -p ml --example train_mamba2_dbn --release

# Train TFT-INT8 (moderate, ~3-5 minutes)
cargo run -p ml --example train_tft_dbn --release
```

**Expected Output**:
- Model checkpoint files
- Training loss curves
- Initial inference latency metrics
- Memory usage statistics

**Validation**:
- ✅ All 4 models train without errors
- ✅ 225-feature input accepted
- ✅ Inference produces predictions
- ✅ Performance within expected ranges

---

## Phase 2: Quick Backtest with Test Data (30 minutes)

### Run Wave D Backtest

```bash
# Already passing 7/7 tests (Sharpe 2.00, Win Rate 60%, Drawdown 15%)
cargo test -p backtesting_service integration_wave_d_backtest --release -- --nocapture

# Run wave comparison backtest (Wave C vs Wave D)
cargo build -p backtesting_service --example wave_comparison --release
cargo run -p backtesting_service --example wave_comparison --release
```

**Expected Metrics** (from test data):
- **Sharpe Ratio**: 1.5-2.5 range
- **Win Rate**: 55-65%
- **Max Drawdown**: 10-20%
- **Trades/Day**: 5-15

**Validation**:
- ✅ Wave D outperforms Wave C baseline
- ✅ Regime detection triggers correctly
- ✅ Adaptive position sizing applies (0.2x-1.5x range)
- ✅ Dynamic stop-loss adjusts (1.5x-4.0x ATR range)

---

## Phase 3: Live System Smoke Test (1 hour)

### Start All Services

```bash
# Terminal 1: PostgreSQL + Redis (Docker)
docker-compose up -d

# Terminal 2: API Gateway
cargo run -p api_gateway --release

# Terminal 3: Trading Service
cargo run -p trading_service --release

# Terminal 4: Trading Agent Service
cargo run -p trading_agent_service --release

# Terminal 5: ML Training Service (optional for this test)
cargo run -p ml_training_service --release
```

### TLI Commands Test

```bash
# Test ML predictions
tli trade ml predictions --symbol ES.FUT --limit 10

# Test regime detection
tli trade ml regime --symbol ES.FUT

# Test regime transitions
tli trade ml transitions --limit 20

# Test adaptive metrics
tli trade ml adaptive-metrics --symbol ES.FUT
```

**Expected Output**:
- Real-time predictions from all 4 models
- Current regime classification (Trending/Ranging/Volatile)
- Regime transition history
- Adaptive position size multipliers (0.2x-1.5x)
- Dynamic stop-loss multipliers (1.5x-4.0x ATR)

**Validation**:
- ✅ All services start without errors
- ✅ gRPC communication works
- ✅ Models load and infer correctly
- ✅ Database persistence operational
- ✅ Regime detection updates in real-time

---

## Phase 4: Minimal Data Purchase (Optional, $0.50)

If test data proves insufficient, purchase **1 week** of data for **1 symbol**:

### Databento Order

**Symbol**: ES.FUT (most liquid, best for testing)
**Duration**: 7 days
**Schema**: OHLCV-1s (1-second bars)
**Estimated Cost**: ~$0.50

### Training Commands

```bash
# Download data
databento download --symbol ES.FUT --start 2025-10-13 --end 2025-10-20 --schema ohlcv-1s

# Train DQN (15-20 sec)
cargo run -p ml --example train_dqn --release -- --data-path data/ES.FUT_7d.dbn

# Train PPO (7-10 sec)
cargo run -p ml --example train_ppo --release -- --data-path data/ES.FUT_7d.dbn

# Train MAMBA-2 (~30 sec with 7 days)
cargo run -p ml --example train_mamba2_dbn --release -- --data-path data/ES.FUT_7d.dbn

# Train TFT-INT8 (~45 sec with 7 days)
cargo run -p ml --example train_tft_dbn --release -- --data-path data/ES.FUT_7d.dbn
```

**Expected Improvement**:
- More robust training (7 days vs. test snippet)
- Better regime transition coverage
- Realistic Sharpe/Win Rate metrics
- Validation of full pipeline

---

## Expected Results Timeline

### Immediate (30 minutes)
- ✅ DQN trained (~15 sec)
- ✅ PPO trained (~7 sec)
- ✅ MAMBA-2 trained (~2 min)
- ✅ TFT-INT8 trained (~3 min)
- ✅ All models produce predictions

### Short-term (1 hour)
- ✅ Backtest results with test data
- ✅ Baseline metrics established
- ✅ Wave D vs Wave C comparison
- ✅ Regime detection validated

### Medium-term (2 hours)
- ✅ Live system smoke test complete
- ✅ All services operational
- ✅ TLI commands working
- ✅ Real-time predictions flowing

---

## Decision Points

### After Phase 1 (Training)

**If all models train successfully**:
→ Proceed to Phase 2 (Backtest)

**If training fails**:
→ Debug issues (likely 225-feature dimension problem)
→ Fix and re-run

### After Phase 2 (Backtest)

**If Sharpe ≥ 1.5, Win Rate ≥ 55%**:
→ Proceed to Phase 3 (Live System)

**If Sharpe < 1.5 or Win Rate < 55%**:
→ Consider Phase 4 (Minimal Data Purchase)
→ Or proceed with full-scale training plan

### After Phase 3 (Live System)

**If all services operational**:
→ System is production-ready for paper trading
→ Can proceed directly to paper trading phase

**If issues found**:
→ Document blockers
→ Fix and re-test

---

## Success Criteria

### Minimum Viable Results

| Metric | Minimum | Target | Stretch |
|--------|---------|--------|---------|
| **DQN Training** | Completes | <30 sec | <20 sec |
| **PPO Training** | Completes | <15 sec | <10 sec |
| **MAMBA-2 Training** | Completes | <5 min | <3 min |
| **TFT-INT8 Training** | Completes | <10 min | <5 min |
| **Backtest Sharpe** | ≥1.0 | ≥1.5 | ≥2.0 |
| **Backtest Win Rate** | ≥50% | ≥55% | ≥60% |
| **Backtest Drawdown** | ≤30% | ≤20% | ≤15% |
| **Inference Latency** | <10ms | <1ms | <500μs |
| **Service Startup** | <60s | <30s | <10s |

### Production Readiness Gates

- ✅ **Gate 1**: All 4 models train without errors
- ✅ **Gate 2**: Backtest metrics meet minimum criteria
- ✅ **Gate 3**: All 5 services start and communicate
- ✅ **Gate 4**: TLI commands return valid data
- ✅ **Gate 5**: Regime detection updates correctly

---

## Risk Mitigation

### Risk 1: Test Data Insufficient

**Symptom**: Training completes too quickly (<1 sec), poor backtest metrics
**Mitigation**: Proceed to Phase 4 (1-week minimal purchase)
**Cost**: ~$0.50

### Risk 2: 225-Feature Dimension Mismatch

**Symptom**: Training fails with shape errors
**Mitigation**: We already validated 100% with hard migration - should not occur
**Fallback**: Check `common::features::FeatureVector225` integration

### Risk 3: Model Checkpoint Loading Fails

**Symptom**: Inference crashes after training
**Mitigation**: Validate checkpoint format matches inference expectations
**Debug**: Use `cargo test -p ml -- --nocapture` to see detailed errors

### Risk 4: Service Integration Issues

**Symptom**: Services can't communicate or crash on startup
**Mitigation**: Check gRPC port conflicts, database connectivity
**Debug**: Review service logs in each terminal

---

## Next Steps After Initial Testing

### If Results Are Promising (Sharpe ≥ 1.5)

**Option A: Immediate Paper Trading** (Recommended)
- Deploy to paper trading environment
- Monitor for 1-2 weeks
- Collect real-world performance data
- Validate regime detection accuracy

**Option B: Full-Scale Training** (Conservative)
- Purchase 90-180 days data ($2-$4)
- Train on 4 symbols (ES.FUT, NQ.FUT, 6E.FUT, ZN.FUT)
- Expect +25-50% Sharpe improvement
- Timeline: 4-6 weeks

### If Results Need Improvement (Sharpe < 1.5)

**Option C: Incremental Data** (Iterative)
- Purchase 30 days for 1 symbol (~$1)
- Retrain and validate improvement
- Scale up if metrics improve
- Continue iterating

**Option D: Hyperparameter Tuning** (Optimization)
- Use Optuna to optimize existing test data
- Focus on regime detection thresholds
- Tune adaptive position sizing ranges
- Adjust stop-loss multipliers

---

## Resource Requirements

### Compute
- **GPU**: RTX 3050 Ti (4GB) - already available ✅
- **CPU**: Multi-core for parallel training (DQN + PPO)
- **RAM**: 16GB+ for MAMBA-2 + TFT-INT8
- **Disk**: ~10GB for model checkpoints + logs

### Time
- **Developer Time**: 2-4 hours hands-on
- **Wall Clock Time**: 2-4 hours total
- **GPU Time**: ~10 minutes total across all models

### Cost
- **Phase 1-3**: $0 (using existing test data)
- **Phase 4 (optional)**: ~$0.50 (1 week, 1 symbol)
- **Full-scale (future)**: $2-$4 (90-180 days, 4 symbols)

---

## Execution Checklist

### Pre-Flight
- [x] 100% test pass rate achieved
- [x] All services compile without errors
- [x] Docker services running (PostgreSQL + Redis)
- [ ] GPU drivers verified (`nvidia-smi`)
- [ ] Test data accessible (`ls test_data/`)

### Phase 1: Training
- [ ] Run DQN training
- [ ] Run PPO training
- [ ] Run MAMBA-2 training
- [ ] Run TFT-INT8 training
- [ ] Verify checkpoints created
- [ ] Check training logs for errors

### Phase 2: Backtesting
- [ ] Run Wave D integration test
- [ ] Run Wave Comparison backtest
- [ ] Record Sharpe ratio
- [ ] Record Win rate
- [ ] Record Drawdown
- [ ] Validate regime detection logs

### Phase 3: Live System
- [ ] Start API Gateway
- [ ] Start Trading Service
- [ ] Start Trading Agent Service
- [ ] Test TLI predictions command
- [ ] Test TLI regime command
- [ ] Test TLI transitions command
- [ ] Test TLI adaptive-metrics command

### Phase 4 (Optional)
- [ ] Purchase 1-week ES.FUT data
- [ ] Retrain all 4 models
- [ ] Re-run backtests
- [ ] Compare metrics to Phase 2

---

## Monitoring & Logging

### Training Metrics to Capture
- Training time per model
- Training loss curves
- GPU memory usage
- Checkpoint file sizes
- Feature dimension validation

### Backtest Metrics to Capture
- Sharpe ratio (Wave C vs Wave D)
- Win rate (Wave C vs Wave D)
- Max drawdown (Wave C vs Wave D)
- Total trades executed
- Average trade duration
- Regime transition frequency

### Live System Metrics to Capture
- Service startup time
- gRPC request latency
- Model inference latency
- Database query latency
- Regime detection accuracy
- Adaptive multiplier ranges

---

## Conclusion

This **2-4 hour initial training plan** will give us:

1. ✅ **Proof of concept**: Training pipeline works end-to-end
2. ✅ **Actual numbers**: Real Sharpe/Win Rate/Drawdown metrics
3. ✅ **Risk reduction**: Validate before $2-$4 full-scale commitment
4. ✅ **Fast iteration**: Test → Fix → Retest cycle in hours, not weeks

**Recommendation**: Execute Phases 1-3 **immediately** (0 cost, 2 hours). If metrics are promising (Sharpe ≥ 1.5), proceed directly to paper trading. If not, consider Phase 4 ($0.50 minimal purchase) before committing to full-scale training.

---

**Created**: 2025-10-20
**Status**: ✅ READY TO EXECUTE
**Next Action**: Run Phase 1 training commands
**Expected Completion**: 2-4 hours