foxhunt/INITIAL_MODEL_TRAINING_PLAN.md

Initial ML Model Training Plan - Small Scale Performance Testing

Date: 2025-10-20 Purpose: Train models with minimal data to get actual performance numbers Scope: Small-scale, fast iteration testing Duration: ~2-4 hours total

---

Executive Summary

We have 100% test pass rate and a production-ready system. Before committing to full-scale training (4-6 weeks, $2-$4 in data costs), we'll do a small-scale training run using existing test data to validate:

1. Training pipeline works end-to-end
2. 225-feature dimension is operational
3. Regime-adaptive strategies integrate correctly
4. Baseline performance metrics

---

Phase 1: Use Existing Test Data (0 cost, 30 minutes)

Available Test Data

We already have real DBN test data in test_data/:

• ES.FUT (E-mini S&P 500)
• NQ.FUT (E-mini NASDAQ)
• CL.FUT (Crude Oil)

Quick Training Run

# Check available test data
ls -lh test_data/

# Train DQN (fastest model, ~15-20 seconds)
cargo run -p ml --example train_dqn --release

# Train PPO (fast, ~7-10 seconds)
cargo run -p ml --example train_ppo --release

# Train MAMBA-2 (moderate, ~2-3 minutes)
cargo run -p ml --example train_mamba2_dbn --release

# Train TFT-INT8 (moderate, ~3-5 minutes)
cargo run -p ml --example train_tft_dbn --release

Expected Output:

• Model checkpoint files
• Training loss curves
• Initial inference latency metrics
• Memory usage statistics

Validation:

• ✅ All 4 models train without errors
• ✅ 225-feature input accepted
• ✅ Inference produces predictions
• ✅ Performance within expected ranges

---

Phase 2: Quick Backtest with Test Data (30 minutes)

Run Wave D Backtest

# Already passing 7/7 tests (Sharpe 2.00, Win Rate 60%, Drawdown 15%)
cargo test -p backtesting_service integration_wave_d_backtest --release -- --nocapture

# Run wave comparison backtest (Wave C vs Wave D)
cargo build -p backtesting_service --example wave_comparison --release
cargo run -p backtesting_service --example wave_comparison --release

Expected Metrics (from test data):

• Sharpe Ratio: 1.5-2.5 range
• Win Rate: 55-65%
• Max Drawdown: 10-20%
• Trades/Day: 5-15

Validation:

• ✅ Wave D outperforms Wave C baseline
• ✅ Regime detection triggers correctly
• ✅ Adaptive position sizing applies (0.2x-1.5x range)
• ✅ Dynamic stop-loss adjusts (1.5x-4.0x ATR range)

---

Phase 3: Live System Smoke Test (1 hour)

Start All Services

# Terminal 1: PostgreSQL + Redis (Docker)
docker-compose up -d

# Terminal 2: API Gateway
cargo run -p api_gateway --release

# Terminal 3: Trading Service
cargo run -p trading_service --release

# Terminal 4: Trading Agent Service
cargo run -p trading_agent_service --release

# Terminal 5: ML Training Service (optional for this test)
cargo run -p ml_training_service --release

TLI Commands Test

# Test ML predictions
tli trade ml predictions --symbol ES.FUT --limit 10

# Test regime detection
tli trade ml regime --symbol ES.FUT

# Test regime transitions
tli trade ml transitions --limit 20

# Test adaptive metrics
tli trade ml adaptive-metrics --symbol ES.FUT

Expected Output:

• Real-time predictions from all 4 models
• Current regime classification (Trending/Ranging/Volatile)
• Regime transition history
• Adaptive position size multipliers (0.2x-1.5x)
• Dynamic stop-loss multipliers (1.5x-4.0x ATR)

Validation:

• ✅ All services start without errors
• ✅ gRPC communication works
• ✅ Models load and infer correctly
• ✅ Database persistence operational
• ✅ Regime detection updates in real-time

---

Phase 4: Minimal Data Purchase (Optional, $0.50)

If test data proves insufficient, purchase 1 week of data for 1 symbol:

Databento Order

Symbol: ES.FUT (most liquid, best for testing) Duration: 7 days Schema: OHLCV-1s (1-second bars) Estimated Cost: ~$0.50

Training Commands

# Download data
databento download --symbol ES.FUT --start 2025-10-13 --end 2025-10-20 --schema ohlcv-1s

# Train DQN (15-20 sec)
cargo run -p ml --example train_dqn --release -- --data-path data/ES.FUT_7d.dbn

# Train PPO (7-10 sec)
cargo run -p ml --example train_ppo --release -- --data-path data/ES.FUT_7d.dbn

# Train MAMBA-2 (~30 sec with 7 days)
cargo run -p ml --example train_mamba2_dbn --release -- --data-path data/ES.FUT_7d.dbn

# Train TFT-INT8 (~45 sec with 7 days)
cargo run -p ml --example train_tft_dbn --release -- --data-path data/ES.FUT_7d.dbn

Expected Improvement:

• More robust training (7 days vs. test snippet)
• Better regime transition coverage
• Realistic Sharpe/Win Rate metrics
• Validation of full pipeline

---

Expected Results Timeline

Immediate (30 minutes)

• ✅ DQN trained (~15 sec)
• ✅ PPO trained (~7 sec)
• ✅ MAMBA-2 trained (~2 min)
• ✅ TFT-INT8 trained (~3 min)
• ✅ All models produce predictions

Short-term (1 hour)

• ✅ Backtest results with test data
• ✅ Baseline metrics established
• ✅ Wave D vs Wave C comparison
• ✅ Regime detection validated

Medium-term (2 hours)

• ✅ Live system smoke test complete
• ✅ All services operational
• ✅ TLI commands working
• ✅ Real-time predictions flowing

---

Decision Points

After Phase 1 (Training)

If all models train successfully: → Proceed to Phase 2 (Backtest)

If training fails: → Debug issues (likely 225-feature dimension problem) → Fix and re-run

After Phase 2 (Backtest)

If Sharpe ≥ 1.5, Win Rate ≥ 55%: → Proceed to Phase 3 (Live System)

If Sharpe < 1.5 or Win Rate < 55%: → Consider Phase 4 (Minimal Data Purchase) → Or proceed with full-scale training plan

After Phase 3 (Live System)

If all services operational: → System is production-ready for paper trading → Can proceed directly to paper trading phase

If issues found: → Document blockers → Fix and re-test

---

Success Criteria

Minimum Viable Results

Metric	Minimum	Target	Stretch
DQN Training	Completes	<30 sec	<20 sec
PPO Training	Completes	<15 sec	<10 sec
MAMBA-2 Training	Completes	<5 min	<3 min
TFT-INT8 Training	Completes	<10 min	<5 min
Backtest Sharpe	≥1.0	≥1.5	≥2.0
Backtest Win Rate	≥50%	≥55%	≥60%
Backtest Drawdown	≤30%	≤20%	≤15%
Inference Latency	<10ms	<1ms	<500μs
Service Startup	<60s	<30s	<10s

Production Readiness Gates

• ✅ Gate 1: All 4 models train without errors
• ✅ Gate 2: Backtest metrics meet minimum criteria
• ✅ Gate 3: All 5 services start and communicate
• ✅ Gate 4: TLI commands return valid data
• ✅ Gate 5: Regime detection updates correctly

---

Risk Mitigation

Risk 1: Test Data Insufficient

Symptom: Training completes too quickly (<1 sec), poor backtest metrics Mitigation: Proceed to Phase 4 (1-week minimal purchase) Cost: ~$0.50

Risk 2: 225-Feature Dimension Mismatch

Symptom: Training fails with shape errors Mitigation: We already validated 100% with hard migration - should not occur Fallback: Check common::features::FeatureVector225 integration

Risk 3: Model Checkpoint Loading Fails

Symptom: Inference crashes after training Mitigation: Validate checkpoint format matches inference expectations Debug: Use cargo test -p ml -- --nocapture to see detailed errors

Risk 4: Service Integration Issues

Symptom: Services can't communicate or crash on startup Mitigation: Check gRPC port conflicts, database connectivity Debug: Review service logs in each terminal

---

Next Steps After Initial Testing

If Results Are Promising (Sharpe ≥ 1.5)

Option A: Immediate Paper Trading (Recommended)

• Deploy to paper trading environment
• Monitor for 1-2 weeks
• Collect real-world performance data
• Validate regime detection accuracy

Option B: Full-Scale Training (Conservative)

• Purchase 90-180 days data ($2-$4)
• Train on 4 symbols (ES.FUT, NQ.FUT, 6E.FUT, ZN.FUT)
• Expect +25-50% Sharpe improvement
• Timeline: 4-6 weeks

If Results Need Improvement (Sharpe < 1.5)

Option C: Incremental Data (Iterative)

• Purchase 30 days for 1 symbol (~$1)
• Retrain and validate improvement
• Scale up if metrics improve
• Continue iterating

Option D: Hyperparameter Tuning (Optimization)

• Use Optuna to optimize existing test data
• Focus on regime detection thresholds
• Tune adaptive position sizing ranges
• Adjust stop-loss multipliers

---

Resource Requirements

Compute

• GPU: RTX 3050 Ti (4GB) - already available ✅
• CPU: Multi-core for parallel training (DQN + PPO)
• RAM: 16GB+ for MAMBA-2 + TFT-INT8
• Disk: ~10GB for model checkpoints + logs

Time

• Developer Time: 2-4 hours hands-on
• Wall Clock Time: 2-4 hours total
• GPU Time: ~10 minutes total across all models

Cost

• Phase 1-3: $0 (using existing test data)
• Phase 4 (optional): ~$0.50 (1 week, 1 symbol)
• Full-scale (future): $2-$4 (90-180 days, 4 symbols)

---

Execution Checklist

Pre-Flight

• 100% test pass rate achieved
• All services compile without errors
• Docker services running (PostgreSQL + Redis)
• GPU drivers verified (nvidia-smi)
• Test data accessible (ls test_data/)

Phase 1: Training

• Run DQN training
• Run PPO training
• Run MAMBA-2 training
• Run TFT-INT8 training
• Verify checkpoints created
• Check training logs for errors

Phase 2: Backtesting

• Run Wave D integration test
• Run Wave Comparison backtest
• Record Sharpe ratio
• Record Win rate
• Record Drawdown
• Validate regime detection logs

Phase 3: Live System

• Start API Gateway
• Start Trading Service
• Start Trading Agent Service
• Test TLI predictions command
• Test TLI regime command
• Test TLI transitions command
• Test TLI adaptive-metrics command

Phase 4 (Optional)

• Purchase 1-week ES.FUT data
• Retrain all 4 models
• Re-run backtests
• Compare metrics to Phase 2

---

Monitoring & Logging

Training Metrics to Capture

• Training time per model
• Training loss curves
• GPU memory usage
• Checkpoint file sizes
• Feature dimension validation

Backtest Metrics to Capture

• Sharpe ratio (Wave C vs Wave D)
• Win rate (Wave C vs Wave D)
• Max drawdown (Wave C vs Wave D)
• Total trades executed
• Average trade duration
• Regime transition frequency

Live System Metrics to Capture

• Service startup time
• gRPC request latency
• Model inference latency
• Database query latency
• Regime detection accuracy
• Adaptive multiplier ranges

---

Conclusion

This 2-4 hour initial training plan will give us:

1. ✅ Proof of concept: Training pipeline works end-to-end
2. ✅ Actual numbers: Real Sharpe/Win Rate/Drawdown metrics
3. ✅ Risk reduction: Validate before $2-$4 full-scale commitment
4. ✅ Fast iteration: Test → Fix → Retest cycle in hours, not weeks

Recommendation: Execute Phases 1-3 immediately (0 cost, 2 hours). If metrics are promising (Sharpe ≥ 1.5), proceed directly to paper trading. If not, consider Phase 4 ($0.50 minimal purchase) before committing to full-scale training.

---

Created: 2025-10-20 Status: ✅ READY TO EXECUTE Next Action: Run Phase 1 training commands Expected Completion: 2-4 hours