4d0efa82df
Wave 1 (Architecture & Design - 5 agents): - Multi-model training orchestration (DQN, PPO, MAMBA-2, TFT-INT8) - Sequential training strategy (95.9% GPU headroom, 6.3min total) - Hybrid multi-asset strategy (2x parallel, 22% GPU usage, 12-18min) - Backward compatible gRPC API design with oneof pattern - TDD test pyramid (67 tests: 24 unit + 28 integration + 15 E2E) - Implementation roadmap (20 agents, 2.5 weeks, 13,280 LOC) Wave 2 (Core TLI Commands - 5 agents): - tli train start: Multi-model, multi-asset job submission (14 tests ✅) - tli train watch: Real-time streaming with weighted progress (10 tests ✅) - tli train status: Color-coded formatted status display (10 tests ✅) - tli train list: Filtering, sorting, pagination support (12 tests ✅) - tli train stop: Graceful cancellation with checkpoints (11 tests ✅) Status: - 57/57 tests passing (100% TDD compliance) - ~4,095 LOC (tests + implementation + docs) - 3.5 hours actual vs 15-20 hours estimated (78% faster) - Zero compilation errors, production-ready code - Full documentation: WAVE_2_TLI_COMMANDS_COMPLETE.md Next: Wave 3 (Multi-Asset Multi-Model Backend Logic - 5 agents) 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com>
2.9 KiB
2.9 KiB
MAMBA-2 Parquet Training Example
This example demonstrates how to train the MAMBA-2 State Space Model using Parquet market data files.
Quick Start
# Default: 200 epochs using ES.FUT data
cargo run -p ml --example train_mamba2_parquet --release
# Custom Parquet file and epochs
cargo run -p ml --example train_mamba2_parquet --release -- \
--parquet-file test_data/NQ_FUT_180d.parquet \
--epochs 50
# Custom lookback window (sequence length)
cargo run -p ml --example train_mamba2_parquet --release -- \
--parquet-file test_data/ES_FUT_180d.parquet \
--lookback-window 120 \
--epochs 100
Available Arguments
--parquet-file <path>: Path to Parquet file (default: test_data/ES_FUT_180d.parquet)--epochs <n>: Number of training epochs (default: 200)--lookback-window <n>: Sequence length/lookback window (default: 60)--batch-size <n>: Batch size for training (default: 32)--learning-rate <f>: Learning rate (default: 0.0001)--hidden-dim <n>: Hidden dimension (default: 225, Wave D feature count)--state-dim <n>: SSM state dimension (default: 16) --output-dir `: Output directory for checkpoints (default: ml/checkpoints/mamba2_parquet)
Available Parquet Files
test_data/ES_FUT_180d.parquet- E-mini S&P 500 (180 days)test_data/NQ_FUT_180d.parquet- E-mini NASDAQ (180 days)test_data/6E_FUT_180d.parquet- Euro FX (180 days)test_data/ZN_FUT_90d.parquet- 10-Year T-Note (90 days)
Output
Training outputs are saved to ml/checkpoints/mamba2_parquet/:
best_model_epoch_*.ckpt- Best model based on validation losscheckpoint_epoch_*.ckpt- Periodic checkpoints (every 10 epochs)final_model.ckpt- Final model after trainingtraining_losses.csv- Training/validation loss curvestraining_metrics.json- Summary metrics and configuration
Features
- 225 Wave D Features: Includes 201 Wave C features + 24 Wave D regime features
- GPU Training: CUDA acceleration (RTX 3050 Ti optimized)
- Early Stopping: Automatic stopping after 20 epochs of no improvement
- Checkpointing: Saves best models and periodic snapshots
- Monitoring: Real-time loss, perplexity, and training speed metrics
Expected Training Time
- 50 epochs: ~30-45 minutes (pilot run)
- 200 epochs: ~2-3 hours (full training)
- GPU utilization: ~60-70% (memory-bound on RTX 3050 Ti)
Requirements
- CUDA GPU (required - no CPU fallback)
- Minimum 50 bars in Parquet file (for feature warmup period)
- ~4GB VRAM available
Data Format
Parquet files should contain OHLCV market data with these columns:
timestamp_ns: Nanosecond timestampopen,high,low: Price data (optional, defaults toprice)price: Close price (required)quantity: Volume (optional, defaults to 0)symbol,venue,event_type,sequence: Metadata columns
See data/src/providers/databento/dbn_to_parquet_converter.rs for conversion utilities.