foxhunt/MAMBA2_TUNING_QUICKSTART.md

# MAMBA-2 Hyperparameter Tuning - Quick Start Guide

**Status**: ✅ **READY TO EXECUTE**
**Duration**: 6-10 hours (40 trials)
**GPU**: RTX 3050 Ti (4GB VRAM)
**Created**: 2025-10-14

---

## 🚀 Quick Commands

### 1. Start MAMBA-2 Tuning (Recommended)

```bash
# Login first
tli login

# Start 40-trial tuning with live monitoring
tli tune start \
  --model MAMBA_2 \
  --trials 40 \
  --watch
```

**Expected Output**:
```
Job ID: 8a7b9c3d-4e5f-6a1b-2c3d-4e5f6a7b8c9d
Model: MAMBA_2
Trials: 40
Status: Running
Estimated time: 6-10 hours

[Trial 1/40] lr=0.0001, batch=32, state=16, hidden=256, sharpe=1.42
[Trial 2/40] lr=0.001, batch=16, state=32, hidden=512, sharpe=1.38 (PRUNED)
[Trial 3/40] lr=0.0001, batch=32, state=16, hidden=256, sharpe=1.68 ⭐
...
```

---

### 2. Monitor Progress

```bash
# Check status anytime
tli tune status --job-id 8a7b9c3d-4e5f-6a1b-2c3d-4e5f6a7b8c9d
```

**Output**:
```
Job ID: 8a7b9c3d-4e5f-6a1b-2c3d-4e5f6a7b8c9d
Status: Running
Progress: 23/40 trials (57.5%)
Best Sharpe: 1.72
Elapsed: 4h 15m
Estimated remaining: 3h 10m
```

---

### 3. Get Best Hyperparameters

```bash
# After completion (or anytime during run)
tli tune best --job-id 8a7b9c3d-4e5f-6a1b-2c3d-4e5f6a7b8c9d
```

**Example Output**:
```yaml
Best Trial: 23
Sharpe Ratio: 1.78
Hyperparameters:
  learning_rate: 0.0001
  batch_size: 32
  hidden_dim: 256
  state_size: 16
  num_layers: 4
  expansion_factor: 2
  dropout: 0.15
  dt_min: 0.001
  dt_max: 0.08
  use_ssd: true
  use_selective_state: true
  hardware_aware: true
  grad_clip: 1.25
  weight_decay: 0.0005
  warmup_steps: 800
```

---

### 4. Stop Tuning (Optional)

```bash
# Cancel running job if needed
tli tune stop --job-id 8a7b9c3d-4e5f-6a1b-2c3d-4e5f6a7b8c9d
```

**Note**: Partial results are saved and can be retrieved with `tune best`

---

## 📊 Search Space Summary

MAMBA-2 tuning optimizes **14 hyperparameters** across **40 trials**:

### Core Architecture (State-Space Dynamics)
- **learning_rate**: [0.00001, 0.0001, 0.001]
- **batch_size**: [16, 32, 64] (memory-constrained)
- **hidden_dim**: [128, 256, 512] (d_model)
- **state_size**: [8, 16, 32] (d_state - critical for dynamics)
- **num_layers**: [2, 4, 8]
- **expansion_factor**: [2, 4]

### State-Space Dynamics
- **dt_min**: [0.0001, 0.01] (log scale)
- **dt_max**: [0.01, 1.0] (log scale)

### Architecture Features
- **use_ssd**: [true, false] (Structured State Duality)
- **use_selective_state**: [true, false] (context-aware transitions)
- **hardware_aware**: [true, false] (RTX 3050 Ti optimizations)

### Regularization
- **dropout**: [0.0, 0.3]
- **grad_clip**: [0.5, 2.0] (critical for state-space stability)
- **weight_decay**: [0.0001, 0.01] (log scale)
- **warmup_steps**: [100, 2000]

**Total search space**: ~50,000+ unique configurations

---

## ⏱️ Time Estimates

| Stage | Duration | Activity |
|-------|----------|----------|
| **Trial 1-5** | 60-90 min | Baseline establishment (no pruning) |
| **Trial 6-40** | 4-8 hours | Optuna TPE optimization with MedianPruner |
| **Total** | **6-10 hours** | Full 40-trial study |

### MedianPruner Savings
- **30-50% time reduction** by stopping unpromising trials early
- Prunes trials with poor Sharpe ratio trends at epoch 10-30
- Estimated 16/40 trials will be pruned (saves ~2 hours)

---

## 🎯 Expected Performance

### Baseline (DQN/PPO from prior tuning)
```
DQN:  Sharpe 1.50, Win Rate 52%, Max DD -15%
PPO:  Sharpe 1.30, Win Rate 50%, Max DD -18%
```

### MAMBA-2 Expected Outcome (40 trials)

**Conservative**:
```
Sharpe: 1.60-1.80  (10-20% improvement)
Win Rate: 53-56%
Max Drawdown: -12-14%
Inference: <100μs
VRAM: ~2.2GB
```

**Optimistic** (if state-space excels at HFT patterns):
```
Sharpe: 1.90-2.20  (30-50% improvement)
Win Rate: 57-62%
Max Drawdown: -10-12%
Inference: <80μs
VRAM: ~2.2GB
```

---

## 🔬 Key Research Questions

1. **State Size vs Performance**: Does larger state size (32) justify memory cost?
2. **Memory-Accuracy Tradeoff**: hidden_dim=512 vs hidden_dim=256?
3. **Time-Step Dynamics**: Optimal dt_min/dt_max for HFT tick capture?
4. **Advanced Features**: Do use_ssd and use_selective_state provide lift?

**Hypothesis**: State size 16-32, hidden_dim 256, use_selective_state=true likely optimal

---

## 📁 Output Artifacts

### MinIO Storage
```
s3://foxhunt-ml-models/mamba2/tuning_jobs/{job_id}/
├── optuna_study.db           # JournalStorage for crash recovery
├── trial_results.json         # All 40 trial results
├── best_checkpoint.safetensors
└── analysis/
    ├── sharpe_vs_state_size.png
    ├── memory_vs_accuracy.png
    └── feature_importance.png
```

### Local Analysis
```bash
# Download tuning results
tli tune best --job-id <uuid> --export mamba2_best.yaml

# Use best config for final training
tli train \
  --model MAMBA_2 \
  --config mamba2_best.yaml \
  --epochs 500 \
  --symbols ES.FUT,NQ.FUT,ZN.FUT,6E.FUT
```

---

## 🔒 GPU Memory Safety

### VRAM Monitoring Strategy
- **Pre-trial estimation**: Check if config exceeds 3.5GB
- **Auto-skip**: Dangerous configs (hidden_dim=512, state_size=32, batch_size=64)
- **Graceful fallback**: CPU inference if GPU OOM

### Safe Configurations
```
hidden_dim=256, state_size=16, batch_size=32: 2.2GB (optimal)
hidden_dim=128, state_size=8, batch_size=64:  1.4GB (conservative)
```

### Risky Configurations (will be skipped)
```
hidden_dim=512, state_size=32, batch_size=64: 4.8GB (OOM risk)
```

---

## ✅ Success Criteria

### Must-Have (Critical)
- ✅ Complete 40 trials without crashes
- ✅ Sharpe ratio > 1.50 (match DQN baseline)
- ✅ Inference latency < 200μs
- ✅ VRAM usage < 3.5GB
- ✅ No training instability

### Should-Have (Important)
- ✅ Sharpe ratio > 1.60 (10%+ improvement)
- ✅ MedianPruner saves 30%+ time
- ✅ State-space features (use_ssd, use_selective_state) provide lift
- ✅ Clear hyperparameter trends

### Nice-to-Have (Aspirational)
- ✅ Sharpe ratio > 1.80 (20%+ improvement)
- ✅ Inference latency < 100μs
- ✅ Win rate > 55%
- ✅ Max drawdown < -12%

---

## 🐛 Troubleshooting

### Issue: OOM Errors
**Solution**: Batch size auto-reduced to 16, or trial skipped

### Issue: Training Instability (NaN loss)
**Solution**: Gradient clipping [0.5, 2.0], warmup steps [100, 2000]

### Issue: Poor Sharpe Ratio (<1.50)
**Investigation**:
- Check data quality (665K samples across ES/NQ/ZN/6E)
- Verify feature engineering (16 features + 10 indicators)
- Inspect state-space dynamics (dt_min, dt_max)

### Issue: Slow Tuning (>12 hours)
**Optimization**:
- Increase MedianPruner aggressiveness (n_warmup_steps: 5)
- Reduce trials to 30
- Use cloud GPU (A100: 8x speedup)

---

## 📞 Next Steps After Tuning

1. **Extract Best Config** (5 minutes):
   ```bash
   tli tune best --job-id <uuid> > mamba2_best.yaml
   ```

2. **Train Final Model** (2-3 days):
   ```bash
   tli train --model MAMBA_2 --config mamba2_best.yaml --epochs 500
   ```

3. **Backtest & Validate** (1 day):
   ```bash
   tli backtest \
     --model MAMBA_2 \
     --checkpoint mamba2_final.safetensors \
     --start-date 2024-10-01 \
     --end-date 2024-11-01
   ```

4. **Compare with DQN/PPO** (1 hour):
   - Sharpe ratio comparison
   - Win rate analysis
   - Drawdown risk assessment

5. **Production Deployment Decision**:
   - If Sharpe > 1.70: Deploy to production ensemble
   - If Sharpe 1.50-1.70: Use as diversification model
   - If Sharpe < 1.50: Investigate failure modes

---

## 📚 References

- **Full Technical Report**: `/home/jgrusewski/Work/foxhunt/MAMBA2_HYPERPARAMETER_TUNING_REPORT.md`
- **Tuning Config**: `/home/jgrusewski/Work/foxhunt/services/ml_training_service/tuning_config.yaml`
- **MAMBA-2 Trainer**: `/home/jgrusewski/Work/foxhunt/ml/src/trainers/mamba2.rs`
- **Optuna Integration**: `/home/jgrusewski/Work/foxhunt/OPTUNA_QUICKSTART.md`

---

## 🚀 Ready to Start?

```bash
# Login
tli login

# Start MAMBA-2 tuning (6-10 hours)
tli tune start --model MAMBA_2 --trials 40 --watch

# Let it run overnight, check progress in the morning
tli tune status --job-id <uuid>

# Get best hyperparameters
tli tune best --job-id <uuid>
```

**Expected Completion**: Tomorrow morning (overnight run)
**Best Sharpe Estimate**: 1.60-1.80 (conservative), 1.90-2.20 (optimistic)

---

**Document Version**: 1.0
**Status**: ✅ READY TO EXECUTE
**Last Updated**: 2025-10-14