foxhunt/SUCCESS_METRICS.md

# Foxhunt FP32 Production Deployment - Success Metrics

**Last Updated**: 2025-10-25  
**Purpose**: Define measurable success criteria for FP32 production deployment  
**Status**: ✅ BASELINE ESTABLISHED (Wave D Backtest)

---

## 📊 Executive Summary

This document defines success metrics for Foxhunt FP32 production deployment to Runpod GPU infrastructure. Metrics are derived from Wave D backtest validation (Sharpe 2.00, Win Rate 60%, Drawdown 15%) and local GPU benchmarks.

**Key Metrics Categories**:
1. **Training Performance** - Speed, memory, cost
2. **Model Accuracy** - Sharpe, win rate, drawdown
3. **Operational Metrics** - Uptime, latency, errors
4. **Cost Efficiency** - Budget adherence, ROI

---

## 1️⃣ Training Performance Metrics

### Training Time Targets (Tesla V100 @ 16GB VRAM)

| Model | Epochs | Target Time | Acceptable Range | Status |
|---|---|---|---|---|
| **TFT-225 (FP32)** | 50 | ~2 min | 1-5 min | 🎯 Target |
| **TFT-INT8 (PTQ)** | 50 | ~2 min | 1-5 min | 🎯 Target |
| **MAMBA-2** | 50 | ~2 min | 1-3 min | 🎯 Target |
| **DQN** | 100 | ~15 sec | 10-30 sec | 🎯 Target |
| **PPO** | 100 | ~7 sec | 5-15 sec | 🎯 Target |

**Measurement Method**:
```bash
# Capture training time from logs
time ./scripts/runpod_deploy.py --datacenter EUR-IS-1 \
  --command '/runpod-volume/binaries/train_tft_parquet --parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet --epochs 50 --use-gpu'

# Expected log output:
# "[TIMESTAMP] Epoch 50/50 completed in 120.5s"
# "[TIMESTAMP] Total training time: 120.5s"
```

### GPU Memory Targets

| Model | FP32 Memory | INT8 Memory | Target Utilization | Status |
|---|---|---|---|---|
| **TFT-225** | ~500MB | ~125MB | <60% of 16GB | ✅ Within budget |
| **MAMBA-2** | ~164MB | N/A | <60% of 16GB | ✅ Within budget |
| **DQN** | ~6MB | N/A | <60% of 16GB | ✅ Within budget |
| **PPO** | ~145MB | N/A | <60% of 16GB | ✅ Within budget |
| **ALL (FP32)** | ~815MB | ~440MB (mixed) | <60% of 16GB | ✅ Within budget |

**Measurement Method**:
```bash
# Monitor GPU memory during training
ssh root@${POD_ID}.ssh.runpod.io 'watch -n 1 nvidia-smi'

# Expected output (TFT-225 FP32):
# GPU Memory-Usage: 500MiB / 16GB (3%)

# Expected output (All FP32 models concurrent):
# GPU Memory-Usage: 815MiB / 16GB (5%)
```

**Success Criteria**:
- ✅ **PASS**: Peak GPU memory ≤9.6GB (60% of 16GB)
- ⚠️ **WARNING**: Peak GPU memory 9.6-12.8GB (60-80%)
- ❌ **FAIL**: Peak GPU memory >12.8GB (80%) or OOM error

### Cost Targets (Tesla V100 @ $0.29/hr)

| Model | Training Time | Cost/Run | Daily Runs | Daily Cost | Monthly Cost |
|---|---|---|---|---|---|
| **TFT-225** | 100 min | ~$0.48 | 1 | ~$0.48 | ~$14.40 |
| **MAMBA-2** | 20 min | ~$0.10 | 1 | ~$0.10 | ~$3.00 |
| **DQN** | 2 min | ~$0.01 | 2 | ~$0.02 | ~$0.60 |
| **PPO** | 10 min | ~$0.05 | 1 | ~$0.05 | ~$1.50 |
| **Total** | - | - | **5** | **~$0.65** | **~$19.50** |
| **Volume** | - | - | - | ~$0.16 | ~$5.00 |
| **TOTAL** | - | - | - | **~$0.81** | **~$24.50** |

**Success Criteria**:
- ✅ **PASS**: Monthly cost ≤$30
- ⚠️ **WARNING**: Monthly cost $30-$50
- ❌ **FAIL**: Monthly cost >$50 (investigate GPU selection or optimize training time)

---

## 2️⃣ Model Accuracy Metrics

### Wave D Backtest Targets (Baseline)

| Metric | Wave D Target | Achieved (Backtest) | Production Target | Status |
|---|---|---|---|---|
| **Sharpe Ratio** | ≥2.0 | 2.00 | ≥1.8 (90% of backtest) | 🎯 Target |
| **Win Rate** | ≥60% | 60.0% | ≥54% (90% of backtest) | 🎯 Target |
| **Max Drawdown** | ≤15% | 15.0% | ≤17% (110% tolerance) | 🎯 Target |
| **Avg Trade Duration** | - | 4.2 hours | 2-6 hours | 🎯 Target |
| **Profit Factor** | ≥1.5 | 1.65 | ≥1.35 (90% of backtest) | 🎯 Target |

**Measurement Method**:
```bash
# Run backtest with trained model
cargo run -p backtesting_service --release -- \
  --model ml/trained_models/tft_225_epoch_49.safetensors \
  --start-date 2024-01-01 \
  --end-date 2024-03-31 \
  --symbols ES.FUT

# Expected output:
# Sharpe Ratio: 2.00 (target: ≥1.8)
# Win Rate: 60.0% (target: ≥54%)
# Max Drawdown: 15.0% (target: ≤17%)
```

**Success Criteria**:
- ✅ **PASS**: All 3 core metrics (Sharpe, Win Rate, Drawdown) within targets
- ⚠️ **WARNING**: 1-2 metrics slightly below target (<10% deviation)
- ❌ **FAIL**: ≥2 metrics significantly below target (>15% deviation)

### Inference Latency Targets

| Model | Target Latency | Acceptable Range | Production Target | Status |
|---|---|---|---|---|
| **TFT-225 (FP32)** | ~2.9ms | 2-5ms | <5ms (P99) | 🎯 Target |
| **TFT-INT8 (PTQ)** | ~3.2ms | 2-5ms | <5ms (P99) | 🎯 Target |
| **MAMBA-2** | ~500μs | 300-700μs | <1ms (P99) | 🎯 Target |
| **DQN** | ~200μs | 100-300μs | <500μs (P99) | 🎯 Target |
| **PPO** | ~324μs | 200-500μs | <500μs (P99) | 🎯 Target |

**Measurement Method**:
```bash
# Run inference benchmark
cargo test -p ml --release -- --exact tft_model_inference --nocapture

# Expected output:
# Inference time: 2.9ms (FP32)
# Inference time: 3.2ms (INT8)
```

**Success Criteria**:
- ✅ **PASS**: P99 latency within target range
- ⚠️ **WARNING**: P99 latency 10-20% above target
- ❌ **FAIL**: P99 latency >20% above target

---

## 3️⃣ Operational Metrics

### Deployment Success Rate

| Metric | Target | Measurement Period | Status |
|---|---|---|---|
| **Successful Deployments** | ≥95% | Per week (7 days) | 🎯 Target |
| **Time to Deploy** | <90 sec | Per deployment | 🎯 Target |
| **Pod Initialization Time** | <60 sec | Per deployment | 🎯 Target |
| **Volume Mount Success** | 100% | Per deployment | 🎯 Target |

**Measurement Method**:
```bash
# Track deployment success in logs
./scripts/runpod_deploy.py --datacenter EUR-IS-1 | tee deployment_log.txt

# Expected output:
# "[TIMESTAMP] ✅ Pod created successfully! ID: xxx..."
# "[TIMESTAMP] Deployment time: 45 seconds"

# Count failures
grep -c "❌ ERROR" deployment_log.txt
# Expected: 0 (or ≤1 per 20 deployments for 95% success rate)
```

**Success Criteria**:
- ✅ **PASS**: ≥95% deployment success rate
- ⚠️ **WARNING**: 90-95% success rate
- ❌ **FAIL**: <90% success rate

### Training Stability

| Metric | Target | Measurement Period | Status |
|---|---|---|---|
| **Training Completion Rate** | ≥98% | Per week (7 days) | 🎯 Target |
| **GPU Utilization** | 70-95% | During training | 🎯 Target |
| **Out-of-Memory (OOM) Errors** | 0 | Per month | 🎯 Target |
| **Crash Rate** | <2% | Per week (7 days) | 🎯 Target |

**Measurement Method**:
```bash
# Monitor GPU utilization during training
ssh root@${POD_ID}.ssh.runpod.io 'nvidia-smi dmon -s u'

# Expected output:
# GPU util: 80-90% (good utilization)

# Check for crashes in logs
ssh root@${POD_ID}.ssh.runpod.io 'grep "CRASH DETECTED" /tmp/foxhunt-crash.log'
# Expected: No output (zero crashes)

# Check for OOM errors
ssh root@${POD_ID}.ssh.runpod.io 'grep "out of memory" /tmp/foxhunt-crash.log'
# Expected: No output (zero OOM errors)
```

**Success Criteria**:
- ✅ **PASS**: ≥98% completion rate, 0 OOM errors
- ⚠️ **WARNING**: 95-98% completion rate, 1-2 OOM errors/month
- ❌ **FAIL**: <95% completion rate or >2 OOM errors/month

### Model Save/Load Success

| Metric | Target | Measurement Period | Status |
|---|---|---|---|
| **Model Save Success** | 100% | Per training run | 🎯 Target |
| **Model Load Success** | 100% | Per inference test | 🎯 Target |
| **Checkpoint Integrity** | 100% | Per training run | 🎯 Target |

**Measurement Method**:
```bash
# Verify model saved after training
ssh root@${POD_ID}.ssh.runpod.io 'ls -lh /runpod-volume/models/tft_225_epoch_49.safetensors'
# Expected: File exists, size ~200MB

# Test model loading
cargo test -p ml --release -- --exact tft_model_load
# Expected: test tft_model_load ... ok
```

**Success Criteria**:
- ✅ **PASS**: 100% save/load success
- ⚠️ **WARNING**: 1 failure per 100 runs (<1%)
- ❌ **FAIL**: >1% failure rate

---

## 4️⃣ Cost Efficiency Metrics

### Return on Investment (ROI)

| Metric | Calculation | Target | Status |
|---|---|---|---|
| **Training Cost per Sharpe Point** | Total training cost / Sharpe improvement | <$15/point | 🎯 Target |
| **Cost per Winning Trade** | Monthly cost / (Win Rate × Trades) | <$0.50 | 🎯 Target |
| **GPU Utilization Efficiency** | GPU hours utilized / GPU hours paid | ≥90% | 🎯 Target |

**Example Calculation**:
```
Wave D Improvement: Sharpe +0.50 (1.50 → 2.00)
Training Cost: TFT 50 epochs = $0.48

Training Cost per Sharpe Point = $0.48 / 0.50 = $0.96/point
Status: ✅ PASS (target: <$15/point)

Monthly Trades: ~240 (8/day)
Win Rate: 60%
Monthly Cost: ~$24.50

Cost per Winning Trade = $24.50 / (0.60 × 240) = $0.17/trade
Status: ✅ PASS (target: <$0.50/trade)
```

### Budget Adherence

| Budget Category | Monthly Budget | Actual | Variance | Status |
|---|---|---|---|---|
| **GPU Compute** | $25 | TBD | TBD | 🎯 Target |
| **Volume Storage** | $5 | $5 | $0 | ✅ On track |
| **Data Transfer** | $2 | TBD | TBD | 🎯 Target |
| **Total** | **$32** | **TBD** | **TBD** | 🎯 Target |

**Success Criteria**:
- ✅ **PASS**: ≤$32/month total cost
- ⚠️ **WARNING**: $32-$40/month (25% over budget)
- ❌ **FAIL**: >$40/month (>25% over budget)

---

## 5️⃣ Quality Metrics

### Code Quality (Non-Blocking)

| Metric | Current | Target | Status |
|---|---|---|---|
| **Test Pass Rate** | 99.4% (2,062/2,074) | ≥99% | ✅ Exceeds target |
| **Clippy Warnings** | 1,821 | <500 | ⚠️ Below target |
| **Code Coverage** | 47% | ≥60% | ⚠️ Below target |
| **Build Time (Release)** | 5m 55s | <10min | ✅ Exceeds target |

**Note**: These are non-blocking for FP32 deployment but tracked for future improvements.

### Security Metrics

| Metric | Target | Status |
|---|---|---|
| **No Hardcoded Credentials** | 100% | ✅ Verified |
| **Docker Image Private** | 100% | ✅ Verified |
| **Volume Access Restricted** | 100% | ✅ Verified |
| **SSH Key-Only Auth** | 100% | ✅ Verified |

---

## 6️⃣ Success Criteria Summary

### Minimum Viable Deployment (MVD)

**PASS Requirements** (All must be true):
- ✅ Training time within acceptable range (all models)
- ✅ GPU memory usage <60% of 16GB
- ✅ Monthly cost ≤$30
- ✅ Sharpe ratio ≥1.8 (90% of Wave D backtest)
- ✅ Win rate ≥54% (90% of Wave D backtest)
- ✅ Max drawdown ≤17% (110% tolerance)
- ✅ Deployment success rate ≥95%
- ✅ Training completion rate ≥98%
- ✅ Zero OOM errors for 1 week
- ✅ Model save/load 100% success

**WARNING Conditions** (Investigate but don't block):
- ⚠️ 1-2 metrics slightly below target (<10% deviation)
- ⚠️ Monthly cost $30-$40 (within 25% tolerance)
- ⚠️ Deployment success 90-95%
- ⚠️ Training completion 95-98%

**FAIL Conditions** (Block deployment):
- ❌ ≥2 accuracy metrics significantly below target (>15% deviation)
- ❌ Monthly cost >$40 (>25% over budget)
- ❌ Deployment success <90%
- ❌ Training completion <95%
- ❌ >2 OOM errors per month

---

## 7️⃣ Monitoring & Alerting

### Real-Time Alerts

| Alert Type | Threshold | Action | Priority |
|---|---|---|---|
| **OOM Error** | 1 occurrence | Investigate immediately | 🔥 P0 |
| **Training Crash** | 2 in 24 hours | Review logs, adjust config | 🔥 P0 |
| **Deployment Failure** | 3 in 24 hours | Check Runpod status | 🟡 P1 |
| **Cost Spike** | >$5/day | Review pod usage | 🟡 P1 |
| **Low GPU Utilization** | <50% for >5 min | Check training progress | 🟢 P2 |

### Daily Metrics Review

**Checklist**:
- [ ] Review deployment logs for errors
- [ ] Verify all training runs completed successfully
- [ ] Check GPU utilization (target: 70-95%)
- [ ] Validate model save/load success
- [ ] Monitor daily cost (target: <$1)
- [ ] Review Runpod console for pod status

### Weekly Metrics Report

**Template**:
```
Week of: [DATE]

Training Runs: X successful / Y total (Z% success rate)
Total GPU Hours: X hours
Total Cost: $X.XX (budget: $7/week)

Models Trained:
  • TFT-225: X runs, avg time Y min
  • MAMBA-2: X runs, avg time Y min
  • DQN: X runs, avg time Y sec
  • PPO: X runs, avg time Y sec

Accuracy (latest backtest):
  • Sharpe Ratio: X.XX (target: ≥1.8)
  • Win Rate: XX% (target: ≥54%)
  • Max Drawdown: XX% (target: ≤17%)

Issues:
  • [List any issues or warnings]

Action Items:
  • [List follow-up actions]
```

---

## 8️⃣ Baseline Establishment

### Week 1 Goals (Baseline)

**Primary Objectives**:
1. Complete 5 TFT-225 training runs (50 epochs each)
2. Complete 5 MAMBA-2 training runs (50 epochs each)
3. Complete 10 DQN smoke tests (1 epoch each)
4. Establish baseline metrics for all models
5. Validate cost estimates

**Success Criteria**:
- ✅ ≥95% training completion rate
- ✅ Zero OOM errors
- ✅ Cost within $30 budget
- ✅ All models achieve backtest targets (Sharpe ≥1.8, Win Rate ≥54%, Drawdown ≤17%)

### Week 2-4 Goals (Optimization)

**Primary Objectives**:
1. Optimize training hyperparameters for cost/accuracy tradeoff
2. Test INT8 quantization for memory-constrained scenarios
3. Benchmark alternative GPU types (RTX A4000 vs Tesla V100)
4. Establish production training cadence (daily/weekly)
5. Validate model performance in paper trading

**Success Criteria**:
- ✅ Training time reduced by 10-20%
- ✅ Cost reduced by 10-20% (while maintaining accuracy)
- ✅ Paper trading Sharpe ≥1.8 (matches backtest)

---

## 📞 Contact & Escalation

**Metrics Owner**: Deployment Lead  
**Escalation Path**:
1. **P0 (OOM, crashes)**: Immediate investigation, pause deployments
2. **P1 (deployment failures, cost spikes)**: Investigate within 4 hours
3. **P2 (low utilization, warnings)**: Investigate within 24 hours

**Review Cadence**:
- **Daily**: Cost and deployment success
- **Weekly**: Full metrics review and report
- **Monthly**: ROI analysis and budget forecast

---

## 📚 Related Documentation

- `deploy_fp32_production.sh` - Build script
- `DEPLOYMENT_COMMANDS.md` - Command reference
- `PRE_FLIGHT_CHECKLIST.md` - Deployment validation
- `RUNPOD_REGION_FIX_COMPLETE.md` - Datacenter configuration
- `CLAUDE.md` - System architecture

---

**Last Updated**: 2025-10-25  
**Metrics Version**: 1.0 (FP32 Production Baseline)  
**Next Review**: After Week 1 deployment (establish baseline)