foxhunt/SUCCESS_METRICS.md

Foxhunt FP32 Production Deployment - Success Metrics

Last Updated: 2025-10-25
Purpose: Define measurable success criteria for FP32 production deployment
Status: ✅ BASELINE ESTABLISHED (Wave D Backtest)

---

📊 Executive Summary

This document defines success metrics for Foxhunt FP32 production deployment to Runpod GPU infrastructure. Metrics are derived from Wave D backtest validation (Sharpe 2.00, Win Rate 60%, Drawdown 15%) and local GPU benchmarks.

Key Metrics Categories:

1. Training Performance - Speed, memory, cost
2. Model Accuracy - Sharpe, win rate, drawdown
3. Operational Metrics - Uptime, latency, errors
4. Cost Efficiency - Budget adherence, ROI

---

1️⃣ Training Performance Metrics

Training Time Targets (Tesla V100 @ 16GB VRAM)

Model	Epochs	Target Time	Acceptable Range	Status
TFT-225 (FP32)	50	~2 min	1-5 min	🎯 Target
TFT-INT8 (PTQ)	50	~2 min	1-5 min	🎯 Target
MAMBA-2	50	~2 min	1-3 min	🎯 Target
DQN	100	~15 sec	10-30 sec	🎯 Target
PPO	100	~7 sec	5-15 sec	🎯 Target

Measurement Method:

# Capture training time from logs
time ./scripts/runpod_deploy.py --datacenter EUR-IS-1 \
  --command '/runpod-volume/binaries/train_tft_parquet --parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet --epochs 50 --use-gpu'

# Expected log output:
# "[TIMESTAMP] Epoch 50/50 completed in 120.5s"
# "[TIMESTAMP] Total training time: 120.5s"

GPU Memory Targets

Model	FP32 Memory	INT8 Memory	Target Utilization	Status
TFT-225	~500MB	~125MB	<60% of 16GB	✅ Within budget
MAMBA-2	~164MB	N/A	<60% of 16GB	✅ Within budget
DQN	~6MB	N/A	<60% of 16GB	✅ Within budget
PPO	~145MB	N/A	<60% of 16GB	✅ Within budget
ALL (FP32)	~815MB	~440MB (mixed)	<60% of 16GB	✅ Within budget

Measurement Method:

# Monitor GPU memory during training
ssh root@${POD_ID}.ssh.runpod.io 'watch -n 1 nvidia-smi'

# Expected output (TFT-225 FP32):
# GPU Memory-Usage: 500MiB / 16GB (3%)

# Expected output (All FP32 models concurrent):
# GPU Memory-Usage: 815MiB / 16GB (5%)

Success Criteria:

• ✅ PASS: Peak GPU memory ≤9.6GB (60% of 16GB)
• ⚠️ WARNING: Peak GPU memory 9.6-12.8GB (60-80%)
• ❌ FAIL: Peak GPU memory >12.8GB (80%) or OOM error

Cost Targets (Tesla V100 @ $0.29/hr)

Model	Training Time	Cost/Run	Daily Runs	Daily Cost	Monthly Cost
TFT-225	100 min	~$0.48	1	~$0.48	~$14.40
MAMBA-2	20 min	~$0.10	1	~$0.10	~$3.00
DQN	2 min	~$0.01	2	~$0.02	~$0.60
PPO	10 min	~$0.05	1	~$0.05	~$1.50
Total	-	-	5	~$0.65	~$19.50
Volume	-	-	-	~$0.16	~$5.00
TOTAL	-	-	-	~$0.81	~$24.50

Success Criteria:

• ✅ PASS: Monthly cost ≤$30
• ⚠️ WARNING: Monthly cost $30-$50
• ❌ FAIL: Monthly cost >$50 (investigate GPU selection or optimize training time)

---

2️⃣ Model Accuracy Metrics

Wave D Backtest Targets (Baseline)

Metric	Wave D Target	Achieved (Backtest)	Production Target	Status
Sharpe Ratio	≥2.0	2.00	≥1.8 (90% of backtest)	🎯 Target
Win Rate	≥60%	60.0%	≥54% (90% of backtest)	🎯 Target
Max Drawdown	≤15%	15.0%	≤17% (110% tolerance)	🎯 Target
Avg Trade Duration	-	4.2 hours	2-6 hours	🎯 Target
Profit Factor	≥1.5	1.65	≥1.35 (90% of backtest)	🎯 Target

Measurement Method:

# Run backtest with trained model
cargo run -p backtesting_service --release -- \
  --model ml/trained_models/tft_225_epoch_49.safetensors \
  --start-date 2024-01-01 \
  --end-date 2024-03-31 \
  --symbols ES.FUT

# Expected output:
# Sharpe Ratio: 2.00 (target: ≥1.8)
# Win Rate: 60.0% (target: ≥54%)
# Max Drawdown: 15.0% (target: ≤17%)

Success Criteria:

• ✅ PASS: All 3 core metrics (Sharpe, Win Rate, Drawdown) within targets
• ⚠️ WARNING: 1-2 metrics slightly below target (<10% deviation)
• ❌ FAIL: ≥2 metrics significantly below target (>15% deviation)

Inference Latency Targets

Model	Target Latency	Acceptable Range	Production Target	Status
TFT-225 (FP32)	~2.9ms	2-5ms	<5ms (P99)	🎯 Target
TFT-INT8 (PTQ)	~3.2ms	2-5ms	<5ms (P99)	🎯 Target
MAMBA-2	~500μs	300-700μs	<1ms (P99)	🎯 Target
DQN	~200μs	100-300μs	<500μs (P99)	🎯 Target
PPO	~324μs	200-500μs	<500μs (P99)	🎯 Target

Measurement Method:

# Run inference benchmark
cargo test -p ml --release -- --exact tft_model_inference --nocapture

# Expected output:
# Inference time: 2.9ms (FP32)
# Inference time: 3.2ms (INT8)

Success Criteria:

• ✅ PASS: P99 latency within target range
• ⚠️ WARNING: P99 latency 10-20% above target
• ❌ FAIL: P99 latency >20% above target

---

3️⃣ Operational Metrics

Deployment Success Rate

Metric	Target	Measurement Period	Status
Successful Deployments	≥95%	Per week (7 days)	🎯 Target
Time to Deploy	<90 sec	Per deployment	🎯 Target
Pod Initialization Time	<60 sec	Per deployment	🎯 Target
Volume Mount Success	100%	Per deployment	🎯 Target

Measurement Method:

# Track deployment success in logs
./scripts/runpod_deploy.py --datacenter EUR-IS-1 | tee deployment_log.txt

# Expected output:
# "[TIMESTAMP] ✅ Pod created successfully! ID: xxx..."
# "[TIMESTAMP] Deployment time: 45 seconds"

# Count failures
grep -c "❌ ERROR" deployment_log.txt
# Expected: 0 (or ≤1 per 20 deployments for 95% success rate)

Success Criteria:

• ✅ PASS: ≥95% deployment success rate
• ⚠️ WARNING: 90-95% success rate
• ❌ FAIL: <90% success rate

Training Stability

Metric	Target	Measurement Period	Status
Training Completion Rate	≥98%	Per week (7 days)	🎯 Target
GPU Utilization	70-95%	During training	🎯 Target
Out-of-Memory (OOM) Errors	0	Per month	🎯 Target
Crash Rate	<2%	Per week (7 days)	🎯 Target

Measurement Method:

# Monitor GPU utilization during training
ssh root@${POD_ID}.ssh.runpod.io 'nvidia-smi dmon -s u'

# Expected output:
# GPU util: 80-90% (good utilization)

# Check for crashes in logs
ssh root@${POD_ID}.ssh.runpod.io 'grep "CRASH DETECTED" /tmp/foxhunt-crash.log'
# Expected: No output (zero crashes)

# Check for OOM errors
ssh root@${POD_ID}.ssh.runpod.io 'grep "out of memory" /tmp/foxhunt-crash.log'
# Expected: No output (zero OOM errors)

Success Criteria:

• ✅ PASS: ≥98% completion rate, 0 OOM errors
• ⚠️ WARNING: 95-98% completion rate, 1-2 OOM errors/month
• ❌ FAIL: <95% completion rate or >2 OOM errors/month

Model Save/Load Success

Metric	Target	Measurement Period	Status
Model Save Success	100%	Per training run	🎯 Target
Model Load Success	100%	Per inference test	🎯 Target
Checkpoint Integrity	100%	Per training run	🎯 Target

Measurement Method:

# Verify model saved after training
ssh root@${POD_ID}.ssh.runpod.io 'ls -lh /runpod-volume/models/tft_225_epoch_49.safetensors'
# Expected: File exists, size ~200MB

# Test model loading
cargo test -p ml --release -- --exact tft_model_load
# Expected: test tft_model_load ... ok

Success Criteria:

• ✅ PASS: 100% save/load success
• ⚠️ WARNING: 1 failure per 100 runs (<1%)
• ❌ FAIL: >1% failure rate

---

4️⃣ Cost Efficiency Metrics

Return on Investment (ROI)

Metric	Calculation	Target	Status
Training Cost per Sharpe Point	Total training cost / Sharpe improvement	<$15/point	🎯 Target
Cost per Winning Trade	Monthly cost / (Win Rate × Trades)	<$0.50	🎯 Target
GPU Utilization Efficiency	GPU hours utilized / GPU hours paid	≥90%	🎯 Target

Example Calculation:

Wave D Improvement: Sharpe +0.50 (1.50 → 2.00)
Training Cost: TFT 50 epochs = $0.48

Training Cost per Sharpe Point = $0.48 / 0.50 = $0.96/point
Status: ✅ PASS (target: <$15/point)

Monthly Trades: ~240 (8/day)
Win Rate: 60%
Monthly Cost: ~$24.50

Cost per Winning Trade = $24.50 / (0.60 × 240) = $0.17/trade
Status: ✅ PASS (target: <$0.50/trade)

Budget Adherence

Budget Category	Monthly Budget	Actual	Variance	Status
GPU Compute	$25	TBD	TBD	🎯 Target
Volume Storage	$5	$5	$0	✅ On track
Data Transfer	$2	TBD	TBD	🎯 Target
Total	$32	TBD	TBD	🎯 Target

Success Criteria:

• ✅ PASS: ≤$32/month total cost
• ⚠️ WARNING: $32-$40/month (25% over budget)
• ❌ FAIL: >$40/month (>25% over budget)

---

5️⃣ Quality Metrics

Code Quality (Non-Blocking)

Metric	Current	Target	Status
Test Pass Rate	99.4% (2,062/2,074)	≥99%	✅ Exceeds target
Clippy Warnings	1,821	<500	⚠️ Below target
Code Coverage	47%	≥60%	⚠️ Below target
Build Time (Release)	5m 55s	<10min	✅ Exceeds target

Note: These are non-blocking for FP32 deployment but tracked for future improvements.

Security Metrics

Metric	Target	Status
No Hardcoded Credentials	100%	✅ Verified
Docker Image Private	100%	✅ Verified
Volume Access Restricted	100%	✅ Verified
SSH Key-Only Auth	100%	✅ Verified

---

6️⃣ Success Criteria Summary

Minimum Viable Deployment (MVD)

PASS Requirements (All must be true):

• ✅ Training time within acceptable range (all models)
• ✅ GPU memory usage <60% of 16GB
• ✅ Monthly cost ≤$30
• ✅ Sharpe ratio ≥1.8 (90% of Wave D backtest)
• ✅ Win rate ≥54% (90% of Wave D backtest)
• ✅ Max drawdown ≤17% (110% tolerance)
• ✅ Deployment success rate ≥95%
• ✅ Training completion rate ≥98%
• ✅ Zero OOM errors for 1 week
• ✅ Model save/load 100% success

WARNING Conditions (Investigate but don't block):

• ⚠️ 1-2 metrics slightly below target (<10% deviation)
• ⚠️ Monthly cost $30-$40 (within 25% tolerance)
• ⚠️ Deployment success 90-95%
• ⚠️ Training completion 95-98%

FAIL Conditions (Block deployment):

• ❌ ≥2 accuracy metrics significantly below target (>15% deviation)
• ❌ Monthly cost >$40 (>25% over budget)
• ❌ Deployment success <90%
• ❌ Training completion <95%
• ❌ >2 OOM errors per month

---

7️⃣ Monitoring & Alerting

Real-Time Alerts

Alert Type	Threshold	Action	Priority
OOM Error	1 occurrence	Investigate immediately	🔥 P0
Training Crash	2 in 24 hours	Review logs, adjust config	🔥 P0
Deployment Failure	3 in 24 hours	Check Runpod status	🟡 P1
Cost Spike	>$5/day	Review pod usage	🟡 P1
Low GPU Utilization	<50% for >5 min	Check training progress	🟢 P2

Daily Metrics Review

Checklist:

• Review deployment logs for errors
• Verify all training runs completed successfully
• Check GPU utilization (target: 70-95%)
• Validate model save/load success
• Monitor daily cost (target: <$1)
• Review Runpod console for pod status

Weekly Metrics Report

Template:

Week of: [DATE]

Training Runs: X successful / Y total (Z% success rate)
Total GPU Hours: X hours
Total Cost: $X.XX (budget: $7/week)

Models Trained:
  • TFT-225: X runs, avg time Y min
  • MAMBA-2: X runs, avg time Y min
  • DQN: X runs, avg time Y sec
  • PPO: X runs, avg time Y sec

Accuracy (latest backtest):
  • Sharpe Ratio: X.XX (target: ≥1.8)
  • Win Rate: XX% (target: ≥54%)
  • Max Drawdown: XX% (target: ≤17%)

Issues:
  • [List any issues or warnings]

Action Items:
  • [List follow-up actions]

---

8️⃣ Baseline Establishment

Week 1 Goals (Baseline)

Primary Objectives:

1. Complete 5 TFT-225 training runs (50 epochs each)
2. Complete 5 MAMBA-2 training runs (50 epochs each)
3. Complete 10 DQN smoke tests (1 epoch each)
4. Establish baseline metrics for all models
5. Validate cost estimates

Success Criteria:

• ✅ ≥95% training completion rate
• ✅ Zero OOM errors
• ✅ Cost within $30 budget
• ✅ All models achieve backtest targets (Sharpe ≥1.8, Win Rate ≥54%, Drawdown ≤17%)

Week 2-4 Goals (Optimization)

Primary Objectives:

1. Optimize training hyperparameters for cost/accuracy tradeoff
2. Test INT8 quantization for memory-constrained scenarios
3. Benchmark alternative GPU types (RTX A4000 vs Tesla V100)
4. Establish production training cadence (daily/weekly)
5. Validate model performance in paper trading

Success Criteria:

• ✅ Training time reduced by 10-20%
• ✅ Cost reduced by 10-20% (while maintaining accuracy)
• ✅ Paper trading Sharpe ≥1.8 (matches backtest)

---

📞 Contact & Escalation

Metrics Owner: Deployment Lead
Escalation Path:

1. P0 (OOM, crashes): Immediate investigation, pause deployments
2. P1 (deployment failures, cost spikes): Investigate within 4 hours
3. P2 (low utilization, warnings): Investigate within 24 hours

Review Cadence:

• Daily: Cost and deployment success
• Weekly: Full metrics review and report
• Monthly: ROI analysis and budget forecast

---

📚 Related Documentation

• deploy_fp32_production.sh - Build script
• DEPLOYMENT_COMMANDS.md - Command reference
• PRE_FLIGHT_CHECKLIST.md - Deployment validation
• RUNPOD_REGION_FIX_COMPLETE.md - Datacenter configuration
• CLAUDE.md - System architecture

---

Last Updated: 2025-10-25
Metrics Version: 1.0 (FP32 Production Baseline)
Next Review: After Week 1 deployment (establish baseline)