foxhunt/WAVE_13.2_AGENT_20_QUICK_REFERENCE.md

Wave 13.2 Agent 20: ML Trading Dashboard - Quick Reference

🎯 Mission Summary

Status: ✅ COMPLETE File: /home/jgrusewski/Work/foxhunt/monitoring/grafana/ml_trading_dashboard.json Dashboard URL: http://localhost:3000/d/ml-trading (admin/foxhunt123)

---

📊 Dashboard at a Glance

30 Panels across 6 Rows:

1. Model Performance Overview (3 panels): Win rate, Sharpe ratios, confidence
2. Order Activity (3 panels): Order flow, fill rates, rejection reasons
3. Ensemble Monitoring (3 panels): Agreement rate, disagreement events, active models
4. Performance Metrics (2 panels): Summary table, inference latency
5. Trading Volume & Risk (6 panels): Predictions, order flow, errors, risk rejections
6. System Health & Alerts (2 panels): Active alerts, model deployments
7. GPU & Infrastructure (5 panels): GPU utilization/memory, service status, cache

---

🎨 Key Panels

Critical KPIs (Top Row)

• Panel 2: ML Model Win Rate (Gauge) - Target: >65% green
• Panel 3: Sharpe Ratios (Bar Chart) - Target: >1.5 green
• Panel 4: Prediction Confidence (Histogram) - Distribution 0-1

Order Monitoring

• Panel 6: ML Orders by Model (Stacked Time Series) - Orders/min
• Panel 7: Fill Rate (Stat) - Target: >95% green
• Panel 8: Rejection Reasons (Pie Chart) - Identify issues

Ensemble Health

• Panel 10: Agreement Rate (Gauge) - Target: >85%, Alert: <70%
• Panel 11: Disagreement Events (Time Series) - Spikes = regime change
• Panel 12: Active Models (Stat) - Expected: 4-6 models

Performance Table

• Panel 14: Model Summary (Table) - Accuracy, Sharpe, Returns, Drawdown, Trades
• Panel 15: Inference Latency P99 (Time Series) - Target: <100ms

Alerts & Deployments

• Panel 23: Active ML Alerts (Alert List) - From Agent 19's alerts
• Panel 24: Recent Deployments (Table) - Model version history

GPU Monitoring

• Panel 26: GPU Utilization (Time Series) - Target: 50-80%
• Panel 27: GPU Memory (Time Series) - Alert: >90%

---

🎛️ Dashboard Variables

Variable	Type	Options	Default
$model_filter	Multi-select	All, DQN, MAMBA2, PPO, TFT, TLOB, Liquid	All
$symbol_filter	Multi-select	All, ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT, CL.FUT	All
$datasource	Datasource	Prometheus	Prometheus

---

🚦 Color Thresholds

Model Performance

Metric	Green	Yellow	Red
Win Rate	>65%	55-65%	<55%
Sharpe Ratio	>1.5	1.0-1.5	<0
Fill Rate	>95%	85-95%	<70%
Agreement Rate	>85%	70-85%	<70%

Infrastructure

Metric	Green	Yellow	Red
GPU Utilization	50-80%	<30% or 80-95%	>95%
GPU Memory	<70%	70-90%	>90%
Inference Latency	<50ms	50-100ms	>100ms
Error Rate	<2%	2-5%	>5%

---

📈 Key Metrics (Prometheus)

Model Performance

ml_model_win_rate{model, symbol}              # Win rate (0-1)
ml_model_sharpe_ratio{model, symbol}          # Sharpe ratio
ml_model_accuracy{model}                      # Accuracy (0-1)
ml_predictions_confidence{model, symbol}      # Confidence (0-1)

Orders

ml_orders_submitted_total{model, symbol}      # Counter
ml_orders_filled_total{model, symbol}         # Counter
ml_orders_rejected_total{model, reason}       # Counter

Ensemble

ml_ensemble_agreement_rate{symbol}            # Agreement (0-1)
ml_ensemble_disagreement_events{symbol}       # Counter

GPU

ml_gpu_utilization_percent{gpu_id}            # 0-100%
ml_gpu_memory_used_bytes{gpu_id}              # Bytes
ml_gpu_memory_total_bytes{gpu_id}             # Bytes

---

🔔 Alert Integration

Source: Agent 19's ml_training_alerts.yml

Annotations on Timeline:

1. ML Alerts (red): Firing alerts from Prometheus
2. Model Deployments (green): Deployment events

Alert Categories:

• Performance (NaN, latency, accuracy)
• Availability (service down, model loading)
• Resources (GPU utilization, memory, temperature)
• Quality (drift, distribution shift, confidence)
• Data Pipeline (stale data, feature errors)
• Storage (S3 errors, checkpoint failures)

---

🛠️ Quick Actions

Access Dashboard

# Direct URL
http://localhost:3000/d/ml-trading

# Credentials
admin / foxhunt123

Import Dashboard (First Time)

# Copy to Grafana provisioning
sudo cp monitoring/grafana/ml_trading_dashboard.json \
        /etc/grafana/provisioning/dashboards/

# Restart Grafana
sudo systemctl restart grafana-server

Verify Metrics

# Check ML metrics endpoint
curl http://localhost:9094/metrics | grep ml_

# Check Prometheus targets
curl http://localhost:9090/api/v1/targets | \
  jq '.data.activeTargets[] | select(.job=="ml_training_service")'

---

🚨 Common Scenarios

🔴 High Disagreement (<70% agreement)

• Meaning: Models predict different directions
• Cause: Market regime change, volatility spike
• Action: Reduce position size, review market events

🔴 Low Fill Rate (<70%)

• Meaning: Orders not executing
• Cause: Aggressive pricing, low liquidity
• Action: Check rejection reasons (Panel 8), adjust pricing

🔴 High Latency (>100ms)

• Meaning: Slow predictions
• Cause: GPU contention, cache misses
• Action: Check GPU (Panel 26), cache (Panel 30)

🔴 GPU Memory >90%

• Meaning: Risk of OOM crash
• Cause: Too many models, large batches
• Action: Restart service, reduce batch size

---

📁 Files Created

1. Dashboard JSON: monitoring/grafana/ml_trading_dashboard.json (1,354 lines)
2. Summary Report: WAVE_13.2_AGENT_20_ML_TRADING_DASHBOARD.md (600+ lines)
3. Quick Reference: WAVE_13.2_AGENT_20_QUICK_REFERENCE.md (this file)

---

✅ Validation Checklist

• 30 panels defined across 6 rows
• 3 dashboard variables (model_filter, symbol_filter, datasource)
• Alert integration (annotations for firing alerts)
• Model deployment tracking (annotations for deployments)
• Color thresholds for all KPI panels
• PromQL queries syntactically correct
• 10-second auto-refresh configured
• Multi-model filtering support
• Multi-symbol filtering support
• Integration with Agent 19's alert rules

---

🔗 Related Files

• Prometheus Alerts: monitoring/prometheus/alerts/ml_training_alerts.yml (Agent 19)
• Other Dashboards:
  • monitoring/grafana/api_gateway_dashboard.json
  • monitoring/grafana/ensemble_ml_production.json
  • monitoring/grafana/ml_training_dashboard.json

---

📊 Dashboard Statistics

• Total Panels: 30
• Rows: 6
• Variables: 3
• Annotations: 2
• File Size: 40 KB
• Lines: 1,354
• Refresh Rate: 10 seconds
• Schema Version: 39 (latest Grafana)

---

Status: ✅ PRODUCTION READY Coordinates with: Agent 19 (Prometheus alerts) Access: http://localhost:3000/d/ml-trading