foxhunt/WAVE_13_AGENT_19_QUICK_REFERENCE.md

Agent 19 Quick Reference: ML Trading Metrics

Mission: Add Prometheus metrics for ML trading operations Status: ✅ COMPLETE Date: 2025-10-16

---

📦 Files Created

1. services/trading_service/src/metrics.rs (700+ lines)

   • 16 base Prometheus metrics
   • 5 helper functions
   • 11 unit tests

2. monitoring/prometheus/trading_service_metrics.yml (400+ lines)

   • Metric definitions
   • Derived metrics (5)
   • Grafana query examples

3. monitoring/prometheus/alerts/ml_trading_alerts.yml (500+ lines)

   • 25 alerts across 9 groups
   • Severity levels: INFO, WARNING, CRITICAL, EMERGENCY

---

🎯 Key Metrics

Predictions (4 metrics):

ml_predictions_total{model_id, symbol, action}        // Counter
ml_predictions_confidence{model_id}                   // Histogram
ml_prediction_accuracy{model_id}                      // Gauge (0-100%)
ml_model_last_prediction_time{model_id}               // Gauge (Unix epoch)

Orders (3 metrics):

ml_orders_submitted_total{model_id, symbol}           // Counter
ml_orders_filled_total{model_id, symbol}              // Counter
ml_orders_rejected_total{model_id, symbol, reason}    // Counter

Performance (6 metrics):

ml_model_sharpe_ratio{model_id}                       // Gauge (target: >1.5)
ml_model_win_rate{model_id}                           // Gauge (0.0-1.0)
ml_model_avg_return{model_id}                         // Gauge (dollars)
ml_model_inference_latency{model_id}                  // Histogram (μs)
ml_model_cumulative_pnl{model_id}                     // Gauge (dollars)
ml_model_max_drawdown{model_id}                       // Gauge (dollars)

Ensemble (2 metrics):

ml_ensemble_agreement_rate{symbol}                    // Gauge (0.0-1.0)
ml_ensemble_disagreement_events{symbol, threshold}    // Counter

---

🔧 Usage Examples

Record Prediction:

use trading_service::metrics::record_ml_prediction;

record_ml_prediction("DQN", "ES.FUT", "buy", 0.85, 125.5);
//                   model   symbol    action conf  latency_us

Record Orders:

use trading_service::metrics::{
    record_ml_order_submitted,
    record_ml_order_filled,
    record_ml_order_rejected,
};

record_ml_order_submitted("DQN", "ES.FUT");
record_ml_order_filled("DQN", "ES.FUT");
record_ml_order_rejected("DQN", "ES.FUT", "risk_limit");

Update Performance:

use trading_service::metrics::update_ml_model_performance;

update_ml_model_performance("DQN", 1.85, 0.62, 125.50, 0.68);
//                          model  sharpe  win   avg_ret  accuracy

Record Ensemble Vote:

use trading_service::metrics::record_ensemble_vote;

record_ensemble_vote("ES.FUT", 0.85);
//                   symbol    agreement_rate

---

🚨 Critical Alerts

EMERGENCY (Priority: High):

MLTradingSystemDown          # No predictions for 10m
MLOrderSubmissionFailure     # Predictions but no orders for 5m

CRITICAL:

MLOrderRiskLimitRejections   # >0.1/sec risk rejections
MLInferenceLatencyCritical   # P99 >5ms for 2m
MLModelLargeDrawdown         # Drawdown <-$5000 for 1h
MLEnsembleVotingStopped      # No votes for 15m

WARNING:

MLModelLowAccuracy           # <55% accuracy for 1h
MLModelLowWinRate            # <55% win rate for 2h
MLModelStalePredictions      # No predictions >1h
MLOrderRejectionRateHigh     # >10% rejection for 5m
MLEnsembleDisagreementFrequent  # >0.3 events/sec

---

📊 Grafana Queries (for Agent 20)

Model Performance Comparison:

ml_model_sharpe_ratio > 1.0

Order Fill Rate:

100 * (
  sum by (model_id, symbol) (rate(ml_orders_filled_total[5m]))
  /
  sum by (model_id, symbol) (rate(ml_orders_submitted_total[5m]))
)

Ensemble Disagreement Heatmap:

ml_ensemble_disagreement_rate{symbol="ES.FUT"}

Prediction Volume by Action:

sum by (action) (rate(ml_predictions_total[5m]))

Model PnL Leaderboard:

topk(5, ml_model_cumulative_pnl)

Inference Latency P99:

histogram_quantile(0.99,
  sum by (model_id, le) (rate(ml_model_inference_latency_bucket[5m]))
)

---

🎯 Production Targets

Metric	Target	Alert Threshold
Accuracy	>55%	<55% for 1h
Win Rate	>55%	<55% for 2h
Sharpe Ratio	>1.5	<1.5 for 6h
Inference P99	<1ms	>1ms for 5m
Fill Rate	>90%	<80% for 10m
Rejection Rate	<10%	>10% for 5m
Agreement	>50%	<50% for 10m

---

✅ Validation

Compile:

cargo check -p trading_service

Test:

cargo test -p trading_service metrics

Lint:

cargo clippy -p trading_service -- -D warnings

---

🔗 Integration Points

Existing Metrics Modules:

• ml_metrics.rs - Model health, fallback, drift
• ensemble_metrics.rs - Aggregation, A/B testing, swaps
• metrics_server.rs - HTTP server for Prometheus

Alert Files:

• ml_training_alerts.yml - Training performance, GPU
• ensemble_ml_alerts.yml - Ensemble aggregation, A/B tests

Database:

• ml_performance_metrics.rs - PostgreSQL persistence
• Hourly updates for accuracy, Sharpe, PnL

---

📈 Metric Cardinality

• Predictions: 5 models × 10 symbols × 3 actions = 150 series
• Orders: 5 models × 10 symbols = 50 series
• Performance: 5 models × 6 metrics = 30 series
• Ensemble: 10 symbols × 2 metrics = 20 series

Total: ~250 time series

---

🚀 Next Steps

Agent 20: Create Grafana dashboard with panels for:

1. Model performance (Sharpe, win rate, accuracy)
2. Prediction activity (rate, confidence, staleness)
3. Order execution (fill rate, rejections)
4. Ensemble behavior (agreement, disagreement events)
5. Inference performance (latency P99, distribution)
6. Risk metrics (drawdown, PnL)

---

End of Quick Reference