foxhunt/docs/archive/ml_models/ENSEMBLE_METRICS_QUICK_REFERENCE.md

Ensemble Metrics Quick Reference

Status: ✅ Production Ready Updated: 2025-10-14

---

🚀 Quick Start (3 Commands)

# 1. Run test harness (verify metrics collection)
cargo run -p trading_service --example test_ensemble_metrics

# 2. Check Prometheus metrics endpoint
curl http://localhost:9092/metrics | grep ensemble_

# 3. Import Grafana dashboard
# Open http://localhost:3000 → Import → Upload ensemble_ml_production.json

---

📊 10 Metrics at a Glance

#	Metric	Type	Purpose	Alert Threshold
1	ensemble_aggregation_latency_microseconds	Histogram	Aggregation time	P99 > 50μs
2	ensemble_confidence_score	Gauge	Prediction confidence	< 0.6 (low)
3	ensemble_disagreement_rate	Gauge	Model disagreement	> 0.5 (high)
4	ensemble_predictions_total	Counter	Prediction count	-
5	ensemble_model_weight	Gauge	Model contribution	Sum ≠ 1.0
6	ensemble_high_disagreement_total	Counter	High disagreement events	Rate spike
7	ensemble_model_pnl_contribution_dollars	Histogram	P&L attribution	Negative trend
8	checkpoint_swaps_total	Counter	Checkpoint updates	Rollback > 10%
9	ab_test_assignments_total	Counter	A/B test assignments	Imbalance > 55/45
10	ab_test_metric_difference	Gauge	A/B test lift	-

---

💻 Code Usage Examples

Automatic Recording (Default)

// Metrics auto-recorded on every prediction
let decision = coordinator.predict(&features).await?;
// ✅ Metrics 1-4, 6 recorded automatically

Manual P&L Recording

// Record P&L attribution per model
coordinator.record_model_pnl("DQN", "ES.FUT", 125.50);
coordinator.record_model_pnl("PPO", "ES.FUT", 110.30);

Checkpoint Swap Events

use trading_service::ensemble_metrics::{CheckpointSwapEvent, CheckpointSwapStatus};

let swap = CheckpointSwapEvent {
    model_id: "DQN".to_string(),
    status: CheckpointSwapStatus::Success,
};
swap.record();

A/B Test Recording

use trading_service::ensemble_metrics::{ABTestAssignment, ABTestGroup};

let assignment = ABTestAssignment {
    test_id: "test-001".to_string(),
    group: ABTestGroup::Treatment,
};
assignment.record();

---

🔍 Key PromQL Queries

Monitor Disagreement Spikes

rate(ensemble_high_disagreement_total{threshold="0.5"}[5m]) > 10

P99 Latency Monitoring

histogram_quantile(0.99, rate(ensemble_aggregation_latency_microseconds_bucket[5m])) > 50

Model P&L Ranking

topk(3, sum by (model_id) (ensemble_model_pnl_contribution_dollars_sum))

Checkpoint Rollback Rate

sum(checkpoint_swaps_total{status="rollback"}) / sum(checkpoint_swaps_total) > 0.1

A/B Test Sharpe Lift

ab_test_metric_difference{metric="sharpe_ratio"} > 0.2

---

📈 Grafana Dashboard Panels

1. Confidence & Disagreement: Line chart (0-1 scale, alert at 0.5)
2. Model Weights: Stacked area (shows dominance over time)
3. P&L Attribution: Color-coded table (red/yellow/green)
4. Aggregation Latency: P50/P95/P99 lines (alert at 50μs)
5. High Disagreement: Bar chart (regime shift detection)
6. Checkpoint Health: Success vs rollback (alert at 10%)
7. A/B Test Lift: Gauge (Sharpe ratio improvement)
8. A/B Assignments: Pie chart (balance verification)

---

⚙️ Configuration

Prometheus Scraping

# prometheus.yml
scrape_configs:
  - job_name: 'trading_service'
    static_configs:
      - targets: ['localhost:9092']
    scrape_interval: 5s

Alert Rules

# alerts.yml
groups:
  - name: ensemble_alerts
    interval: 30s
    rules:
      - alert: HighDisagreement
        expr: ensemble_disagreement_rate > 0.7
        for: 5m
        annotations:
          summary: "High model disagreement detected"

      - alert: HighCheckpointRollbackRate
        expr: sum(checkpoint_swaps_total{status="rollback"}) / sum(checkpoint_swaps_total) > 0.1
        for: 10m
        annotations:
          summary: "Checkpoint rollback rate exceeds 10%"

---

🧪 Testing Checklist

• Run test harness: cargo run -p trading_service --example test_ensemble_metrics
• Verify 1000 predictions complete
• Check metrics endpoint: curl http://localhost:9092/metrics | grep ensemble_
• Import Grafana dashboard
• Verify all 8 panels render
• Test variable filters (symbol, aggregation_method, test_id)
• Confirm 5-second auto-refresh
• Validate alert thresholds

---

🚨 Alert Thresholds

Metric	Warning	Critical	Action
Disagreement Rate	> 0.5	> 0.7	Reduce position size
P99 Latency	> 25μs	> 50μs	Investigate bottleneck
Rollback Rate	> 5%	> 10%	Review checkpoint quality
Confidence	< 0.7	< 0.6	Switch to single model

---

📁 File Locations

File	Purpose
services/trading_service/src/ensemble_metrics.rs	Metrics definitions + helpers
services/trading_service/src/ensemble_coordinator.rs	Integration point
monitoring/grafana/ensemble_ml_production.json	Dashboard JSON
services/trading_service/examples/test_ensemble_metrics.rs	Test harness

---

🔗 Related Documentation

• Full Status: ENSEMBLE_METRICS_IMPLEMENTATION_STATUS.md
• Strategy: ENSEMBLE_PRODUCTION_DEPLOYMENT_STRATEGY.md
• System Architecture: CLAUDE.md

---

Last Updated: 2025-10-14 Status: ✅ Production Ready