foxhunt/ENSEMBLE_METRICS_QUICK_REFERENCE.md

# Ensemble Metrics Quick Reference

**Status**: ✅ Production Ready
**Updated**: 2025-10-14

---

## 🚀 Quick Start (3 Commands)

```bash
# 1. Run test harness (verify metrics collection)
cargo run -p trading_service --example test_ensemble_metrics

# 2. Check Prometheus metrics endpoint
curl http://localhost:9092/metrics | grep ensemble_

# 3. Import Grafana dashboard
# Open http://localhost:3000 → Import → Upload ensemble_ml_production.json
```

---

## 📊 10 Metrics at a Glance

| # | Metric | Type | Purpose | Alert Threshold |
|---|--------|------|---------|-----------------|
| 1 | `ensemble_aggregation_latency_microseconds` | Histogram | Aggregation time | P99 > 50μs |
| 2 | `ensemble_confidence_score` | Gauge | Prediction confidence | < 0.6 (low) |
| 3 | `ensemble_disagreement_rate` | Gauge | Model disagreement | > 0.5 (high) |
| 4 | `ensemble_predictions_total` | Counter | Prediction count | - |
| 5 | `ensemble_model_weight` | Gauge | Model contribution | Sum ≠ 1.0 |
| 6 | `ensemble_high_disagreement_total` | Counter | High disagreement events | Rate spike |
| 7 | `ensemble_model_pnl_contribution_dollars` | Histogram | P&L attribution | Negative trend |
| 8 | `checkpoint_swaps_total` | Counter | Checkpoint updates | Rollback > 10% |
| 9 | `ab_test_assignments_total` | Counter | A/B test assignments | Imbalance > 55/45 |
| 10 | `ab_test_metric_difference` | Gauge | A/B test lift | - |

---

## 💻 Code Usage Examples

### Automatic Recording (Default)
```rust
// Metrics auto-recorded on every prediction
let decision = coordinator.predict(&features).await?;
// ✅ Metrics 1-4, 6 recorded automatically
```

### Manual P&L Recording
```rust
// Record P&L attribution per model
coordinator.record_model_pnl("DQN", "ES.FUT", 125.50);
coordinator.record_model_pnl("PPO", "ES.FUT", 110.30);
```

### Checkpoint Swap Events
```rust
use trading_service::ensemble_metrics::{CheckpointSwapEvent, CheckpointSwapStatus};

let swap = CheckpointSwapEvent {
    model_id: "DQN".to_string(),
    status: CheckpointSwapStatus::Success,
};
swap.record();
```

### A/B Test Recording
```rust
use trading_service::ensemble_metrics::{ABTestAssignment, ABTestGroup};

let assignment = ABTestAssignment {
    test_id: "test-001".to_string(),
    group: ABTestGroup::Treatment,
};
assignment.record();
```

---

## 🔍 Key PromQL Queries

### Monitor Disagreement Spikes
```promql
rate(ensemble_high_disagreement_total{threshold="0.5"}[5m]) > 10
```

### P99 Latency Monitoring
```promql
histogram_quantile(0.99, rate(ensemble_aggregation_latency_microseconds_bucket[5m])) > 50
```

### Model P&L Ranking
```promql
topk(3, sum by (model_id) (ensemble_model_pnl_contribution_dollars_sum))
```

### Checkpoint Rollback Rate
```promql
sum(checkpoint_swaps_total{status="rollback"}) / sum(checkpoint_swaps_total) > 0.1
```

### A/B Test Sharpe Lift
```promql
ab_test_metric_difference{metric="sharpe_ratio"} > 0.2
```

---

## 📈 Grafana Dashboard Panels

1. **Confidence & Disagreement**: Line chart (0-1 scale, alert at 0.5)
2. **Model Weights**: Stacked area (shows dominance over time)
3. **P&L Attribution**: Color-coded table (red/yellow/green)
4. **Aggregation Latency**: P50/P95/P99 lines (alert at 50μs)
5. **High Disagreement**: Bar chart (regime shift detection)
6. **Checkpoint Health**: Success vs rollback (alert at 10%)
7. **A/B Test Lift**: Gauge (Sharpe ratio improvement)
8. **A/B Assignments**: Pie chart (balance verification)

---

## ⚙️ Configuration

### Prometheus Scraping
```yaml
# prometheus.yml
scrape_configs:
  - job_name: 'trading_service'
    static_configs:
      - targets: ['localhost:9092']
    scrape_interval: 5s
```

### Alert Rules
```yaml
# alerts.yml
groups:
  - name: ensemble_alerts
    interval: 30s
    rules:
      - alert: HighDisagreement
        expr: ensemble_disagreement_rate > 0.7
        for: 5m
        annotations:
          summary: "High model disagreement detected"

      - alert: HighCheckpointRollbackRate
        expr: sum(checkpoint_swaps_total{status="rollback"}) / sum(checkpoint_swaps_total) > 0.1
        for: 10m
        annotations:
          summary: "Checkpoint rollback rate exceeds 10%"
```

---

## 🧪 Testing Checklist

- [ ] Run test harness: `cargo run -p trading_service --example test_ensemble_metrics`
- [ ] Verify 1000 predictions complete
- [ ] Check metrics endpoint: `curl http://localhost:9092/metrics | grep ensemble_`
- [ ] Import Grafana dashboard
- [ ] Verify all 8 panels render
- [ ] Test variable filters (symbol, aggregation_method, test_id)
- [ ] Confirm 5-second auto-refresh
- [ ] Validate alert thresholds

---

## 🚨 Alert Thresholds

| Metric | Warning | Critical | Action |
|--------|---------|----------|--------|
| Disagreement Rate | > 0.5 | > 0.7 | Reduce position size |
| P99 Latency | > 25μs | > 50μs | Investigate bottleneck |
| Rollback Rate | > 5% | > 10% | Review checkpoint quality |
| Confidence | < 0.7 | < 0.6 | Switch to single model |

---

## 📁 File Locations

| File | Purpose |
|------|---------|
| `services/trading_service/src/ensemble_metrics.rs` | Metrics definitions + helpers |
| `services/trading_service/src/ensemble_coordinator.rs` | Integration point |
| `monitoring/grafana/ensemble_ml_production.json` | Dashboard JSON |
| `services/trading_service/examples/test_ensemble_metrics.rs` | Test harness |

---

## 🔗 Related Documentation

- **Full Status**: `ENSEMBLE_METRICS_IMPLEMENTATION_STATUS.md`
- **Strategy**: `ENSEMBLE_PRODUCTION_DEPLOYMENT_STRATEGY.md`
- **System Architecture**: `CLAUDE.md`

---

**Last Updated**: 2025-10-14
**Status**: ✅ Production Ready