foxhunt/docs/ENSEMBLE_AUDIT_QUICKSTART.md

# Ensemble Audit Logging Quickstart Guide

**Purpose**: Get started with ensemble prediction audit logging in 5 minutes

---

## Quick Setup

### 1. Apply Database Migration (1 minute)

```bash
# Run migration to create audit tables
cargo sqlx migrate run

# Verify tables created
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "\dt ensemble*"
```

**Expected Output**:
```
              List of relations
 Schema |           Name            | Type  |  Owner
--------+---------------------------+-------+---------
 public | ensemble_predictions      | table | foxhunt
 public | model_performance_attribution | table | foxhunt
 public | ab_test_experiments       | table | foxhunt
```

### 2. Log Your First Prediction (1 minute)

```rust
use ensemble_audit_logger::{EnsembleAuditLogger, EnsemblePredictionAudit};

// Initialize logger (in TradingServiceState)
let pool = PgPool::connect(&db_url).await?;
let logger = EnsembleAuditLogger::new(pool);

// Get ensemble prediction
let decision = ensemble_coordinator.predict(&features).await?;

// Log to PostgreSQL
let audit = EnsemblePredictionAudit::from_decision(&decision, "ES.FUT".to_string())
    .with_latency(42, 8); // inference_us=42, aggregation_us=8

let prediction_id = logger.log_prediction(audit).await?;
println!("Logged prediction: {}", prediction_id);
```

### 3. Query Your Data (1 minute)

```sql
-- See latest predictions
SELECT
    timestamp,
    symbol,
    ensemble_action,
    ensemble_confidence,
    disagreement_rate,
    dqn_vote,
    ppo_vote,
    tft_vote
FROM ensemble_predictions
ORDER BY timestamp DESC
LIMIT 10;
```

---

## Common Use Cases

### Use Case 1: Monitor Model Performance (Last 24 Hours)

**Query**:
```sql
SELECT * FROM get_top_models_24h(NULL, 5);
```

**Sample Output**:
```
 model_id | total_predictions | accuracy |  sharpe_ratio  | total_pnl | avg_weight
----------+-------------------+----------+---------------+-----------+------------
 PPO      |              5432 |     0.58 |          2.14 |   542300  |       0.34
 DQN      |              5389 |     0.56 |          1.92 |   498200  |       0.33
 TFT      |              5401 |     0.55 |          1.87 |   475100  |       0.33
 MAMBA2   |              5420 |     0.54 |          1.75 |   445800  |       0.33
```

**Insight**: PPO is the top performer with 2.14 Sharpe ratio (14% better than DQN)

---

### Use Case 2: Detect Regime Shifts (High Disagreement)

**Query**:
```sql
SELECT * FROM get_high_disagreement_events_24h('ES.FUT', 0.7, 20);
```

**Sample Output**:
```
      timestamp       | symbol  | ensemble_action | disagreement_rate | dqn_vote | ppo_vote | tft_vote
----------------------+---------+-----------------+-------------------+----------+----------+----------
 2025-10-14 14:32:15  | ES.FUT  | BUY            |              0.75 | BUY      | SELL     | HOLD
 2025-10-14 14:31:02  | ES.FUT  | HOLD           |              0.75 | SELL     | BUY      | HOLD
 2025-10-14 14:29:18  | ES.FUT  | SELL           |              0.75 | SELL     | BUY      | BUY
```

**Insight**: High disagreement at 14:30-14:32 suggests market regime shift (possible news event)

---

### Use Case 3: Compare Ensemble vs Single Model (Last 30 Days)

**Query** (simplified from Query 7):
```sql
WITH ensemble_perf AS (
    SELECT
        COUNT(*) as predictions,
        AVG(pnl::DOUBLE PRECISION) / STDDEV(pnl::DOUBLE PRECISION) as sharpe
    FROM ensemble_predictions
    WHERE timestamp >= NOW() - INTERVAL '30 days' AND pnl IS NOT NULL
),
dqn_perf AS (
    SELECT AVG(sharpe_ratio) as sharpe
    FROM model_performance_attribution
    WHERE model_id = 'DQN'
      AND timestamp >= NOW() - INTERVAL '30 days'
      AND window_hours = 24
)
SELECT
    ep.predictions,
    ep.sharpe as ensemble_sharpe,
    dp.sharpe as dqn_sharpe,
    (ep.sharpe - dp.sharpe) / dp.sharpe * 100 as ensemble_lift_percent
FROM ensemble_perf ep, dqn_perf dp;
```

**Sample Output**:
```
 predictions | ensemble_sharpe | dqn_sharpe | ensemble_lift_percent
-------------+-----------------+------------+----------------------
     125,432 |           2.14  |      1.82  |                17.6%
```

**Insight**: Ensemble achieves 17.6% Sharpe lift over single DQN model

---

### Use Case 4: Analyze Failed Predictions

**Query**:
```sql
SELECT
    timestamp,
    symbol,
    ensemble_action,
    ensemble_confidence,
    disagreement_rate,
    pnl / 100.0 as pnl_dollars,
    dqn_vote,
    ppo_vote,
    tft_vote,
    feature_snapshot->'technical_indicators'->>'rsi' as rsi
FROM ensemble_predictions
WHERE timestamp >= NOW() - INTERVAL '7 days'
  AND pnl IS NOT NULL
  AND pnl < -500 -- Lost >$5
ORDER BY pnl ASC
LIMIT 20;
```

**Sample Output**:
```
      timestamp       | symbol | action | confidence | disagreement | pnl_dollars | dqn  | ppo  | tft  |  rsi
----------------------+--------+--------+------------+--------------+-------------+------+------+------+------
 2025-10-14 10:15:32  | ES.FUT | BUY    |       0.82 |         0.15 |      -24.50 | BUY  | BUY  | BUY  | 68.5
 2025-10-14 09:42:18  | NQ.FUT | SELL   |       0.79 |         0.25 |      -18.20 | SELL | HOLD | SELL | 42.3
```

**Insight**: All models agreed on BUY at 10:15 (low disagreement 0.15), but RSI was overbought (68.5) → false signal

---

### Use Case 5: Track A/B Test Progress

**Query**:
```sql
WITH ab_metrics AS (
    SELECT
        ab_group,
        COUNT(*) as predictions,
        AVG(ensemble_confidence) as avg_confidence,
        SUM(pnl) / 100.0 as total_pnl_dollars,
        COUNT(CASE WHEN pnl > 0 THEN 1 END)::FLOAT / NULLIF(COUNT(CASE WHEN pnl IS NOT NULL THEN 1 END), 0) as win_rate
    FROM ensemble_predictions
    WHERE ab_test_id = 'your-test-id-here' -- Replace with actual test_id
    GROUP BY ab_group
)
SELECT
    control.predictions as control_n,
    treatment.predictions as treatment_n,
    control.total_pnl_dollars as control_pnl,
    treatment.total_pnl_dollars as treatment_pnl,
    (treatment.total_pnl_dollars - control.total_pnl_dollars) as pnl_diff,
    control.win_rate as control_win_rate,
    treatment.win_rate as treatment_win_rate
FROM ab_metrics control, ab_metrics treatment
WHERE control.ab_group = 'control' AND treatment.ab_group = 'treatment';
```

**Sample Output**:
```
 control_n | treatment_n | control_pnl | treatment_pnl | pnl_diff | control_win_rate | treatment_win_rate
-----------+-------------+-------------+---------------+----------+------------------+--------------------
      5432 |        5389 |    12,450   |      15,200   |  2,750   |            0.543 |              0.581
```

**Insight**: Treatment (ensemble) outperforms control (DQN) by $2,750 (+22%) with 58.1% vs 54.3% win rate

---

### Use Case 6: Monitor Inference Latency

**Query**:
```sql
SELECT
    symbol,
    COUNT(*) as predictions,
    PERCENTILE_CONT(0.50) WITHIN GROUP (ORDER BY inference_latency_us) as p50_us,
    PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY inference_latency_us) as p95_us,
    PERCENTILE_CONT(0.99) WITHIN GROUP (ORDER BY inference_latency_us) as p99_us,
    MAX(inference_latency_us) as max_us
FROM ensemble_predictions
WHERE timestamp >= NOW() - INTERVAL '1 hour'
  AND inference_latency_us IS NOT NULL
GROUP BY symbol
ORDER BY p99_us DESC;
```

**Sample Output**:
```
 symbol  | predictions | p50_us | p95_us | p99_us | max_us
---------+-------------+--------+--------+--------+--------
 ES.FUT  |        2154 |     38 |     45 |     52 |     68
 NQ.FUT  |        2089 |     40 |     47 |     54 |     71
 ZN.FUT  |        1832 |     37 |     44 |     50 |     65
```

**Insight**: All symbols meet <50μs P99 latency target (52μs worst case for ES.FUT)

---

## Batch Operations

### Batch Insert (High Throughput)

```rust
// Create 100 predictions
let mut audits = Vec::new();
for i in 0..100 {
    let decision = ensemble_coordinator.predict(&features).await?;
    let audit = EnsemblePredictionAudit::from_decision(&decision, "ES.FUT".to_string());
    audits.push(audit);
}

// Batch insert (transaction-wrapped)
let ids = logger.log_predictions_batch(audits).await?;
println!("Logged {} predictions in batch", ids.len());
```

**Performance**: >1,000 predictions/sec (100 predictions in <100ms)

---

## Grafana Dashboard Setup

### Panel 1: Ensemble Confidence Over Time

```sql
SELECT
    time_bucket('5 minutes', timestamp) as time,
    AVG(ensemble_confidence) as avg_confidence,
    AVG(disagreement_rate) as avg_disagreement
FROM ensemble_predictions
WHERE $__timeFilter(timestamp)
  AND symbol = 'ES.FUT'
GROUP BY time
ORDER BY time;
```

### Panel 2: Model Performance Comparison

```sql
SELECT
    time_bucket('1 hour', timestamp) as time,
    model_id,
    AVG(sharpe_ratio) as avg_sharpe
FROM model_performance_attribution
WHERE $__timeFilter(timestamp)
  AND window_hours = 24
GROUP BY time, model_id
ORDER BY time;
```

### Panel 3: High Disagreement Rate

```sql
SELECT
    time_bucket('5 minutes', timestamp) as time,
    COUNT(*) as high_disagreement_count
FROM ensemble_predictions
WHERE $__timeFilter(timestamp)
  AND disagreement_rate > 0.5
GROUP BY time
ORDER BY time;
```

---

## Troubleshooting

### Issue: Slow Queries

**Check Index Usage**:
```sql
SELECT
    schemaname,
    tablename,
    indexname,
    idx_scan as scans,
    idx_tup_read as tuples_read
FROM pg_stat_user_indexes
WHERE tablename = 'ensemble_predictions'
ORDER BY idx_scan DESC;
```

**Expected**: All indexes should show `scans > 0` after 24 hours

---

### Issue: High Storage Usage

**Check Compression**:
```sql
SELECT
    hypertable_name,
    pg_size_pretty(total_bytes) as total_size,
    pg_size_pretty(compressed_total_bytes) as compressed_size,
    ROUND(100.0 * (1 - compressed_total_bytes::FLOAT / NULLIF(total_bytes, 0)), 2) as compression_percent
FROM timescaledb_information.hypertable
WHERE hypertable_name = 'ensemble_predictions';
```

**Expected**: ~60-70% compression ratio after 7 days

---

### Issue: Missing Model Votes

**Data Quality Check**:
```sql
SELECT
    COUNT(*) as total,
    COUNT(dqn_signal) as dqn_present,
    COUNT(ppo_signal) as ppo_present,
    COUNT(tft_signal) as tft_present,
    COUNT(*) - COUNT(dqn_signal) as dqn_missing
FROM ensemble_predictions
WHERE timestamp >= NOW() - INTERVAL '24 hours';
```

**Expected**: All models should be present (0 missing)

---

## Integration Tests

Run all ensemble audit tests:
```bash
cargo test --test ensemble_audit_tests -- --nocapture
```

**Expected Output**:
```
running 11 tests
test test_ensemble_audit_logger_initialization ... ok
test test_log_ensemble_prediction ... ok
test test_update_prediction_pnl ... ok
test test_model_performance_attribution ... ok
test test_ab_test_experiment_tracking ... ok
test test_batch_prediction_insert_performance ... ok
test test_analysis_query_performance ... ok
test test_timescaledb_hypertable_functionality ... ok
test test_feature_snapshot_jsonb ... ok
test test_continuous_aggregate_views ... ok
test test_utility_functions ... ok

test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
```

---

## Next Steps

1. ✅ **Apply Migration**: `cargo sqlx migrate run`
2. ✅ **Test Logging**: Insert first prediction
3. ✅ **Query Data**: Run sample queries
4. ⏳ **Create Dashboards**: Grafana panels for real-time monitoring
5. ⏳ **Start A/B Test**: Compare ensemble vs single model
6. ⏳ **Monitor Performance**: Track latency, throughput, storage

---

## Documentation

- **Full Implementation**: `ENSEMBLE_AUDIT_IMPLEMENTATION_STATUS.md`
- **All Queries (26)**: `docs/ENSEMBLE_AUDIT_QUERIES.sql`
- **Migration**: `migrations/022_create_ensemble_tables.sql`
- **Audit Logger**: `services/trading_service/src/ensemble_audit_logger.rs`
- **Tests**: `services/trading_service/tests/ensemble_audit_tests.rs`

---

**Quick Start Complete!** You're now ready to track ensemble predictions, attribute P&L, and run A/B tests.