foxhunt/docs/ENSEMBLE_AUDIT_QUICKSTART.md

Ensemble Audit Logging Quickstart Guide

Purpose: Get started with ensemble prediction audit logging in 5 minutes

---

Quick Setup

1. Apply Database Migration (1 minute)

# Run migration to create audit tables
cargo sqlx migrate run

# Verify tables created
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "\dt ensemble*"

Expected Output:

              List of relations
 Schema |           Name            | Type  |  Owner
--------+---------------------------+-------+---------
 public | ensemble_predictions      | table | foxhunt
 public | model_performance_attribution | table | foxhunt
 public | ab_test_experiments       | table | foxhunt

2. Log Your First Prediction (1 minute)

use ensemble_audit_logger::{EnsembleAuditLogger, EnsemblePredictionAudit};

// Initialize logger (in TradingServiceState)
let pool = PgPool::connect(&db_url).await?;
let logger = EnsembleAuditLogger::new(pool);

// Get ensemble prediction
let decision = ensemble_coordinator.predict(&features).await?;

// Log to PostgreSQL
let audit = EnsemblePredictionAudit::from_decision(&decision, "ES.FUT".to_string())
    .with_latency(42, 8); // inference_us=42, aggregation_us=8

let prediction_id = logger.log_prediction(audit).await?;
println!("Logged prediction: {}", prediction_id);

3. Query Your Data (1 minute)

-- See latest predictions
SELECT
    timestamp,
    symbol,
    ensemble_action,
    ensemble_confidence,
    disagreement_rate,
    dqn_vote,
    ppo_vote,
    tft_vote
FROM ensemble_predictions
ORDER BY timestamp DESC
LIMIT 10;

---

Common Use Cases

Use Case 1: Monitor Model Performance (Last 24 Hours)

Query:

SELECT * FROM get_top_models_24h(NULL, 5);

Sample Output:

 model_id | total_predictions | accuracy |  sharpe_ratio  | total_pnl | avg_weight
----------+-------------------+----------+---------------+-----------+------------
 PPO      |              5432 |     0.58 |          2.14 |   542300  |       0.34
 DQN      |              5389 |     0.56 |          1.92 |   498200  |       0.33
 TFT      |              5401 |     0.55 |          1.87 |   475100  |       0.33
 MAMBA2   |              5420 |     0.54 |          1.75 |   445800  |       0.33

Insight: PPO is the top performer with 2.14 Sharpe ratio (14% better than DQN)

---

Use Case 2: Detect Regime Shifts (High Disagreement)

Query:

SELECT * FROM get_high_disagreement_events_24h('ES.FUT', 0.7, 20);

Sample Output:

      timestamp       | symbol  | ensemble_action | disagreement_rate | dqn_vote | ppo_vote | tft_vote
----------------------+---------+-----------------+-------------------+----------+----------+----------
 2025-10-14 14:32:15  | ES.FUT  | BUY            |              0.75 | BUY      | SELL     | HOLD
 2025-10-14 14:31:02  | ES.FUT  | HOLD           |              0.75 | SELL     | BUY      | HOLD
 2025-10-14 14:29:18  | ES.FUT  | SELL           |              0.75 | SELL     | BUY      | BUY

Insight: High disagreement at 14:30-14:32 suggests market regime shift (possible news event)

---

Use Case 3: Compare Ensemble vs Single Model (Last 30 Days)

Query (simplified from Query 7):

WITH ensemble_perf AS (
    SELECT
        COUNT(*) as predictions,
        AVG(pnl::DOUBLE PRECISION) / STDDEV(pnl::DOUBLE PRECISION) as sharpe
    FROM ensemble_predictions
    WHERE timestamp >= NOW() - INTERVAL '30 days' AND pnl IS NOT NULL
),
dqn_perf AS (
    SELECT AVG(sharpe_ratio) as sharpe
    FROM model_performance_attribution
    WHERE model_id = 'DQN'
      AND timestamp >= NOW() - INTERVAL '30 days'
      AND window_hours = 24
)
SELECT
    ep.predictions,
    ep.sharpe as ensemble_sharpe,
    dp.sharpe as dqn_sharpe,
    (ep.sharpe - dp.sharpe) / dp.sharpe * 100 as ensemble_lift_percent
FROM ensemble_perf ep, dqn_perf dp;

Sample Output:

 predictions | ensemble_sharpe | dqn_sharpe | ensemble_lift_percent
-------------+-----------------+------------+----------------------
     125,432 |           2.14  |      1.82  |                17.6%

Insight: Ensemble achieves 17.6% Sharpe lift over single DQN model

---

Use Case 4: Analyze Failed Predictions

Query:

SELECT
    timestamp,
    symbol,
    ensemble_action,
    ensemble_confidence,
    disagreement_rate,
    pnl / 100.0 as pnl_dollars,
    dqn_vote,
    ppo_vote,
    tft_vote,
    feature_snapshot->'technical_indicators'->>'rsi' as rsi
FROM ensemble_predictions
WHERE timestamp >= NOW() - INTERVAL '7 days'
  AND pnl IS NOT NULL
  AND pnl < -500 -- Lost >$5
ORDER BY pnl ASC
LIMIT 20;

Sample Output:

      timestamp       | symbol | action | confidence | disagreement | pnl_dollars | dqn  | ppo  | tft  |  rsi
----------------------+--------+--------+------------+--------------+-------------+------+------+------+------
 2025-10-14 10:15:32  | ES.FUT | BUY    |       0.82 |         0.15 |      -24.50 | BUY  | BUY  | BUY  | 68.5
 2025-10-14 09:42:18  | NQ.FUT | SELL   |       0.79 |         0.25 |      -18.20 | SELL | HOLD | SELL | 42.3

Insight: All models agreed on BUY at 10:15 (low disagreement 0.15), but RSI was overbought (68.5) → false signal

---

Use Case 5: Track A/B Test Progress

Query:

WITH ab_metrics AS (
    SELECT
        ab_group,
        COUNT(*) as predictions,
        AVG(ensemble_confidence) as avg_confidence,
        SUM(pnl) / 100.0 as total_pnl_dollars,
        COUNT(CASE WHEN pnl > 0 THEN 1 END)::FLOAT / NULLIF(COUNT(CASE WHEN pnl IS NOT NULL THEN 1 END), 0) as win_rate
    FROM ensemble_predictions
    WHERE ab_test_id = 'your-test-id-here' -- Replace with actual test_id
    GROUP BY ab_group
)
SELECT
    control.predictions as control_n,
    treatment.predictions as treatment_n,
    control.total_pnl_dollars as control_pnl,
    treatment.total_pnl_dollars as treatment_pnl,
    (treatment.total_pnl_dollars - control.total_pnl_dollars) as pnl_diff,
    control.win_rate as control_win_rate,
    treatment.win_rate as treatment_win_rate
FROM ab_metrics control, ab_metrics treatment
WHERE control.ab_group = 'control' AND treatment.ab_group = 'treatment';

Sample Output:

 control_n | treatment_n | control_pnl | treatment_pnl | pnl_diff | control_win_rate | treatment_win_rate
-----------+-------------+-------------+---------------+----------+------------------+--------------------
      5432 |        5389 |    12,450   |      15,200   |  2,750   |            0.543 |              0.581

Insight: Treatment (ensemble) outperforms control (DQN) by $2,750 (+22%) with 58.1% vs 54.3% win rate

---

Use Case 6: Monitor Inference Latency

Query:

SELECT
    symbol,
    COUNT(*) as predictions,
    PERCENTILE_CONT(0.50) WITHIN GROUP (ORDER BY inference_latency_us) as p50_us,
    PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY inference_latency_us) as p95_us,
    PERCENTILE_CONT(0.99) WITHIN GROUP (ORDER BY inference_latency_us) as p99_us,
    MAX(inference_latency_us) as max_us
FROM ensemble_predictions
WHERE timestamp >= NOW() - INTERVAL '1 hour'
  AND inference_latency_us IS NOT NULL
GROUP BY symbol
ORDER BY p99_us DESC;

Sample Output:

 symbol  | predictions | p50_us | p95_us | p99_us | max_us
---------+-------------+--------+--------+--------+--------
 ES.FUT  |        2154 |     38 |     45 |     52 |     68
 NQ.FUT  |        2089 |     40 |     47 |     54 |     71
 ZN.FUT  |        1832 |     37 |     44 |     50 |     65

Insight: All symbols meet <50μs P99 latency target (52μs worst case for ES.FUT)

---

Batch Operations

Batch Insert (High Throughput)

// Create 100 predictions
let mut audits = Vec::new();
for i in 0..100 {
    let decision = ensemble_coordinator.predict(&features).await?;
    let audit = EnsemblePredictionAudit::from_decision(&decision, "ES.FUT".to_string());
    audits.push(audit);
}

// Batch insert (transaction-wrapped)
let ids = logger.log_predictions_batch(audits).await?;
println!("Logged {} predictions in batch", ids.len());

Performance: >1,000 predictions/sec (100 predictions in <100ms)

---

Grafana Dashboard Setup

Panel 1: Ensemble Confidence Over Time

SELECT
    time_bucket('5 minutes', timestamp) as time,
    AVG(ensemble_confidence) as avg_confidence,
    AVG(disagreement_rate) as avg_disagreement
FROM ensemble_predictions
WHERE $__timeFilter(timestamp)
  AND symbol = 'ES.FUT'
GROUP BY time
ORDER BY time;

Panel 2: Model Performance Comparison

SELECT
    time_bucket('1 hour', timestamp) as time,
    model_id,
    AVG(sharpe_ratio) as avg_sharpe
FROM model_performance_attribution
WHERE $__timeFilter(timestamp)
  AND window_hours = 24
GROUP BY time, model_id
ORDER BY time;

Panel 3: High Disagreement Rate

SELECT
    time_bucket('5 minutes', timestamp) as time,
    COUNT(*) as high_disagreement_count
FROM ensemble_predictions
WHERE $__timeFilter(timestamp)
  AND disagreement_rate > 0.5
GROUP BY time
ORDER BY time;

---

Troubleshooting

Issue: Slow Queries

Check Index Usage:

SELECT
    schemaname,
    tablename,
    indexname,
    idx_scan as scans,
    idx_tup_read as tuples_read
FROM pg_stat_user_indexes
WHERE tablename = 'ensemble_predictions'
ORDER BY idx_scan DESC;

Expected: All indexes should show scans > 0 after 24 hours

---

Issue: High Storage Usage

Check Compression:

SELECT
    hypertable_name,
    pg_size_pretty(total_bytes) as total_size,
    pg_size_pretty(compressed_total_bytes) as compressed_size,
    ROUND(100.0 * (1 - compressed_total_bytes::FLOAT / NULLIF(total_bytes, 0)), 2) as compression_percent
FROM timescaledb_information.hypertable
WHERE hypertable_name = 'ensemble_predictions';

Expected: ~60-70% compression ratio after 7 days

---

Issue: Missing Model Votes

Data Quality Check:

SELECT
    COUNT(*) as total,
    COUNT(dqn_signal) as dqn_present,
    COUNT(ppo_signal) as ppo_present,
    COUNT(tft_signal) as tft_present,
    COUNT(*) - COUNT(dqn_signal) as dqn_missing
FROM ensemble_predictions
WHERE timestamp >= NOW() - INTERVAL '24 hours';

Expected: All models should be present (0 missing)

---

Integration Tests

Run all ensemble audit tests:

cargo test --test ensemble_audit_tests -- --nocapture

Expected Output:

running 11 tests
test test_ensemble_audit_logger_initialization ... ok
test test_log_ensemble_prediction ... ok
test test_update_prediction_pnl ... ok
test test_model_performance_attribution ... ok
test test_ab_test_experiment_tracking ... ok
test test_batch_prediction_insert_performance ... ok
test test_analysis_query_performance ... ok
test test_timescaledb_hypertable_functionality ... ok
test test_feature_snapshot_jsonb ... ok
test test_continuous_aggregate_views ... ok
test test_utility_functions ... ok

test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

---

Next Steps

1. ✅ Apply Migration: cargo sqlx migrate run
2. ✅ Test Logging: Insert first prediction
3. ✅ Query Data: Run sample queries
4. ⏳ Create Dashboards: Grafana panels for real-time monitoring
5. ⏳ Start A/B Test: Compare ensemble vs single model
6. ⏳ Monitor Performance: Track latency, throughput, storage

---

Documentation

• Full Implementation: ENSEMBLE_AUDIT_IMPLEMENTATION_STATUS.md
• All Queries (26): docs/ENSEMBLE_AUDIT_QUERIES.sql
• Migration: migrations/022_create_ensemble_tables.sql
• Audit Logger: services/trading_service/src/ensemble_audit_logger.rs
• Tests: services/trading_service/tests/ensemble_audit_tests.rs

---

Quick Start Complete! You're now ready to track ensemble predictions, attribute P&L, and run A/B tests.