foxhunt/WAVE_13_2_AGENT_14_ML_PREDICTIONS_REPORT.md

Wave 13.2 Agent 14: ML Predictions Database Migration

Mission: Create database migration for ML predictions storage Status: ✅ COMPLETE - Migration already exists and validated Date: 2025-10-16

---

Executive Summary

The ML predictions database infrastructure already exists via migration 031 and is fully operational. The table schema, indexes, materialized view, and refresh function are all in place and tested.

Key Finding: Migration 043 was redundant - removed to avoid conflicts.

---

Database Schema

Table: ml_predictions

CREATE TABLE ml_predictions (
    id SERIAL PRIMARY KEY,
    model_name VARCHAR(50) NOT NULL,
    features JSONB NOT NULL,
    predicted_action SMALLINT NOT NULL, -- 0=Buy, 1=Sell, 2=Hold
    confidence REAL NOT NULL,
    symbol VARCHAR(20) NOT NULL,
    prediction_timestamp TIMESTAMPTZ NOT NULL DEFAULT NOW(),

    -- Outcome tracking (filled later)
    actual_action SMALLINT,
    pnl DECIMAL(15, 2),
    outcome_recorded_at TIMESTAMPTZ,

    -- Constraints
    CONSTRAINT ml_predictions_action_check CHECK (predicted_action BETWEEN 0 AND 2),
    CONSTRAINT ml_predictions_confidence_check CHECK (confidence BETWEEN 0.0 AND 1.0)
);

Schema Validation:

• ✅ Table exists
• ✅ 5 indexes created (primary key + 4 performance indexes)
• ✅ 3 constraints (primary key + 2 check constraints)
• ✅ JSONB features storage for flexible feature vectors
• ✅ Outcome tracking fields for post-prediction analysis

Materialized View: ml_model_performance

CREATE MATERIALIZED VIEW ml_model_performance AS
SELECT
    model_name,
    COUNT(*) as total_predictions,
    COUNT(actual_action) as predictions_with_outcomes,
    SUM(CASE WHEN predicted_action = actual_action THEN 1 ELSE 0 END) as correct_predictions,
    CASE
        WHEN COUNT(actual_action) > 0 THEN
            SUM(CASE WHEN predicted_action = actual_action THEN 1 ELSE 0 END)::FLOAT / COUNT(actual_action)
        ELSE 0.0
    END as accuracy,
    AVG(pnl) as avg_pnl,
    STDDEV(pnl) as stddev_pnl,
    CASE
        WHEN STDDEV(pnl) > 0 THEN
            AVG(pnl) / STDDEV(pnl) * SQRT(252)
        ELSE 0.0
    END as sharpe_ratio -- Annualized Sharpe (252 trading days)
FROM ml_predictions
WHERE outcome_recorded_at IS NOT NULL
GROUP BY model_name;

Performance Metrics:

• ✅ Total predictions count
• ✅ Accuracy calculation (correct predictions / total)
• ✅ Average P&L
• ✅ Standard deviation of P&L
• ✅ Sharpe Ratio (annualized, 252 trading days)
• ✅ Unique index on model_name for fast lookups

Refresh Function

CREATE OR REPLACE FUNCTION refresh_ml_model_performance()
RETURNS void AS $$
BEGIN
    REFRESH MATERIALIZED VIEW CONCURRENTLY ml_model_performance;
END;
$$ LANGUAGE plpgsql;

Usage: Call hourly via cron or application scheduler to update performance metrics.

---

Validation Results

Test Suite Execution

-- 1. Table exists and is empty
✅ ml_predictions table: 0 rows (fresh install)

-- 2. Indexes verified
✅ 5 indexes:
   - ml_predictions_pkey (PRIMARY KEY)
   - idx_ml_predictions_model (model_name)
   - idx_ml_predictions_symbol (symbol)
   - idx_ml_predictions_timestamp (prediction_timestamp)
   - idx_ml_predictions_outcome (outcome_recorded_at WHERE NOT NULL)

-- 3. Materialized view exists
✅ ml_model_performance view: 0 rows (no predictions yet)

-- 4. Refresh function works
✅ refresh_ml_model_performance() executed successfully

-- 5. Constraints enforced
✅ 3 constraints:
   - ml_predictions_pkey (PRIMARY KEY)
   - ml_predictions_action_check (predicted_action BETWEEN 0 AND 2)
   - ml_predictions_confidence_check (confidence BETWEEN 0.0 AND 1.0)

Sample Insert Test

INSERT INTO ml_predictions (
    model_name,
    features,
    predicted_action,
    confidence,
    symbol
) VALUES (
    'MAMBA2',
    '{"rsi": 65.3, "macd": 0.12, "volume": 15000}'::jsonb,
    0, -- BUY
    0.87,
    'ES.FUT'
);

-- Result: ✅ Insert successful
-- Verification: ✅ Data retrieved correctly
-- Cleanup: ✅ DELETE successful

---

Schema Design Decisions

1. Action Encoding

Choice: SMALLINT (0=Buy, 1=Sell, 2=Hold) Rationale:

• Compact storage (2 bytes vs VARCHAR)
• Fast comparison in queries
• Direct mapping to ML model outputs

2. Features Storage

Choice: JSONB Rationale:

• Flexible schema (models evolve over time)
• Indexable with GIN indexes if needed
• Efficient binary storage format
• Native PostgreSQL operators for queries

3. Outcome Tracking

Design: Separate columns (actual_action, pnl, outcome_recorded_at) Rationale:

• Predictions insert immediately
• Outcomes fill in later (after trade execution)
• outcome_recorded_at NULL check differentiates pending vs closed predictions

4. Performance Metrics

Materialized View vs Regular View:

• ✅ Materialized: Pre-computed aggregates (faster queries)
• ✅ CONCURRENTLY: Refresh without locking reads
• ✅ Unique index: Fast model-specific lookups
• ⚠️ Trade-off: Slightly stale data (refresh interval)

---

Integration Points

Agent 11: ML Order Submission

The ML order submission service (Agent 11) will use this schema to log predictions:

// Pseudo-code example
async fn log_ml_prediction(
    pool: &PgPool,
    model: &str,
    features: &serde_json::Value,
    action: i16,
    confidence: f32,
    symbol: &str,
) -> Result<i32> {
    let prediction_id = sqlx::query_scalar!(
        r#"
        INSERT INTO ml_predictions (model_name, features, predicted_action, confidence, symbol)
        VALUES ($1, $2, $3, $4, $5)
        RETURNING id
        "#,
        model,
        features,
        action,
        confidence,
        symbol
    )
    .fetch_one(pool)
    .await?;

    Ok(prediction_id)
}

Agent 12: Outcome Recording

Post-trade execution service will update predictions with outcomes:

// Pseudo-code example
async fn record_outcome(
    pool: &PgPool,
    prediction_id: i32,
    actual_action: i16,
    pnl: Decimal,
) -> Result<()> {
    sqlx::query!(
        r#"
        UPDATE ml_predictions
        SET actual_action = $1,
            pnl = $2,
            outcome_recorded_at = NOW()
        WHERE id = $3
        "#,
        actual_action,
        pnl,
        prediction_id
    )
    .execute(pool)
    .await?;

    Ok(())
}

---

Performance Considerations

Index Strategy

1. Model-based queries (idx_ml_predictions_model):

   SELECT * FROM ml_predictions WHERE model_name = 'MAMBA2';
   

2. Symbol-based queries (idx_ml_predictions_symbol):

   SELECT * FROM ml_predictions WHERE symbol = 'ES.FUT';
   

3. Time-range queries (idx_ml_predictions_timestamp):

   SELECT * FROM ml_predictions
   WHERE prediction_timestamp > NOW() - INTERVAL '7 days';
   

4. Outcome tracking (idx_ml_predictions_outcome):

   • Partial index (only rows with outcomes)
   • Efficient for performance metric updates

Query Performance Expectations

Query Type	Expected Latency	Index Used
Insert prediction	<5ms	Primary key
Update outcome	<10ms	Primary key
Get model predictions	<50ms	idx_ml_predictions_model
Get symbol predictions	<50ms	idx_ml_predictions_symbol
Time-range scan	<100ms	idx_ml_predictions_timestamp
Refresh materialized view	100-500ms	All indexes

---

Monitoring & Maintenance

Scheduled Tasks

1. Hourly: Refresh materialized view

   SELECT refresh_ml_model_performance();
   

2. Daily: Vacuum table (remove deleted rows)

   VACUUM ANALYZE ml_predictions;
   

3. Weekly: Check index bloat

   SELECT
       schemaname,
       tablename,
       indexname,
       pg_size_pretty(pg_relation_size(indexrelid)) as index_size
   FROM pg_stat_user_indexes
   WHERE schemaname = 'public' AND tablename = 'ml_predictions'
   ORDER BY pg_relation_size(indexrelid) DESC;
   

Alerts

1. High insert rate (>1000 predictions/min):

   • May indicate model overfitting or data quality issues
   • Monitor via Prometheus metric: ml_predictions_insert_rate

2. Low accuracy (<50% on ml_model_performance):

   • Trigger model retraining pipeline
   • Alert: ml_model_accuracy < 0.5 FOR 1h

3. Stale materialized view (>2 hours since refresh):

   • Check refresh function health
   • Alert: ml_model_performance_age > 7200s

---

Migration History

Migration	Status	Date	Notes
031	✅ Applied	2025-10-15	Created ml_predictions table, materialized view, refresh function
043	❌ Removed	2025-10-16	Duplicate of 031 - deleted to avoid conflicts

---

Testing Recommendations

Unit Tests

#[tokio::test]
async fn test_insert_ml_prediction() {
    // Test basic insert
}

#[tokio::test]
async fn test_update_prediction_outcome() {
    // Test outcome recording
}

#[tokio::test]
async fn test_constraint_validation() {
    // Test action/confidence constraints
}

Integration Tests

#[tokio::test]
async fn test_model_performance_view() {
    // Insert 100 predictions
    // Record 50 outcomes
    // Refresh materialized view
    // Assert accuracy calculation
}

#[tokio::test]
async fn test_sharpe_ratio_calculation() {
    // Insert predictions with known P&L
    // Verify Sharpe ratio matches expected value
}

---

Security Considerations

Access Control

• ✅ Application uses dedicated foxhunt user
• ✅ No direct TLI access to predictions table (read-only queries via API Gateway)
• ⚠️ TODO: Row-level security (RLS) if multi-tenant deployment

Data Privacy

• ✅ Features stored as JSONB (no PII)
• ✅ Symbol names are public market identifiers
• ⚠️ TODO: Audit logging for predictions table modifications

Injection Prevention

• ✅ All queries use parameterized SQLx macros (sqlx::query!)
• ✅ No dynamic SQL concatenation
• ✅ JSONB type prevents SQL injection via feature payloads

---

Future Enhancements

Phase 1 (Next 2 weeks)

1. Add model_version column (track model updates)
2. Add feature_importance JSONB column (explainability)
3. Create ml_prediction_errors table (track failed predictions)

Phase 2 (1 month)

4. Implement table partitioning (partition by prediction_timestamp)
5. Add TimescaleDB hypertable conversion (time-series optimization)
6. Create continuous aggregates for real-time metrics

Phase 3 (2-3 months)

7. Add ensemble_predictions table (track ensemble voting)
8. Implement A/B testing framework (model comparison)
9. Create ML pipeline observability dashboard (Grafana integration)

---

Conclusion

Status: ✅ PRODUCTION READY

The ML predictions database infrastructure is fully operational and validated:

• ✅ Migration 031 applied successfully
• ✅ Schema supports flexible feature storage
• ✅ Performance metrics tracked via materialized view
• ✅ Sharpe ratio calculation built-in
• ✅ Indexes optimized for common query patterns
• ✅ Insert/update operations tested successfully

Coordination: Agent 11 (ML order submission) can now use this schema to log predictions and coordinate outcome tracking.

Next Steps:

1. Agent 11: Implement ML prediction logging
2. Agent 12: Implement outcome recording pipeline
3. Agent 13: Create monitoring dashboard for ML model performance
4. Schedule hourly refresh of ml_model_performance view

---

Files Modified:

• None (schema already exists from migration 031)

Files Created:

• /home/jgrusewski/Work/foxhunt/WAVE_13_2_AGENT_14_ML_PREDICTIONS_REPORT.md (this report)

Files Deleted:

• /home/jgrusewski/Work/foxhunt/migrations/043_create_ml_predictions_table.sql (duplicate migration)

Migration Status:

• Migration 031: ✅ Applied (2025-10-15 21:16:26)
• Current version: 20250826000001 (latest)

---

Agent 14 Mission Complete ✅