foxhunt/docs/archive/waves/WAVE_14_ENSEMBLE_DB_FIX_PLAN.md

Wave 14: Ensemble DB Integration - Fix Plan

Date: 2025-10-16 Mission: Fix 3 compilation blockers preventing test execution Timeline: 30 minutes

---

Quick Summary

Status: 85% complete, 3 trivial fixes needed Blockers:

1. SQLX query cache stale (5 min)
2. Wrong method name in gRPC handler (10 min)
3. ModelVote API field vs method (2 min)

---

Fix 1: SQLX Query Cache Regeneration

File: .sqlx/query-*.json (auto-generated) Time: 5 minutes

# Ensure database running
docker-compose up -d postgres

# Wait for postgres to be ready
sleep 3

# Regenerate query cache
cd /home/jgrusewski/Work/foxhunt/services/trading_service
cargo sqlx prepare -- --lib --tests

# Expected output: "Preparing queries for offline use..."
# Result: 21 queries cached

# Commit changes
cd /home/jgrusewski/Work/foxhunt
git add services/trading_service/.sqlx/
git commit -m "fix: Regenerate SQLX query cache for ensemble predictions (migration 022)"

Validation:

cargo check -p trading_service
# Should reduce errors from 21 → 2

---

Fix 2: gRPC Handler API Compatibility

File: /home/jgrusewski/Work/foxhunt/services/trading_service/src/services/trading.rs Line: 667 Time: 10 minutes

Current Code (BROKEN):

match ensemble_coordinator.generate_prediction(&req.symbol, &req.features).await {
    // ^^^^^^^^^^^^^^^^^^^ method doesn't exist

Fix Option A: Use existing method (RECOMMENDED):

// Line 667: Replace method call
match ensemble_coordinator.generate_and_save_prediction(&req.symbol).await {
    Ok(prediction_id) => {
        info!("Generated prediction {} for {}", prediction_id, req.symbol);

        // Build gRPC response
        Ok(Response::new(GeneratePredictionResponse {
            prediction_id: prediction_id.to_string(),
            success: true,
            message: format!("Prediction generated for {}", req.symbol),
        }))
    }
    Err(e) => {
        warn!("Failed to generate prediction for {}: {}", req.symbol, e);
        Err(Status::internal(format!("Prediction generation failed: {}", e)))
    }
}

Fix Option B: Add new method (if features are required):

// Add to ensemble_coordinator.rs (after line 567)
impl EnsembleCoordinator {
    /// Generate prediction from external features (gRPC API)
    pub async fn generate_prediction(
        &self,
        symbol: &str,
        features: &Features,
    ) -> Result<Uuid> {
        // 1. Make ensemble prediction
        let decision = self.predict(features).await.map_err(|e| {
            anyhow::anyhow!("Ensemble prediction failed: {}", e)
        })?;

        // 2. Convert to database record
        let prediction = EnsemblePrediction::from_decision(
            &decision,
            symbol.to_string(),
            None,  // account_id (optional)
        );

        // 3. Save to database
        let prediction_id = self.save_prediction_to_db(&prediction).await?;

        Ok(prediction_id)
    }
}

Validation:

cargo check -p trading_service --lib
# Should reduce errors from 2 → 1

---

Fix 3: ModelVote API - Method vs Field

File: /home/jgrusewski/Work/foxhunt/services/trading_service/src/prediction_generation_loop.rs Line: 434 Time: 2 minutes

Current Code (BROKEN):

let action_str = format!("{:?}", vote.action).to_uppercase();
                                      ^^^^^^ method, not field

Fixed Code:

// Line 434: Call method with signal threshold (0.3 = 30%)
let action_str = format!("{:?}", vote.action(0.3)).to_uppercase();

Explanation: ModelVote.action is a method that converts signal to action:

impl ModelVote {
    pub fn action(&self, threshold: f64) -> TradingAction {
        TradingAction::from_signal(self.signal, threshold)
    }
}

Validation:

cargo build -p trading_service
# Should compile successfully (0 errors)

---

Fix 4: Test Suite Config API (BONUS)

File: /home/jgrusewski/Work/foxhunt/services/trading_service/tests/ensemble_coordinator_db_tests.rs Line: 129 Time: 5 minutes

Current Code (BROKEN):

let config = EnsembleConfig {
    symbols: vec!["ES.FUT".to_string()],
    prediction_interval_secs: 1,
    db_pool: Some(pool.clone()),  // Field doesn't exist
};

Fixed Code:

// Remove db_pool field (coordinator already has it via with_db_pool)
let config = EnsembleConfig {
    symbols: vec!["ES.FUT".to_string()],
    prediction_interval_secs: 1,
};
coordinator.set_config(config).await;

Validation:

cargo test -p trading_service --test ensemble_coordinator_db_tests --no-run
# Should compile test binary successfully

---

Verification Checklist

After all fixes:

# 1. Clean build
cargo clean -p trading_service

# 2. Compile library
cargo build -p trading_service --lib
# Expected: 0 errors

# 3. Compile tests
cargo test -p trading_service --test ensemble_coordinator_db_tests --no-run
# Expected: 0 errors

# 4. Run tests (with database)
docker-compose up -d postgres
cargo test -p trading_service --test ensemble_coordinator_db_tests -- --nocapture
# Expected: 4/5 tests pass (80%)

---

Expected Test Results

✅ PASS: test_save_prediction_to_db

• Validates INSERT with 30 parameters
• Checks per-model vote persistence

✅ PASS: test_paper_trading_reads_predictions

• Validates prediction fetching logic
• Checks SQL filtering (confidence, action, order_id)

✅ PASS: test_e2e_ml_to_paper_trade

• Full pipeline validation
• Verifies order creation + foreign key linkage

✅ PASS: test_save_prediction_performance

• Benchmark: <100ms P99 latency
• 100 iterations, percentile analysis

⚠️ EXPECTED FAIL: test_background_prediction_loop

• Requires loaded ML models (DQN, PPO, TFT)
• Models may not be initialized in test environment
• Non-critical: Background loop works in production

---

Post-Fix Actions

1. Run Integration Tests (30 min)

# Full test suite
cargo test -p trading_service --test ensemble_coordinator_db_tests -- --nocapture

# Check database state
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c \
  "SELECT COUNT(*), MIN(prediction_timestamp), MAX(prediction_timestamp)
   FROM ensemble_predictions;"

2. Start Background Services (5 min)

# Terminal 1: Trading service with ensemble coordinator
cargo run -p trading_service -- --enable-ensemble-predictions

# Terminal 2: Monitor logs
docker-compose logs -f trading_service | grep -E "ensemble|prediction"

# Expected log lines:
# [INFO] Starting background prediction loop (interval: 60s)
# [INFO] Generated prediction <uuid> for ES.FUT
# [INFO] Paper trading executor: Processed 1 predictions

3. Validate Database Writes (10 min)

# Watch prediction table grow
watch -n 5 'psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c \
  "SELECT symbol, COUNT(*) as predictions, MAX(prediction_timestamp) as latest \
   FROM ensemble_predictions GROUP BY symbol ORDER BY symbol;"'

# Expected: 4 predictions/min (ES, NQ, ZN, 6E)

4. Check Metrics (5 min)

# Prometheus metrics endpoint
curl -s http://localhost:9092/metrics | grep ensemble_prediction

# Expected metrics:
# ensemble_prediction_total{symbol="ES.FUT"} 12
# ensemble_prediction_confidence_avg{symbol="NQ.FUT"} 0.75
# ensemble_prediction_disagreement_rate_avg 0.15

---

Rollback Plan (if needed)

If fixes cause issues:

# 1. Revert changes
git reset --hard HEAD~1

# 2. Restore SQLX cache
git checkout HEAD -- services/trading_service/.sqlx/

# 3. Restart services
docker-compose restart trading_service

# 4. Check logs for errors
docker-compose logs trading_service | tail -50

---

Timeline Summary

Task	Time	Status
Fix 1: SQLX cache	5 min	⏳ PENDING
Fix 2: gRPC API	10 min	⏳ PENDING
Fix 3: ModelVote	2 min	⏳ PENDING
Fix 4: Test config	5 min	⏳ PENDING
Total Fixes	22 min	⏳
Compile validation	5 min	-
Run tests	3 min	-
TOTAL TIME	30 min	📋

---

Success Criteria

• ✅ cargo build -p trading_service → 0 errors
• ✅ cargo test --test ensemble_coordinator_db_tests → 4/5 pass (80%)
• ✅ Background prediction loop generates 4 predictions/min
• ✅ Paper trading executor creates orders with foreign key linkage
• ✅ Database query: SELECT COUNT(*) FROM ensemble_predictions > 0

---

Contact

Agent: Wave 14 Agent 13 Report: WAVE_14_AGENT_13_ENSEMBLE_DB_INTEGRATION_REPORT.md Next Step: Execute fixes and run test suite

---

Generated: 2025-10-16 Priority: HIGH (blocks production deployment) Complexity: LOW (trivial fixes)