foxhunt/PAPER_TRADING_FIX_REPORT.md

Paper Trading Execution Fix Report

Date: 2025-10-14 Agent: Agent 131 (Paper Trading Fix) Status: 🔴 CRITICAL BUG IDENTIFIED

---

Executive Summary

Problem: 3,000 Predictions → 0 Orders (0% Conversion Rate)

Root Cause: Missing paper trading execution consumer service

The ML ensemble system is generating predictions successfully and logging them to the ensemble_predictions table, but there is no code to consume these predictions and convert them into orders. This is a critical missing component in the paper trading pipeline.

---

Investigation Findings

1. Prediction Generation: ✅ WORKING

Evidence:

• 3,000 predictions in ensemble_predictions table
• Generated between 15:06:07 and 16:06:36 UTC (1 hour)
• All predictions have valid ensemble decisions (BUY/SELL/HOLD)
• Average confidence: 49.93%
• Average disagreement: 50.31%

SELECT COUNT(*) FROM ensemble_predictions;
-- Result: 3000

SELECT ensemble_action, COUNT(*), AVG(ensemble_confidence)::numeric(5,2)
FROM ensemble_predictions
GROUP BY ensemble_action;
-- SELL: 1296 (43.2%), avg confidence 0.50
-- BUY:  980  (32.7%), avg confidence 0.49
-- HOLD: 724  (24.1%), avg confidence 0.50

2. Order Execution: ❌ NOT WORKING

Evidence:

• 0 orders in orders table with paper trading account
• All predictions have order_id = NULL
• No code found that:
  • Queries ensemble_predictions table
  • Filters by confidence threshold
  • Creates orders in orders table
  • Links orders to predictions

SELECT COUNT(*) FROM orders WHERE account_id LIKE '%paper%';
-- Result: 0 rows

SELECT COUNT(*) FROM ensemble_predictions WHERE order_id IS NOT NULL;
-- Result: 0 (no predictions linked to orders)

3. Symbol Routing: ❌ USING TEST DATA

Evidence:

• All 3,000 predictions use symbol TEST_SYM
• No real market symbols (ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT)
• Predictions likely generated by E2E test: /home/jgrusewski/Work/foxhunt/ml/tests/e2e_ensemble_integration.rs

SELECT DISTINCT symbol FROM ensemble_predictions;
-- Result: TEST_SYM (not a real trading symbol)

4. Confidence Thresholds: ⚠️ MEDIOCRE

Evidence:

• Average confidence: 49.93% (barely above random)
• Confidence range: 9.37% to 98.56%
• 49.9% threshold in docs may be too low
• No documented minimum confidence for order execution

High confidence predictions (>70%):

SELECT COUNT(*) FROM ensemble_predictions WHERE ensemble_confidence > 0.70;
-- Result: 916 predictions (30.5% of total)
-- These could be executed if consumer existed

5. Trading Service Code: ❌ NO CONSUMER

Missing Components:

1. Paper Trading Consumer: No service polling ensemble_predictions
2. Order Creation Logic: No code converting predictions → orders
3. Position Management: No tracking of open positions
4. Risk Checks: No validation before order submission
5. Execution Loop: No background task executing predictions

Existing Code (Audit Only):

• /home/jgrusewski/Work/foxhunt/services/trading_service/src/ensemble_audit_logger.rs: Writes predictions to DB ✅
• /home/jgrusewski/Work/foxhunt/services/trading_service/src/ensemble_metrics.rs: Prometheus metrics ✅
• Paper trading consumer: ❌ DOES NOT EXIST

---

Root Cause Analysis

Why 0% Conversion Rate?

The paper trading execution pipeline is incomplete:

┌─────────────────────────────────────────────────────────────┐
│                    Current Pipeline                         │
└─────────────────────────────────────────────────────────────┘

ML Ensemble → EnsembleAuditLogger → ensemble_predictions table
    ✅            ✅                        ✅
                                            │
                                            ❌ MISSING CONSUMER
                                            │
                                            ▼
                              [NO CODE HERE TO CONSUME]
                                            │
                                            ▼
                              Paper Trading Order Executor
                                        ❌ MISSING
                                            │
                                            ▼
                              orders table (account: paper_trading)
                                        ❌ EMPTY

What Should Exist (But Doesn't)

Missing Component: /home/jgrusewski/Work/foxhunt/services/trading_service/src/paper_trading_executor.rs

Required Functionality:

1. Background Task: Poll ensemble_predictions every 100ms
2. Filter Logic: Select predictions with:
   • order_id IS NULL (not yet executed)
   • ensemble_confidence >= THRESHOLD (e.g., 60%)
   • ensemble_action IN ('BUY', 'SELL') (exclude HOLD)
   • symbol IN (real_symbols) (exclude TEST_SYM)
3. Risk Checks: Validate against:
   • Position limits
   • Circuit breakers
   • Account balance
4. Order Creation: Insert into orders table
5. Link Prediction: Update ensemble_predictions.order_id
6. Position Tracking: Maintain open positions

---

Design: Paper Trading Executor Service

Architecture

// services/trading_service/src/paper_trading_executor.rs

pub struct PaperTradingExecutor {
    db_pool: PgPool,
    config: PaperTradingConfig,
    position_tracker: Arc<RwLock<HashMap<String, Position>>>,
    audit_logger: Arc<EnsembleAuditLogger>,
}

pub struct PaperTradingConfig {
    pub enabled: bool,
    pub min_confidence: f64,              // Default: 0.60 (60%)
    pub poll_interval_ms: u64,            // Default: 100ms
    pub max_position_size: f64,           // Default: 10,000 USD
    pub allowed_symbols: Vec<String>,     // ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT
    pub account_id: String,               // "paper_trading_001"
    pub initial_capital: f64,             // Default: 100,000 USD
}

impl PaperTradingExecutor {
    /// Start background task to consume predictions
    pub async fn start(&self) -> Result<()> {
        let mut interval = tokio::time::interval(
            Duration::from_millis(self.config.poll_interval_ms)
        );

        loop {
            interval.tick().await;

            // 1. Fetch unexecuted predictions
            let predictions = self.fetch_pending_predictions().await?;

            // 2. Filter by confidence and symbol
            let executable = self.filter_executable(predictions)?;

            // 3. Execute each prediction
            for prediction in executable {
                if let Err(e) = self.execute_prediction(prediction).await {
                    error!("Failed to execute prediction: {}", e);
                }
            }
        }
    }

    /// Fetch predictions ready for execution
    async fn fetch_pending_predictions(&self) -> Result<Vec<PendingPrediction>> {
        sqlx::query_as!(
            PendingPrediction,
            r#"
            SELECT id, symbol, ensemble_action, ensemble_signal, ensemble_confidence
            FROM ensemble_predictions
            WHERE order_id IS NULL
              AND ensemble_action IN ('BUY', 'SELL')
              AND ensemble_confidence >= $1
              AND symbol = ANY($2)
              AND timestamp > NOW() - INTERVAL '5 minutes'
            ORDER BY timestamp ASC
            LIMIT 100
            "#,
            self.config.min_confidence,
            &self.config.allowed_symbols,
        )
        .fetch_all(&self.db_pool)
        .await
        .map_err(|e| anyhow!("Failed to fetch predictions: {}", e))
    }

    /// Execute a single prediction as paper trading order
    async fn execute_prediction(&self, prediction: PendingPrediction) -> Result<()> {
        // 1. Check risk limits
        self.check_risk_limits(&prediction)?;

        // 2. Calculate position size
        let position_size = self.calculate_position_size(&prediction)?;

        // 3. Get current price (from market data or last trade)
        let current_price = self.get_current_price(&prediction.symbol).await?;

        // 4. Create order
        let order_id = self.create_order(&prediction, position_size, current_price).await?;

        // 5. Link order to prediction
        self.link_prediction_to_order(prediction.id, order_id).await?;

        // 6. Update position tracker
        self.update_position_tracker(&prediction.symbol, order_id, position_size).await?;

        info!(
            "Executed paper trade: {} {} @ {} (confidence: {:.2}%, order: {})",
            prediction.ensemble_action,
            prediction.symbol,
            current_price,
            prediction.ensemble_confidence * 100.0,
            order_id
        );

        Ok(())
    }

    /// Create order in database
    async fn create_order(
        &self,
        prediction: &PendingPrediction,
        position_size: f64,
        current_price: f64,
    ) -> Result<Uuid> {
        let order_id = Uuid::new_v4();

        sqlx::query!(
            r#"
            INSERT INTO orders (
                id, symbol, side, order_type, quantity, limit_price,
                status, account_id, created_at
            ) VALUES (
                $1, $2, $3, 'MARKET', $4, $5,
                'FILLED', $6, NOW()
            )
            "#,
            order_id,
            prediction.symbol,
            prediction.ensemble_action,
            position_size,
            current_price as i64,
            self.config.account_id,
        )
        .execute(&self.db_pool)
        .await?;

        Ok(order_id)
    }

    /// Link prediction to executed order
    async fn link_prediction_to_order(&self, prediction_id: Uuid, order_id: Uuid) -> Result<()> {
        sqlx::query!(
            r#"
            UPDATE ensemble_predictions
            SET order_id = $2
            WHERE id = $1
            "#,
            prediction_id,
            order_id,
        )
        .execute(&self.db_pool)
        .await?;

        Ok(())
    }
}

Integration into main.rs

// services/trading_service/src/main.rs

// Add paper trading executor
let paper_trading_config = PaperTradingConfig {
    enabled: true,
    min_confidence: 0.60, // 60% minimum confidence
    poll_interval_ms: 100,
    max_position_size: 10_000.0,
    allowed_symbols: vec![
        "ES.FUT".to_string(),
        "NQ.FUT".to_string(),
        "ZN.FUT".to_string(),
        "6E.FUT".to_string(),
    ],
    account_id: "paper_trading_001".to_string(),
    initial_capital: 100_000.0,
};

let paper_trading_executor = Arc::new(PaperTradingExecutor::new(
    db_pool.clone(),
    paper_trading_config,
    audit_logger.clone(),
));

// Spawn background task
tokio::spawn(async move {
    if let Err(e) = paper_trading_executor.start().await {
        error!("Paper trading executor failed: {}", e);
    }
});

---

Fix Implementation Plan

Phase 1: Core Infrastructure (2 hours)

Files to Create:

1. /home/jgrusewski/Work/foxhunt/services/trading_service/src/paper_trading_executor.rs
2. /home/jgrusewski/Work/foxhunt/services/trading_service/src/paper_trading_config.rs
3. /home/jgrusewski/Work/foxhunt/services/trading_service/src/position_tracker.rs

Implementation Steps:

1. Create PaperTradingExecutor struct (30 min)
2. Implement fetch_pending_predictions() (15 min)
3. Implement execute_prediction() (30 min)
4. Implement create_order() (15 min)
5. Implement link_prediction_to_order() (10 min)
6. Add to main.rs (10 min)
7. Unit tests (20 min)

Phase 2: Risk & Validation (1 hour)

Features:

1. Position size calculation (Kelly Criterion or fixed %)
2. Risk limits (max position, max drawdown)
3. Circuit breaker integration
4. Symbol validation (reject TEST_SYM)
5. Price fetching from market data cache

Phase 3: Testing & Validation (1 hour)

Tests:

1. Unit tests for PaperTradingExecutor
2. Integration test: Generate predictions → verify orders created
3. End-to-end test: Full pipeline (data → ML → predictions → orders)
4. Verify order_id linkage in ensemble_predictions

Validation Queries:

-- Check order creation
SELECT COUNT(*) FROM orders WHERE account_id = 'paper_trading_001';

-- Check prediction linkage
SELECT COUNT(*) FROM ensemble_predictions WHERE order_id IS NOT NULL;

-- Check conversion rate
SELECT
    COUNT(*) as total_predictions,
    SUM(CASE WHEN order_id IS NOT NULL THEN 1 ELSE 0 END) as executed,
    ROUND(100.0 * SUM(CASE WHEN order_id IS NOT NULL THEN 1 ELSE 0 END) / COUNT(*), 2) as conversion_rate
FROM ensemble_predictions
WHERE ensemble_action IN ('BUY', 'SELL');

---

Immediate Actions Required

1. Stop Using TEST_SYM (5 min)

Fix: Update E2E tests to use real symbols

// ml/tests/e2e_ensemble_integration.rs
// Change: "TEST_SYM" → "ES.FUT"

let symbols = vec!["ES.FUT", "NQ.FUT", "ZN.FUT", "6E.FUT"];

2. Implement Paper Trading Executor (4 hours)

Priority: HIGH Complexity: Medium Impact: Unlocks paper trading execution

3. Set Confidence Threshold (Config)

Recommendation: 60% minimum (not 49.9%)

pub const MIN_CONFIDENCE_THRESHOLD: f64 = 0.60;

Rationale:

• Current average: 49.93% (barely above random)
• High-confidence predictions (>70%): 916/3000 (30.5%)
• 60% threshold filters out noise while keeping good signals

4. Restart Trading Service

docker-compose restart trading_service
# Verify background task started
docker-compose logs trading_service | grep "paper_trading_executor"

---

Success Metrics

Target Performance (After Fix)

Metric	Before	Target	Notes
Conversion Rate	0%	>50%	Predictions → orders
Orders Created	0	>1500	3000 predictions × 50%+
Avg Confidence	49.93%	>65%	Filter low-confidence
Symbols	TEST_SYM	ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT	Real markets
Latency	N/A	<10ms	Prediction → order

Validation Queries (After Fix)

# 1. Check order creation
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt \
  -c "SELECT COUNT(*), symbol FROM orders WHERE account_id LIKE '%paper%' GROUP BY symbol;"

# Expected: >0 orders, real symbols

# 2. Check prediction linkage
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt \
  -c "SELECT COUNT(*) FROM ensemble_predictions WHERE order_id IS NOT NULL;"

# Expected: >50% of BUY/SELL predictions

# 3. Check conversion rate
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt \
  -c "SELECT
    COUNT(*) as total,
    SUM(CASE WHEN order_id IS NOT NULL THEN 1 ELSE 0 END) as executed,
    ROUND(100.0 * SUM(CASE WHEN order_id IS NOT NULL THEN 1 ELSE 0 END) / COUNT(*), 2) as rate
  FROM ensemble_predictions
  WHERE ensemble_action IN ('BUY', 'SELL');"

# Expected: >50% conversion rate

---

Conclusion

Summary

Problem: 3,000 predictions generating 0 orders (0% conversion rate)

Root Cause: Missing paper trading executor service to consume predictions and create orders

Solution: Implement PaperTradingExecutor background task in trading service

Estimated Time: 4 hours (2h core + 1h risk + 1h testing)

Impact: HIGH - Unlocks paper trading execution pipeline

Next Steps

1. ✅ Investigate complete (this report)
2. 🔄 Design reviewed (PaperTradingExecutor architecture)
3. ⏳ Implementation required (4 hours)
4. ⏳ Testing & validation (1 hour)
5. ⏳ Deploy & monitor (30 min)

---

Report Generated: 2025-10-14 Agent: 131 (Paper Trading Fix) Status: 🔴 CRITICAL BUG - AWAITING IMPLEMENTATION