================================================================================ PAPER TRADING PIPELINE - CURRENT vs REQUIRED ================================================================================ CURRENT STATE (0% Conversion Rate) ─────────────────────────────────────────────────────────────────────────────── ┌─────────────────────────────────────────────────────────────────────────────┐ │ ML ENSEMBLE PREDICTION FLOW │ └─────────────────────────────────────────────────────────────────────────────┘ Step 1: Data Loading ✅ ┌──────────────┐ │ DBN Data │ → ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT │ Loader │ (Real market data: 1,674-29,937 bars) └──────┬───────┘ │ ▼ Step 2: Feature Engineering ✅ ┌──────────────┐ │ Feature │ → 16 features + 10 technical indicators │ Extractor │ (RSI, MACD, Bollinger, ATR, EMA) └──────┬───────┘ │ ▼ Step 3: ML Model Inference ⚠️ (models not trained) ┌──────────────┐ │ DQN │ → Signal: NULL (no trained checkpoint) │ PPO │ → Signal: NULL (no trained checkpoint) │ MAMBA-2 │ → Signal: NULL (no trained checkpoint) │ TFT │ → Signal: NULL (no trained checkpoint) └──────┬───────┘ │ ▼ Step 4: Ensemble Aggregation ✅ ┌──────────────┐ │ Ensemble │ → Action: BUY/SELL/HOLD │ Coordinator │ Confidence: 49.93% (average) │ │ Disagreement: 50.31% └──────┬───────┘ │ ▼ Step 5: Audit Logging ✅ ┌──────────────────────────────────────────────────────────────┐ │ EnsembleAuditLogger.log_prediction() │ │ │ │ INSERT INTO ensemble_predictions ( │ │ symbol, ensemble_action, ensemble_signal, │ │ ensemble_confidence, disagreement_rate, │ │ dqn_signal, ppo_signal, mamba2_signal, tft_signal, │ │ order_id, executed_price, position_size │ │ ) VALUES (...) │ └──────────────────────┬───────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────────────┐ │ DATABASE: ensemble_predictions │ │ │ │ 3,000 rows: │ │ - symbol: TEST_SYM (not real market data!) │ │ - ensemble_action: BUY (980), SELL (1296), HOLD (724) │ │ - ensemble_confidence: 49.93% avg (low!) │ │ - order_id: NULL (no linkage to orders!) │ │ - executed_price: NULL (no execution!) │ └─────────────────────────────────────────────────────────────────────────────┘ │ │ ▼ ╔════════════════════╗ ║ ❌ MISSING GAP! ║ ║ ║ ║ NO CONSUMER TO ║ ║ READ PREDICTIONS ║ ║ AND CREATE ORDERS ║ ╚════════════════════╝ │ │ ▼ ┌─────────────────────────────────────────────────────────────────────────────┐ │ DATABASE: orders │ │ │ │ 0 rows with account_id = 'paper_trading_001' │ │ │ │ ❌ NO ORDERS CREATED! │ └─────────────────────────────────────────────────────────────────────────────┘ ================================================================================ REQUIRED STATE (Target: >50% Conversion Rate) ─────────────────────────────────────────────────────────────────────────────── ┌─────────────────────────────────────────────────────────────────────────────┐ │ COMPLETE PAPER TRADING EXECUTION PIPELINE │ └─────────────────────────────────────────────────────────────────────────────┘ Step 1-5: Same as above (ML Ensemble → Database) ✅ Step 6: NEW - Paper Trading Executor 🆕 ┌─────────────────────────────────────────────────────────────────────────────┐ │ PaperTradingExecutor (Background Task) │ │ │ │ tokio::spawn(async move { │ │ let mut interval = tokio::time::interval(Duration::from_millis(100)); │ │ │ │ loop { │ │ interval.tick().await; │ │ │ │ // 1. Fetch pending predictions │ │ let predictions = SELECT * FROM ensemble_predictions │ │ WHERE order_id IS NULL │ │ AND ensemble_confidence >= 0.60 │ │ AND ensemble_action IN ('BUY', 'SELL') │ │ AND symbol IN ('ES.FUT', 'NQ.FUT', 'ZN.FUT', '6E.FUT')│ │ AND timestamp > NOW() - INTERVAL '5 minutes' │ │ LIMIT 100; │ │ │ │ for prediction in predictions { │ │ // 2. Risk checks │ │ check_position_limits()?; │ │ check_circuit_breakers()?; │ │ │ │ // 3. Calculate position size │ │ let position_size = calculate_kelly_criterion(prediction); │ │ │ │ // 4. Create order │ │ let order_id = INSERT INTO orders ( │ │ id, symbol, side, order_type, quantity, limit_price, │ │ status, account_id, created_at │ │ ) VALUES ( │ │ UUID(), prediction.symbol, prediction.action, 'MARKET', │ │ position_size, current_price, 'FILLED', │ │ 'paper_trading_001', NOW() │ │ ) RETURNING id; │ │ │ │ // 5. Link prediction to order │ │ UPDATE ensemble_predictions │ │ SET order_id = order_id, │ │ executed_price = current_price, │ │ position_size = position_size │ │ WHERE id = prediction.id; │ │ │ │ info!("Executed: {} {} @ {} (conf: {:.2}%)", │ │ prediction.action, prediction.symbol, current_price, │ │ prediction.confidence * 100.0); │ │ } │ │ } │ │ }); │ └─────────────────────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────────────┐ │ DATABASE: ensemble_predictions │ │ │ │ 3,000 rows (after execution): │ │ - order_id: (LINKED! ✅) │ │ - executed_price: 4,531.25 (FILLED! ✅) │ │ - position_size: 10,000 USD (EXECUTED! ✅) │ └─────────────────────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────────────┐ │ DATABASE: orders │ │ │ │ >1,500 rows with account_id = 'paper_trading_001' │ │ │ │ ✅ ORDERS CREATED! │ │ ✅ 50%+ CONVERSION RATE! │ │ │ │ Example rows: │ │ id | symbol | side | quantity | status │ │ 01234567-89ab-cdef-0123-456789abcdef | ES.FUT | BUY | 2.0 | FILLED │ │ 12345678-9abc-def0-1234-56789abcdef0 | NQ.FUT | SELL | 1.5 | FILLED │ │ 23456789-abcd-ef01-2345-6789abcdef01 | ZN.FUT | BUY | 10.0 | FILLED │ └─────────────────────────────────────────────────────────────────────────────┘ ================================================================================ KEY DIFFERENCES ─────────────────────────────────────────────────────────────────────────────── CURRENT (Broken): ❌ No PaperTradingExecutor ❌ No background task polling predictions ❌ No order creation logic ❌ predictions.order_id = NULL ❌ 0 orders in database ❌ 0% conversion rate REQUIRED (Fixed): ✅ PaperTradingExecutor (600 lines) ✅ Background task (100ms interval) ✅ Order creation + linking ✅ predictions.order_id = ✅ >1,500 orders in database ✅ >50% conversion rate ================================================================================ IMPLEMENTATION CHECKLIST ─────────────────────────────────────────────────────────────────────────────── Files to Create: [ ] services/trading_service/src/paper_trading_executor.rs (600 lines) [ ] services/trading_service/src/paper_trading_config.rs (100 lines) [ ] services/trading_service/src/position_tracker.rs (200 lines) Files to Modify: [ ] services/trading_service/src/main.rs (add background task) [ ] services/trading_service/src/lib.rs (export new modules) Functions to Implement: [ ] PaperTradingExecutor::new() [ ] PaperTradingExecutor::start() - background loop [ ] fetch_pending_predictions() - SELECT query [ ] execute_prediction() - main execution logic [ ] create_order() - INSERT into orders [ ] link_prediction_to_order() - UPDATE prediction [ ] check_risk_limits() - position/circuit breaker validation [ ] calculate_position_size() - Kelly criterion or fixed % [ ] get_current_price() - from market data cache Tests to Create: [ ] Unit tests for PaperTradingExecutor [ ] Integration test: predictions → orders [ ] E2E test: full pipeline [ ] Conversion rate validation Time Estimate: 4 hours - Phase 1 (Core): 2 hours - Phase 2 (Risk): 1 hour - Phase 3 (Testing): 1 hour ================================================================================ VALIDATION COMMANDS ─────────────────────────────────────────────────────────────────────────────── Before Fix: psql -c "SELECT COUNT(*) FROM orders WHERE account_id LIKE '%paper%';" # Expected: 0 After Fix: psql -c "SELECT COUNT(*) FROM orders WHERE account_id LIKE '%paper%';" # Expected: >1500 psql -c "SELECT COUNT(*) FROM ensemble_predictions WHERE order_id IS NOT NULL;" # Expected: >1500 (50%+ of 3000) psql -c "SELECT symbol, side, COUNT(*) FROM orders WHERE account_id LIKE '%paper%' GROUP BY symbol, side;" # Expected: Real symbols (ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT) Monitoring: docker-compose logs trading_service | grep "paper_trading" # Expected: "Executed: BUY ES.FUT @ 4531.25 (conf: 72.34%)" ================================================================================ Report: /home/jgrusewski/Work/foxhunt/PAPER_TRADING_FIX_REPORT.md Summary: /home/jgrusewski/Work/foxhunt/PAPER_TRADING_FIX_SUMMARY.md