foxhunt/WAVE_14_E2E_TEST_SCENARIOS.md

WAVE 14 AGENT 18: E2E Test Expansion Scenarios (85% → 95% Coverage)

Date: 2025-10-16 Mission: Expand end-to-end test coverage from 85% to 95% with realistic user scenarios Status: Implementation Phase

---

📊 Current E2E Test Coverage Analysis

Existing E2E Tests (22/22 passing)

Trading Service E2E Tests:

1. ✅ ml_paper_trading_e2e_test.rs - ML prediction → order → database (8 tests)
2. ✅ integration_e2e_tests.rs - Order placement → risk → execution
3. ✅ integration_end_to_end.rs - Complete order lifecycle
4. ✅ ml_integration_e2e_test.rs - ML model integration
5. ✅ order_execution_integration.rs - Order execution flows
6. ✅ position_lifecycle.rs - Position tracking

API Gateway E2E Tests: 7. ✅ ml_endpoints_test.rs - ML REST API endpoints (validation only) 8. ✅ service_proxy_tests.rs - gRPC proxy functionality 9. ✅ auth_flow_tests.rs - Authentication flows

TLI E2E Tests: 10. ✅ integration/end_to_end_tests.rs - Complete TLI workflows 11. ✅ integration/service_integration_tests.rs - Service integration 12. ✅ ml_trading_commands_test.rs - ML trading commands

Coverage Gaps (15% Missing)

Critical Missing Scenarios:

1. ❌ User Login → ML Order → PnL Update - Complete authenticated user flow
2. ❌ Backtest with Real DBN Data - ML strategy backtesting with ES.FUT
3. ❌ Portfolio Monitoring Flow - Predictions → Portfolio → Performance metrics
4. ❌ Error Scenario Tests - Auth failure, rate limit, invalid orders
5. ❌ Multi-Symbol Trading - Simultaneous ES.FUT + NQ.FUT orders
6. ❌ Ensemble Prediction → Risk Validation → Order Execution - Complete ML pipeline
7. ❌ TLI Command Integration - All tli trade ml commands with real services
8. ❌ Position Updates from Fills - Order fill → position update → PnL
9. ❌ Stop-Loss Triggers - Automatic stop-loss execution
10. ❌ Paper Trading Executor Loop - Continuous prediction polling → order creation

---

🎯 New E2E Test Scenarios

Scenario 1: Complete Authenticated User Flow

User Story: User logs in, submits ML-driven order, order executes, PnL updates

Flow:

1. User Authentication (TLI → API Gateway)
   - `tli auth login --username test_user --password ***`
   - JWT token generated and stored
   - Session validated

2. ML Prediction Request (TLI → API Gateway → Trading Service)
   - `tli trade ml predictions --symbol ES.FUT`
   - Ensemble coordinator generates prediction
   - Database saves prediction with per-model attribution

3. ML Order Submission (TLI → API Gateway → Trading Service)
   - `tli trade ml submit --symbol ES.FUT --confidence 0.75`
   - Risk validation (position limits, margin)
   - Order created in database

4. Order Execution (Trading Service → Database)
   - Order matched (simulated fill)
   - Execution record created
   - Position updated

5. PnL Update (Trading Service)
   - Real-time PnL calculation
   - Portfolio summary updated
   - User queries PnL: `tli portfolio summary`

6. Performance Metrics (TLI → API Gateway → Trading Service)
   - `tli trade ml performance`
   - Sharpe ratio, win rate, P95 latency

Test Assertions:

• ✅ JWT token valid and persisted
• ✅ Prediction saved with confidence ≥0.75
• ✅ Order created with correct side (BUY/SELL)
• ✅ Position reflects order quantity
• ✅ PnL matches expected value
• ✅ E2E latency <5 seconds

Implementation: services/trading_service/tests/e2e_authenticated_user_flow.rs

---

Scenario 2: Backtest ML Strategy with Real DBN Data

User Story: User runs backtest with ML strategy on ES.FUT historical data

Flow:

1. Load Real Market Data (test_data/ES.FUT.dbn.zst)
   - 1,674 OHLCV bars (2024-01-02)
   - 0.70ms load time (validated in Wave 7)

2. Feature Extraction (16 features + 10 technical indicators)
   - RSI, MACD, Bollinger Bands, ATR, EMA
   - 100% RSI validity

3. ML Strategy Execution (Adaptive ML Ensemble)
   - DQN, PPO, MAMBA-2, TFT predictions
   - Confidence-weighted voting
   - Risk-adjusted position sizing

4. Backtest Execution (Backtesting Service)
   - Order simulation
   - Slippage model (5 bps)
   - Commission model ($1.50/contract)

5. Performance Analytics
   - Sharpe ratio, max drawdown, win rate
   - Trade-by-trade breakdown
   - Equity curve generation

6. Report Export
   - JSON report with all metrics
   - CSV trade log
   - PNG equity curve

Test Assertions:

• ✅ 1,674 bars loaded in <10ms
• ✅ 16 features extracted per bar
• ✅ 4 models generate predictions (100% participation)
• ✅ Backtest completes in <30 seconds
• ✅ Sharpe ratio calculated (target: >1.0)
• ✅ Max drawdown <20%
• ✅ Trade count >50 (sufficient sample)

Implementation: services/backtesting_service/tests/e2e_ml_strategy_dbn_backtest.rs

---

Scenario 3: Portfolio Monitoring with ML Predictions

User Story: User monitors portfolio performance with ML-driven insights

Flow:

1. Generate ML Predictions (Multiple Symbols)
   - ES.FUT: BUY (0.82 confidence)
   - NQ.FUT: SELL (0.76 confidence)
   - CL.FUT: HOLD (0.55 confidence - filtered)

2. Execute High-Confidence Predictions
   - ES.FUT: Market BUY order (10 contracts)
   - NQ.FUT: Market SELL order (5 contracts)
   - CL.FUT: No action (below 0.60 threshold)

3. Real-Time Position Tracking
   - ES.FUT: +10 contracts @ $4,500 (Long)
   - NQ.FUT: -5 contracts @ $15,000 (Short)
   - Total exposure: $112,500

4. PnL Monitoring
   - ES.FUT: +$500 (price moved +$50/contract)
   - NQ.FUT: -$250 (adverse move)
   - Net PnL: +$250

5. Risk Metrics
   - VaR (95%): $5,000
   - Expected Shortfall: $7,500
   - Margin utilization: 45%

6. ML Performance Metrics
   - DQN: 8/10 correct (80% win rate)
   - PPO: 7/10 correct (70% win rate)
   - MAMBA-2: 9/10 correct (90% win rate)
   - TFT: 6/10 correct (60% win rate)
   - Ensemble: 85% win rate

Test Assertions:

• ✅ 3 predictions generated (2 execute, 1 filtered)
• ✅ Position tracking matches order fills
• ✅ PnL calculation correct (±$10 tolerance)
• ✅ VaR calculated (>0)
• ✅ Per-model performance tracked
• ✅ Portfolio summary query <100ms

Implementation: services/trading_service/tests/e2e_portfolio_monitoring_ml.rs

---

Scenario 4: Error Scenario Testing

User Story: System gracefully handles errors and provides clear feedback

Subtests:

4a. Authentication Failure

1. Invalid JWT token
   - Request: `tli trade submit --symbol ES.FUT --side buy --quantity 1`
   - Expected: 401 Unauthorized
   - Message: "Invalid or expired JWT token"

2. Expired token
   - Token created 25 hours ago
   - Expected: 401 Unauthorized
   - Message: "Token expired, please re-authenticate"

3. Missing token
   - Request without Authorization header
   - Expected: 401 Unauthorized
   - Message: "Authorization header missing"

4b. Rate Limiting

1. Exceed 100 requests/second
   - Send 150 requests in 1 second
   - Expected: First 100 succeed, next 50 get 429 Too Many Requests
   - Message: "Rate limit exceeded: 100 req/sec"

2. Burst traffic handling
   - 500 requests in 2 seconds (250 req/sec)
   - Expected: Throttled to 100 req/sec
   - Queue or reject excess

4c. Invalid Order Parameters

1. Invalid symbol
   - Request: `tli trade submit --symbol INVALID --side buy --quantity 1`
   - Expected: 400 Bad Request
   - Message: "Symbol 'INVALID' not supported"

2. Negative quantity
   - Request: `--quantity -10`
   - Expected: 400 Bad Request
   - Message: "Quantity must be positive"

3. Insufficient margin
   - Account balance: $10,000
   - Order value: $50,000
   - Expected: 403 Forbidden
   - Message: "Insufficient margin: required $50,000, available $10,000"

4. Invalid order type combination
   - Market order with limit price
   - Expected: 400 Bad Request
   - Message: "Market orders cannot have limit price"

4d. ML Model Errors

1. Model not loaded
   - Request prediction for non-existent model
   - Expected: 503 Service Unavailable
   - Message: "Model 'INVALID_MODEL' not loaded"

2. Feature dimension mismatch
   - Send 10 features instead of 16
   - Expected: 400 Bad Request
   - Message: "Expected 16 features, got 10"

3. GPU out of memory
   - Large batch prediction (>1000)
   - Expected: 503 Service Unavailable
   - Message: "GPU memory exhausted, try smaller batch"

Implementation: services/api_gateway/tests/e2e_error_scenarios.rs

---

Scenario 5: TLI Command Integration (All ML Commands)

User Story: All TLI ML commands work end-to-end with real services

Commands to Test:

5a. tli trade ml submit

tli trade ml submit \
  --symbol ES.FUT \
  --confidence 0.75 \
  --quantity 10

Expected:
✅ Order ID: abc123...
✅ Symbol: ES.FUT
✅ Side: BUY (from ML prediction)
✅ Confidence: 0.82
✅ Status: PENDING

5b. tli trade ml predictions

tli trade ml predictions \
  --symbol ES.FUT \
  --count 5

Expected:
┌──────────┬────────┬────────┬────────────┬──────────────┐
│ Symbol   │ Action │ Signal │ Confidence │ Disagreement │
├──────────┼────────┼────────┼────────────┼──────────────┤
│ ES.FUT   │ BUY    │ 0.65   │ 0.82       │ 0.12         │
│ ES.FUT   │ SELL   │ -0.45  │ 0.75       │ 0.18         │
│ ES.FUT   │ HOLD   │ 0.02   │ 0.55       │ 0.35         │
└──────────┴────────┴────────┴────────────┴──────────────┘

5c. tli trade ml performance

tli trade ml performance \
  --days 30

Expected:
📊 ML Trading Performance (Last 30 Days)
─────────────────────────────────────────
Total Predictions:     1,245
Executed Orders:       856 (68.7%)
Win Rate:              72.4%
Sharpe Ratio:          1.85
P95 Latency:           145ms
─────────────────────────────────────────
Model Performance:
  DQN:      75.2% win rate
  PPO:      68.9% win rate
  MAMBA-2:  78.5% win rate
  TFT:      67.1% win rate
  Ensemble: 72.4% win rate

Implementation: tli/tests/e2e_ml_commands_integration.rs

---

Scenario 6: Ensemble Prediction → Risk Validation → Order Execution

User Story: Complete ML pipeline with risk checks

Flow:

1. Market Data Ingestion
   - ES.FUT: $4,500 (current price)
   - NQ.FUT: $15,000 (current price)
   - Real-time tick data

2. Feature Engineering
   - 16 OHLCV-derived features
   - 10 technical indicators
   - Normalization (z-score)

3. Ensemble Prediction
   - DQN: BUY (0.85 confidence)
   - PPO: BUY (0.78 confidence)
   - MAMBA-2: BUY (0.92 confidence)
   - TFT: HOLD (0.55 confidence)
   - Ensemble: BUY (0.82 confidence, 0.15 disagreement)

4. Risk Validation
   - Position limit: 20 contracts (current: 5, new: 10, total: 15 ✅)
   - Margin requirement: $15,000 (available: $50,000 ✅)
   - VaR check: $5,000 (limit: $10,000 ✅)
   - Symbol whitelist: ES.FUT ✅
   - Confidence threshold: 0.60 ✅

5. Order Creation
   - Symbol: ES.FUT
   - Side: BUY
   - Quantity: 10
   - Type: MARKET
   - Confidence: 0.82

6. Order Execution
   - Fill price: $4,502 (2 bps slippage)
   - Commission: $15 ($1.50/contract)
   - Execution time: 12ms

7. Position Update
   - Previous: 5 contracts @ $4,495 avg
   - New: 15 contracts @ $4,499 avg
   - Unrealized PnL: +$60

8. Database Persistence
   - Prediction: ensemble_predictions table
   - Order: orders table
   - Execution: executions table
   - Position: positions table
   - Link: prediction.order_id = order.id

Test Assertions:

• ✅ 4 model predictions generated
• ✅ Ensemble confidence = 0.82
• ✅ All 5 risk checks pass
• ✅ Order created with correct parameters
• ✅ Execution recorded within 50ms
• ✅ Position reflects new quantity (15)
• ✅ Unrealized PnL = +$60 (±$5 tolerance)
• ✅ All database records linked correctly
• ✅ E2E pipeline latency <2 seconds

Implementation: services/trading_service/tests/e2e_ensemble_risk_execution_pipeline.rs

---

Scenario 7: Multi-Symbol Simultaneous Trading

User Story: System handles concurrent trading across multiple symbols

Flow:

1. Generate Predictions for 3 Symbols
   - ES.FUT: BUY (0.85 confidence) @ $4,500
   - NQ.FUT: SELL (0.78 confidence) @ $15,000
   - CL.FUT: BUY (0.72 confidence) @ $75

2. Submit Orders Simultaneously (Async)
   - Thread 1: ES.FUT BUY 10 contracts
   - Thread 2: NQ.FUT SELL 5 contracts
   - Thread 3: CL.FUT BUY 50 contracts

3. Risk Validation (Per Symbol)
   - ES.FUT: Position limit 20, current 0 → 10 ✅
   - NQ.FUT: Position limit 10, current 0 → 5 ✅
   - CL.FUT: Position limit 100, current 0 → 50 ✅
   - Total margin: $90,000 (available: $200,000 ✅)

4. Concurrent Execution
   - ES.FUT: Filled in 15ms
   - NQ.FUT: Filled in 18ms
   - CL.FUT: Filled in 22ms

5. Portfolio State
   - ES.FUT: +10 contracts (Long)
   - NQ.FUT: -5 contracts (Short)
   - CL.FUT: +50 contracts (Long)
   - Total positions: 3
   - Total exposure: $142,500

6. Aggregate PnL Calculation
   - ES.FUT: +$200 (price +$20)
   - NQ.FUT: -$150 (price +$30, short position)
   - CL.FUT: +$500 (price +$10)
   - Net PnL: +$550

Test Assertions:

• ✅ 3 predictions generated simultaneously
• ✅ 3 orders created without deadlocks
• ✅ All risk checks pass
• ✅ Executions complete within 100ms (P99)
• ✅ No race conditions (positions consistent)
• ✅ Aggregate PnL correct (±$20 tolerance)
• ✅ Database transactions isolated (no conflicts)

Implementation: services/trading_service/tests/e2e_multi_symbol_concurrent_trading.rs

---

Scenario 8: Paper Trading Executor Loop

User Story: Continuous prediction polling and order execution

Flow:

1. Start Paper Trading Executor
   - Poll interval: 100ms
   - Confidence threshold: 0.60
   - Max position: 10 contracts
   - Symbols: ES.FUT, NQ.FUT

2. Background Prediction Generation
   - ML service generates predictions every 5 seconds
   - Saves to ensemble_predictions table
   - No order_id (pending execution)

3. Executor Polling Loop
   Iteration 1 (t=0s):
   - Query: SELECT * FROM ensemble_predictions WHERE order_id IS NULL
   - Found: 1 prediction (ES.FUT BUY, 0.85 confidence)
   - Action: Create market BUY order
   - Update: prediction.order_id = new_order_id

   Iteration 2 (t=5s):
   - Query: 0 pending predictions
   - Action: Sleep 100ms

   Iteration 3 (t=10s):
   - Query: 2 predictions (ES.FUT SELL 0.78, NQ.FUT BUY 0.82)
   - Action: Create 2 orders
   - Update: Link both predictions to orders

   Iteration 4 (t=15s):
   - Query: 1 prediction (CL.FUT BUY 0.55)
   - Action: Skip (confidence <0.60)
   - Update: None (no order created)

4. Position Limits Check
   Iteration 5 (t=20s):
   - Query: ES.FUT BUY 0.88
   - Current position: 10 contracts (at limit)
   - Action: Skip (max position reached)
   - Log: "Position limit reached for ES.FUT"

5. Executor Shutdown
   - Stop signal received
   - Complete in-flight orders
   - Clean shutdown (no orphaned predictions)

Test Assertions:

• ✅ Executor polls every 100ms (±10ms jitter)
• ✅ High-confidence predictions (≥0.60) execute
• ✅ Low-confidence predictions (<0.60) skipped
• ✅ Position limits enforced
• ✅ All predictions linked to orders (or skipped)
• ✅ Clean shutdown with no orphans
• ✅ No database deadlocks (concurrent access)
• ✅ Executor processes 1000+ predictions in test (stress test)

Implementation: services/trading_service/tests/e2e_paper_trading_executor_loop.rs

---

Scenario 9: Stop-Loss Trigger and Execution

User Story: Automatic stop-loss execution on adverse price moves

Flow:

1. Open Position with Stop-Loss
   - Symbol: ES.FUT
   - Side: LONG
   - Quantity: 10 contracts
   - Entry price: $4,500
   - Stop-loss: $4,450 (1.11% below entry)
   - Risk per contract: $50

2. Market Price Movement (Adverse)
   t=0s:  $4,500 (entry)
   t=10s: $4,480 (down $20)
   t=20s: $4,460 (down $40)
   t=30s: $4,450 (STOP TRIGGERED)

3. Stop-Loss Trigger Detection
   - Price monitor: Real-time tick data
   - Trigger condition: market_price <= stop_price
   - Action: Generate market SELL order

4. Stop-Loss Order Execution
   - Symbol: ES.FUT
   - Side: SELL (close long position)
   - Quantity: 10 contracts (full position)
   - Type: MARKET (immediate execution)
   - Fill price: $4,448 (2 bps slippage on exit)

5. Position Close and PnL Calculation
   - Entry: 10 contracts @ $4,500 = $45,000
   - Exit: 10 contracts @ $4,448 = $44,480
   - Gross PnL: -$520 ($44,480 - $45,000)
   - Commission: $30 ($1.50 * 2 sides * 10 contracts)
   - Net PnL: -$550
   - Realized PnL recorded in database

6. Risk Metrics Update
   - Max drawdown: $550 (1.22%)
   - Stop-loss hit rate: 1/1 (100% in test)
   - Average stop-loss slippage: 2 bps

Test Assertions:

• ✅ Position opened with stop-loss
• ✅ Stop-loss triggered at $4,450
• ✅ Market SELL order created immediately
• ✅ Position closed (quantity = 0)
• ✅ Realized PnL = -$550 (±$10 tolerance)
• ✅ Stop-loss execution latency <100ms
• ✅ Database reflects closed position

Implementation: services/trading_service/tests/e2e_stop_loss_trigger_execution.rs

---

Scenario 10: Order Fill → Position Update → PnL Calculation

User Story: Accurate position tracking and PnL calculation from order fills

Flow:

1. Initial State
   - Account: test_user
   - Cash: $100,000
   - Positions: None

2. Order 1: ES.FUT LONG 5 contracts
   - Order submitted: t=0s
   - Fill price: $4,500
   - Commission: $7.50
   - Position update:
     * Quantity: 5
     * Avg price: $4,500
     * Cost basis: $22,500 + $7.50 = $22,507.50
   - Cash remaining: $77,492.50

3. Market Price Movement (Favorable)
   - ES.FUT: $4,500 → $4,520 (+$20)
   - Unrealized PnL: 5 * $20 = +$100
   - Mark-to-market value: $22,600

4. Order 2: ES.FUT LONG 10 contracts (add to position)
   - Order submitted: t=30s
   - Fill price: $4,520
   - Commission: $15
   - Position update:
     * Previous: 5 @ $4,500
     * Add: 10 @ $4,520
     * New quantity: 15
     * New avg price: ($22,500 + $45,200) / 15 = $4,513.33
     * Cost basis: $67,700 + $22.50 = $67,722.50
   - Cash remaining: $32,277.50

5. Market Price Movement (Mixed)
   - ES.FUT: $4,520 → $4,510 (-$10 from recent entry)
   - Unrealized PnL: 15 * ($4,510 - $4,513.33) = -$50
   - Mark-to-market value: $67,650

6. Order 3: ES.FUT SELL 8 contracts (partial close)
   - Order submitted: t=60s
   - Fill price: $4,510
   - Commission: $12
   - Position update:
     * Previous: 15 @ $4,513.33
     * Close: 8 @ $4,510
     * Realized PnL: 8 * ($4,510 - $4,513.33) = -$26.64
     * New quantity: 7
     * Avg price: $4,513.33 (unchanged)
     * Cost basis: 7 * $4,513.33 = $31,593.31
   - Cash: $32,277.50 + $36,080 (proceeds) - $12 (commission) = $68,345.50
   - Realized PnL: -$26.64 - $12 = -$38.64

7. Final State
   - Position: 7 contracts @ $4,513.33 avg
   - Unrealized PnL: 7 * ($4,510 - $4,513.33) = -$23.31
   - Realized PnL: -$38.64
   - Total PnL: -$61.95
   - Cash: $68,345.50
   - Portfolio value: $68,345.50 + $31,570 = $99,915.50 (loss reflects PnL)

Test Assertions:

• ✅ Position quantity correct after each fill (5 → 15 → 7)
• ✅ Average price calculated correctly ($4,513.33)
• ✅ Unrealized PnL accurate (±$5 tolerance)
• ✅ Realized PnL calculated on partial close (-$38.64)
• ✅ Cash balance updated correctly
• ✅ Commission tracked ($7.50 + $15 + $12 = $34.50)
• ✅ Portfolio value = cash + mark-to-market positions

Implementation: services/trading_service/tests/e2e_order_fill_position_pnl.rs

---

📈 Coverage Improvement Roadmap

Phase 1: Critical User Flows (Week 1)

• ✅ Scenario 1: Authenticated user flow
• ✅ Scenario 6: Ensemble → Risk → Execution pipeline
• ✅ Scenario 8: Paper trading executor loop

Target Coverage: 88% (+3%)

Phase 2: ML Integration (Week 1)

• ✅ Scenario 2: Backtest with real DBN data
• ✅ Scenario 3: Portfolio monitoring
• ✅ Scenario 5: TLI command integration

Target Coverage: 92% (+4%)

Phase 3: Error Handling & Edge Cases (Week 2)

• ✅ Scenario 4: Error scenarios
• ✅ Scenario 7: Multi-symbol concurrent trading
• ✅ Scenario 9: Stop-loss triggers
• ✅ Scenario 10: Order fill → position → PnL

Target Coverage: 95% (+3%)

---

🎯 Success Metrics

Quantitative:

• ✅ Coverage: 85% → 95% (+10%)
• ✅ E2E test count: 22 → 50+ tests (+128%)
• ✅ All scenarios pass (100% pass rate)
• ✅ E2E test execution time: <5 minutes
• ✅ Real DBN data used (no mocks for market data)

Qualitative:

• ✅ Realistic user workflows validated
• ✅ Error scenarios comprehensively tested
• ✅ ML pipeline fully integrated
• ✅ TLI commands tested with real services
• ✅ Documentation complete (this file)

---

📝 Implementation Notes

Test Data Requirements

• ✅ ES.FUT DBN data: test_data/ES.FUT.dbn.zst (1,674 bars)
• ✅ NQ.FUT DBN data: Available
• ✅ CL.FUT DBN data: Available
• ✅ PostgreSQL test database: foxhunt (localhost:5432)
• ✅ Redis: localhost:6379 (session management)

Test Infrastructure

• ✅ Real database (not mocks) for E2E tests
• ✅ API Gateway running (port 50051)
• ✅ Trading Service running (port 50052)
• ✅ ML models loaded (DQN, PPO, MAMBA-2, TFT)
• ✅ Test fixtures for user accounts

Performance Targets

• ✅ E2E latency: <5 seconds per scenario
• ✅ Order execution: <50ms (P95)
• ✅ ML prediction: <200ms (4 models)
• ✅ Database operations: <10ms (P95)

---

🚀 Next Steps

1. Complete Analysis (Task 1) → Mark complete
2. Implement Scenario 1 (Task 2) → Start implementation
3. Implement Scenario 6 (Task 2) → High priority
4. Implement remaining scenarios (Tasks 3-6) → Parallel development
5. Measure final coverage (Task 7) → Validation

---

Status: Ready for implementation Estimated Effort: 3-4 days (10 scenarios × 4 hours average) Priority: HIGH (production readiness blocker)