foxhunt/AGENT_11.9_E2E_REAL_IMPLEMENTATIONS.md

Agent 11.9: E2E Tests with Real Implementations

Mission: Replace ALL mock/stub implementations in E2E tests with real production components.

Date: 2025-10-16

---

Current State Analysis

Mock/Stub Usage Identified

1. MLPipelineTestHarness (tests/e2e/src/ml_pipeline.rs):

   • ❌ mock_prediction() method (lines 384-411)
   • ❌ Falls back to mocks even in "real" mode (line 421-427)
   • ❌ Hardcoded model availability checks (line 602-613)

2. Paper Trading Tests (tests/e2e/tests/e2e_ml_paper_trading_test.rs):

   • ❌ MockMLInferenceEngine (lines 87-122)
   • ❌ MockPaperTradingExecutor (lines 125-272)
   • ❌ All tests use mock structures instead of real services

3. Backtesting Tests (tests/e2e/tests/e2e_ml_backtesting_test.rs):

   • ❌ MockBacktestingEngine (lines 86-178)
   • ❌ Simulated backtest execution instead of real service calls

4. Mock Infrastructure (tests/e2e/src/mocks/mod.rs):

   • ❌ dual_provider_mocks module for market data
   • ❌ Should use real DBN data instead

Real Implementations Available

✅ ML Inference:

• ml::inference::RealMLInferenceEngine - Production ML inference with safety checks
• ml::ensemble::EnsembleCoordinator - Real ensemble aggregation
• ml::ensemble::AdaptiveMLEnsemble - Regime-aware ensemble

✅ Trading Service:

• services/trading_service::PaperTradingExecutor - Real paper trading
• common::ml_strategy::SharedMLStrategy - Shared ML strategy interface

✅ Data Sources:

• data::DbnDataSource - Real DBN market data (ES.FUT, NQ.FUT, CL.FUT, ZN.FUT, 6E.FUT)
• data::parquet_persistence - Parquet-based data loading (0.70ms for 1,674 bars)

---

Implementation Plan

Phase 1: Update MLPipelineTestHarness ✅ PRIORITY

File: tests/e2e/src/ml_pipeline.rs

Changes:

use ml::inference::{RealMLInferenceEngine, RealInferenceConfig};
use ml::ensemble::{EnsembleCoordinator, AdaptiveMLEnsemble};
use data::DbnDataSource;

pub struct MLPipelineTestHarness {
    // BEFORE (mock):
    // model_status: MLModelStatus,
    // mock_mode: bool,

    // AFTER (real):
    real_ml_engine: Arc<RealMLInferenceEngine>,
    ensemble_coordinator: Arc<EnsembleCoordinator>,
    adaptive_ensemble: Arc<AdaptiveMLEnsemble>,
    dbn_data_source: Arc<DbnDataSource>,
    model_metrics: HashMap<String, ModelMetrics>,
    feature_cache: HashMap<String, Vec<FeatureVector>>,
}

impl MLPipelineTestHarness {
    pub async fn new() -> Result<Self> {
        // Initialize REAL components
        let config = RealInferenceConfig::default();
        let real_ml_engine = Arc::new(RealMLInferenceEngine::new(config).await?);

        let ensemble_coordinator = Arc::new(EnsembleCoordinator::new());
        ensemble_coordinator.register_model("DQN".to_string(), 0.25).await?;
        ensemble_coordinator.register_model("PPO".to_string(), 0.25).await?;
        ensemble_coordinator.register_model("MAMBA2".to_string(), 0.25).await?;
        ensemble_coordinator.register_model("TFT".to_string(), 0.25).await?;

        let adaptive_ensemble = Arc::new(AdaptiveMLEnsemble::new().await?);

        // Load real DBN data
        let test_data_dir = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
            .parent().unwrap()
            .parent().unwrap()
            .join("test_data");
        let dbn_data_source = Arc::new(DbnDataSource::new(test_data_dir).await?);

        Ok(Self {
            real_ml_engine,
            ensemble_coordinator,
            adaptive_ensemble,
            dbn_data_source,
            model_metrics: HashMap::new(),
            feature_cache: HashMap::new(),
        })
    }

    // REMOVE mock_prediction() entirely
    // REMOVE real_prediction() fallback

    // NEW: Use real ML inference
    async fn predict_with_model(
        &mut self,
        model_name: &str,
        features: &[FeatureVector],
    ) -> Result<MLPrediction> {
        let start_time = Instant::now();

        // Convert features to ML format
        let ml_features = self.convert_to_ml_features(features)?;

        // REAL inference via ensemble coordinator
        let decision = self.ensemble_coordinator.predict(&ml_features).await?;

        // Find model-specific vote
        let model_vote = decision.model_votes.iter()
            .find(|v| v.model_id == model_name)
            .ok_or_else(|| anyhow::anyhow!("Model {} not in ensemble", model_name))?;

        let inference_time = start_time.elapsed();

        Ok(MLPrediction {
            signal: model_vote.predicted_value,
            confidence: model_vote.confidence,
            model_name: model_name.to_string(),
            inference_time,
        })
    }

    // NEW: Real ensemble prediction
    pub async fn predict_ensemble(
        &mut self,
        features: &[FeatureVector],
    ) -> Result<EnsemblePrediction> {
        let start_time = Instant::now();

        // Convert features
        let ml_features = self.convert_to_ml_features(features)?;

        // REAL ensemble decision
        let decision = self.ensemble_coordinator.predict(&ml_features).await?;

        // Convert model votes to ML predictions
        let individual_predictions: Vec<MLPrediction> = decision.model_votes.iter()
            .map(|vote| MLPrediction {
                signal: vote.predicted_value,
                confidence: vote.confidence,
                model_name: vote.model_id.clone(),
                inference_time: Duration::from_micros(50), // Approximate
            })
            .collect();

        let total_inference_time = start_time.elapsed();

        // Map action to prediction type
        let prediction = match decision.action {
            TradingAction::Buy => PredictionType::Buy,
            TradingAction::Sell => PredictionType::Sell,
            TradingAction::Hold => PredictionType::Hold,
            TradingAction::StrongBuy => PredictionType::StrongBuy,
            TradingAction::StrongSell => PredictionType::StrongSell,
        };

        Ok(EnsemblePrediction {
            signal: decision.aggregated_value,
            confidence: decision.confidence,
            individual_predictions,
            ensemble_method: "real_weighted_voting".to_string(),
            total_inference_time,
            prediction,
            signal_strength: decision.aggregated_value.abs(),
        })
    }
}

Expected Outcome:

• ✅ All ML predictions use real models
• ✅ Real ensemble coordination
• ✅ Real feature extraction
• ✅ No fallback to mocks

---

Phase 2: Update Paper Trading Tests

File: tests/e2e/tests/e2e_ml_paper_trading_test.rs

Changes:

// REMOVE MockMLInferenceEngine entirely (lines 87-122)
// REMOVE MockPaperTradingExecutor entirely (lines 125-272)

// USE REAL implementations
use services::trading_service::PaperTradingExecutor;
use ml::inference::RealMLInferenceEngine;
use common::ml_strategy::SharedMLStrategy;

#[tokio::test]
async fn test_e2e_checkpoint_to_order() -> Result<()> {
    // REAL database pool
    let pool = get_test_db_pool().await;

    // REAL ML engine
    let ml_config = RealInferenceConfig {
        checkpoint_dir: PathBuf::from("ml/checkpoints"),
        device: Device::cuda_if_available(0)?,
        max_inference_latency_us: 100,
        ..Default::default()
    };
    let ml_engine = RealMLInferenceEngine::new(ml_config).await?;

    // REAL paper trading executor
    let shared_strategy = SharedMLStrategy::new(Arc::new(ml_engine));
    let mut executor = PaperTradingExecutor::new_with_ml(
        pool.clone(),
        shared_strategy,
    ).await?;

    // REAL DBN market data
    let dbn_source = DbnDataSource::new(PathBuf::from("test_data")).await?;
    let bars = dbn_source.load_ohlcv_bars("ES.FUT").await?;

    // REAL feature extraction
    let features = extract_256_dim_features(&bars)?;

    // REAL ML signal generation
    let signal = executor.generate_ml_signal(&features).await?;

    // Verify REAL prediction
    assert!(signal.action.is_some(), "Real ML signal should have action");
    assert_eq!(signal.source, SignalSource::ML, "Should be from real ML");

    // REAL order execution
    let order = executor.execute_ml_signal(&signal, "ES.FUT").await?;

    // Verify in database
    let prediction = sqlx::query!(...)
        .fetch_one(&pool)
        .await?;

    assert_eq!(prediction.symbol, "ES.FUT");

    Ok(())
}

Expected Outcome:

• ✅ All paper trading tests use real PaperTradingExecutor
• ✅ Real ML inference engine
• ✅ Real DBN data
• ✅ Real database persistence

---

Phase 3: Update Backtesting Tests

File: tests/e2e/tests/e2e_ml_backtesting_test.rs

Changes:

// REMOVE MockBacktestingEngine entirely (lines 86-178)

// USE REAL backtesting service via gRPC
use crate::proto::backtesting::backtesting_service_client::BacktestingServiceClient;

#[tokio::test]
async fn test_e2e_checkpoint_to_backtest_metrics() -> Result<()> {
    let mut framework = E2ETestFramework::new().await?;
    framework.start_services().await?;

    // REAL backtesting client via API Gateway
    let client = framework.get_backtesting_client().await?;

    // REAL backtest request
    let request = tonic::Request::new(BacktestRequest {
        strategy: "MLEnsemble".to_string(),
        symbol: "ES.FUT".to_string(),
        start_date: "2024-01-02".to_string(),
        end_date: "2024-01-10".to_string(),
        initial_capital: 100000.0,
        ml_config: Some(MlConfig {
            models: vec!["DQN", "PPO", "MAMBA2", "TFT"],
            confidence_threshold: 0.6,
            ensemble_method: "weighted_voting",
        }),
    });

    // REAL backtesting service execution
    let response = client.run_backtest(request).await?;
    let results = response.into_inner();

    // Verify REAL metrics
    assert!(results.total_trades > 0);
    assert!(results.sharpe_ratio > 1.5); // Real target
    assert!(results.win_rate > 0.55); // Real target

    // REAL database verification
    let record = sqlx::query!(
        "SELECT * FROM backtest_runs WHERE id = $1",
        results.backtest_id
    )
    .fetch_one(&framework.database_harness.pool)
    .await?;

    assert_eq!(record.strategy, "MLEnsemble");

    framework.stop_services().await?;
    Ok(())
}

Expected Outcome:

• ✅ All backtesting tests use real BacktestingService
• ✅ Real gRPC communication via API Gateway
• ✅ Real ML models in backtest
• ✅ Real performance metrics

---

Phase 4: Remove Mock Infrastructure

Files to DELETE:

1. tests/e2e/src/mocks/mod.rs - Remove entire mocks module
2. tests/e2e/src/mocks/dual_provider_mocks.rs - Remove dual provider mocks

Files to UPDATE:

1. tests/e2e/src/lib.rs - Remove pub mod mocks;
2. All test files using use crate::mocks::*; - Replace with real implementations

Expected Outcome:

• ✅ Zero mock infrastructure
• ✅ All tests use production components
• ✅ Clean separation of concerns

---

Integration with Real Data

DBN Data Usage

Available Data:

• test_data/ES.FUT.20240102.dbn - E-mini S&P 500 (1,674 bars)
• test_data/NQ.FUT.20240102.dbn - Nasdaq futures
• test_data/CL.FUT.20240102.dbn - Crude Oil futures
• test_data/ZN.FUT.dbn - Treasury futures (28,935 bars)
• test_data/6E.FUT.dbn - Euro FX futures (29,937 bars)

Loading Pattern:

use data::DbnDataSource;

let dbn_source = DbnDataSource::new(PathBuf::from("test_data")).await?;
let bars = dbn_source.load_ohlcv_bars("ES.FUT").await?;
// 0.70ms load time - production ready!

// Extract real features
let features = extract_256_dim_features(&bars)?;
// 16 OHLCV + 240 technical indicators = 256 features

---

Feature Extraction

Real Feature Pipeline

Use Existing Implementation:

• ml/src/features/extraction.rs - extract_256_dim_features()
• 5 OHLCV base features
• 10 technical indicators (RSI, MACD, Bollinger, ATR, EMA, etc.)
• 241 additional derived features
• Total: 256 features per bar

Integration:

use ml::features::extraction::extract_256_dim_features;

let bars = dbn_source.load_ohlcv_bars("ES.FUT").await?;
let features = extract_256_dim_features(&bars)?;

// Use in ML inference
let prediction = ml_engine.predict(&features).await?;

---

Test Coverage Validation

E2E Test Suite Structure

After Real Implementation Migration:

tests/e2e/tests/
├── e2e_ml_paper_trading_test.rs (6 tests) ✅ REAL
├── e2e_ml_backtesting_test.rs (6 tests)   ✅ REAL
├── ml_inference_e2e.rs                    ✅ REAL
├── multi_service_integration.rs           ✅ REAL
├── comprehensive_trading_workflows.rs     ✅ REAL
└── integration_test.rs                    ✅ REAL

Test Count: ~80 E2E tests (all using real implementations)

---

Success Criteria

Before Migration (Current State)

• ❌ 14 files with "mock" references
• ❌ 6 files with "Mock" class usage
• ❌ 4 files with "stub" references
• ❌ MLPipelineTestHarness uses mock predictions
• ❌ Paper trading tests use mock executors
• ❌ Backtesting tests use mock engines

After Migration (Target State)

• ✅ ZERO mock/stub references in E2E tests
• ✅ MLPipelineTestHarness uses real ML inference
• ✅ Paper trading tests use real PaperTradingExecutor
• ✅ Backtesting tests use real BacktestingService gRPC
• ✅ All tests use real DBN data
• ✅ All tests use real feature extraction
• ✅ All tests verify actual integration
• ✅ Tests pass with production components

---

Verification Commands

# 1. Verify no mocks remain
grep -r "mock" tests/e2e/src/ tests/e2e/tests/
grep -r "Mock" tests/e2e/src/ tests/e2e/tests/
grep -r "stub" tests/e2e/src/ tests/e2e/tests/

# Expected: Zero matches

# 2. Run E2E tests with real implementations
cargo test -p e2e --test e2e_ml_paper_trading_test -- --test-threads=1
cargo test -p e2e --test e2e_ml_backtesting_test -- --test-threads=1
cargo test -p e2e --test ml_inference_e2e -- --test-threads=1

# Expected: All pass with real components

# 3. Verify database integration
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt \
  -c "SELECT COUNT(*) FROM ml_predictions WHERE confidence >= 0.6;"
# Expected: Real predictions stored

# 4. Verify gRPC integration
grpc_health_probe -addr=localhost:50051  # API Gateway
grpc_health_probe -addr=localhost:50052  # Trading Service
grpc_health_probe -addr=localhost:50053  # Backtesting Service
# Expected: All healthy

# 5. Full E2E test suite
cargo test --workspace --test '*e2e*' -- --test-threads=1
# Expected: 80/80 E2E tests pass (100%)

---

Implementation Timeline

Total Effort: ~4-6 hours

Phase	Task	Duration	Status
1	Update MLPipelineTestHarness	2 hours	⏳ TODO
2	Update Paper Trading Tests	1.5 hours	⏳ TODO
3	Update Backtesting Tests	1 hour	⏳ TODO
4	Remove Mock Infrastructure	0.5 hours	⏳ TODO
5	Verification & Testing	1 hour	⏳ TODO

---

Risk Mitigation

Potential Issues

1. Model Checkpoint Availability:

   • Risk: Tests may fail if trained models not available
   • Mitigation: Use fallback to latest checkpoints, skip gracefully if missing

2. GPU Availability:

   • Risk: CI/CD may not have GPU access
   • Mitigation: Use Device::cuda_if_available(0) - auto-falls back to CPU

3. Service Dependencies:

   • Risk: Tests require all services running
   • Mitigation: Use E2ETestFramework::start_services() - handles orchestration

4. Database State:

   • Risk: Tests may interfere with each other
   • Mitigation: Use transactions, rollback after each test

---

Post-Migration Checklist

• Zero "mock" references in tests/e2e/
• Zero "stub" references in tests/e2e/
• MLPipelineTestHarness uses real ML
• Paper trading tests use real executor
• Backtesting tests use real service
• All tests use real DBN data
• All tests use real features
• 80/80 E2E tests pass
• gRPC integration verified
• Database integration verified
• Documentation updated

---

References

Real Implementations:

• ml/src/inference.rs - RealMLInferenceEngine
• ml/src/ensemble/coordinator.rs - EnsembleCoordinator
• ml/src/ensemble/adaptive_ml_integration.rs - AdaptiveMLEnsemble
• services/trading_service/src/paper_trading_executor.rs - PaperTradingExecutor
• data/src/dbn_data_source.rs - DbnDataSource
• ml/src/features/extraction.rs - extract_256_dim_features()

Test Data:

• test_data/ES.FUT.20240102.dbn - 1,674 bars
• test_data/ZN.FUT.dbn - 28,935 bars
• test_data/6E.FUT.dbn - 29,937 bars

Documentation:

• CLAUDE.md - System architecture (100% production ready)
• ML_TRAINING_ROADMAP.md - 4-6 week training plan
• ML_DATA_VALIDATION_REPORT.md - Real data validation

---

Status: ⏳ READY FOR IMPLEMENTATION

Next Action: Execute Phase 1 - Update MLPipelineTestHarness with real ML inference.