foxhunt/AGENT_11.9_QUICK_REFERENCE.md

Agent 11.9: E2E Real Implementations - Quick Reference

Mission: Replace all mock/stub implementations in E2E tests with real production components.

---

🎯 Current State

Mocks to Replace

Location	Mock Component	Real Replacement
tests/e2e/src/ml_pipeline.rs	mock_prediction()	RealMLInferenceEngine
tests/e2e/tests/e2e_ml_paper_trading_test.rs	MockMLInferenceEngine	RealMLInferenceEngine
tests/e2e/tests/e2e_ml_paper_trading_test.rs	MockPaperTradingExecutor	PaperTradingExecutor
tests/e2e/tests/e2e_ml_backtesting_test.rs	MockBacktestingEngine	BacktestingServiceClient (gRPC)
tests/e2e/src/mocks/mod.rs	Entire module	DELETE (use real data)

Total Files with Mocks: 14 files

---

🔧 Real Implementations Available

ML Components

// Real ML inference engine
use ml::inference::{RealMLInferenceEngine, RealInferenceConfig};

// Real ensemble coordination
use ml::ensemble::{EnsembleCoordinator, AdaptiveMLEnsemble};

// Real feature extraction
use ml::features::extraction::extract_256_dim_features;

Trading Service

// Real paper trading executor
use services::trading_service::PaperTradingExecutor;

// Shared ML strategy
use common::ml_strategy::SharedMLStrategy;

Data Sources

// Real DBN market data
use data::DbnDataSource;

// Available data:
// - ES.FUT: 1,674 bars
// - ZN.FUT: 28,935 bars
// - 6E.FUT: 29,937 bars

---

📋 Implementation Phases

Phase 1: MLPipelineTestHarness (2 hours)

File: tests/e2e/src/ml_pipeline.rs

Changes:

• ❌ Remove mock_prediction() method
• ❌ Remove real_prediction() fallback
• ✅ Add real_ml_engine: Arc<RealMLInferenceEngine>
• ✅ Add ensemble_coordinator: Arc<EnsembleCoordinator>
• ✅ Add dbn_data_source: Arc<DbnDataSource>
• ✅ Use real inference in predict_with_model()
• ✅ Use real ensemble in predict_ensemble()

Phase 2: Paper Trading Tests (1.5 hours)

File: tests/e2e/tests/e2e_ml_paper_trading_test.rs

Changes:

• ❌ Delete MockMLInferenceEngine struct
• ❌ Delete MockPaperTradingExecutor struct
• ✅ Use real RealMLInferenceEngine
• ✅ Use real PaperTradingExecutor
• ✅ Use real DBN data via DbnDataSource
• ✅ Use real feature extraction

Phase 3: Backtesting Tests (1 hour)

File: tests/e2e/tests/e2e_ml_backtesting_test.rs

Changes:

• ❌ Delete MockBacktestingEngine struct
• ✅ Use real BacktestingServiceClient (gRPC)
• ✅ Use E2ETestFramework::get_backtesting_client()
• ✅ Verify real database persistence

Phase 4: Remove Mocks (0.5 hours)

Actions:

• ❌ DELETE tests/e2e/src/mocks/mod.rs
• ❌ DELETE tests/e2e/src/mocks/dual_provider_mocks.rs
• ✅ Update tests/e2e/src/lib.rs - remove pub mod mocks;

---

✅ Verification

Zero Mock References

grep -r "mock" tests/e2e/src/ tests/e2e/tests/
grep -r "Mock" tests/e2e/src/ tests/e2e/tests/
grep -r "stub" tests/e2e/src/ tests/e2e/tests/
# Expected: Zero matches

E2E Tests Pass

cargo test -p e2e --test e2e_ml_paper_trading_test
cargo test -p e2e --test e2e_ml_backtesting_test
cargo test -p e2e --test ml_inference_e2e
# Expected: All pass with real implementations

Real Data Integration

# Verify DBN data loaded
cargo run -p data --example validate_cl_fut

# Verify ML predictions stored
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt \
  -c "SELECT COUNT(*) FROM ml_predictions WHERE confidence >= 0.6;"

Service Health

grpc_health_probe -addr=localhost:50051  # API Gateway ✅
grpc_health_probe -addr=localhost:50052  # Trading Service ✅
grpc_health_probe -addr=localhost:50053  # Backtesting Service ✅

---

🚀 Quick Start

1. Update MLPipelineTestHarness

// tests/e2e/src/ml_pipeline.rs

use ml::inference::{RealMLInferenceEngine, RealInferenceConfig};
use ml::ensemble::EnsembleCoordinator;
use data::DbnDataSource;

pub struct MLPipelineTestHarness {
    real_ml_engine: Arc<RealMLInferenceEngine>,
    ensemble_coordinator: Arc<EnsembleCoordinator>,
    dbn_data_source: Arc<DbnDataSource>,
    model_metrics: HashMap<String, ModelMetrics>,
    feature_cache: HashMap<String, Vec<FeatureVector>>,
}

impl MLPipelineTestHarness {
    pub async fn new() -> Result<Self> {
        // Real ML engine
        let config = RealInferenceConfig::default();
        let real_ml_engine = Arc::new(RealMLInferenceEngine::new(config).await?);

        // Real ensemble
        let ensemble_coordinator = Arc::new(EnsembleCoordinator::new());
        ensemble_coordinator.register_model("DQN".to_string(), 0.25).await?;
        ensemble_coordinator.register_model("PPO".to_string(), 0.25).await?;
        ensemble_coordinator.register_model("MAMBA2".to_string(), 0.25).await?;
        ensemble_coordinator.register_model("TFT".to_string(), 0.25).await?;

        // Real data source
        let test_data_dir = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
            .parent().unwrap()
            .parent().unwrap()
            .join("test_data");
        let dbn_data_source = Arc::new(DbnDataSource::new(test_data_dir).await?);

        Ok(Self {
            real_ml_engine,
            ensemble_coordinator,
            dbn_data_source,
            model_metrics: HashMap::new(),
            feature_cache: HashMap::new(),
        })
    }
}

2. Update Paper Trading Test

// tests/e2e/tests/e2e_ml_paper_trading_test.rs

use ml::inference::RealMLInferenceEngine;
use services::trading_service::PaperTradingExecutor;
use common::ml_strategy::SharedMLStrategy;
use data::DbnDataSource;

#[tokio::test]
async fn test_e2e_checkpoint_to_order() -> Result<()> {
    // Real database
    let pool = get_test_db_pool().await;

    // Real ML engine
    let ml_config = RealInferenceConfig::default();
    let ml_engine = RealMLInferenceEngine::new(ml_config).await?;

    // Real executor
    let shared_strategy = SharedMLStrategy::new(Arc::new(ml_engine));
    let mut executor = PaperTradingExecutor::new_with_ml(
        pool.clone(),
        shared_strategy,
    ).await?;

    // Real data
    let dbn_source = DbnDataSource::new(PathBuf::from("test_data")).await?;
    let bars = dbn_source.load_ohlcv_bars("ES.FUT").await?;
    let features = extract_256_dim_features(&bars)?;

    // Real ML signal
    let signal = executor.generate_ml_signal(&features).await?;

    // Real order execution
    let order = executor.execute_ml_signal(&signal, "ES.FUT").await?;

    // Verify in database
    assert!(order.id != Uuid::nil());
    Ok(())
}

3. Update Backtesting Test

// tests/e2e/tests/e2e_ml_backtesting_test.rs

#[tokio::test]
async fn test_e2e_checkpoint_to_backtest_metrics() -> Result<()> {
    // Real framework
    let mut framework = E2ETestFramework::new().await?;
    framework.start_services().await?;

    // Real backtesting client (gRPC)
    let client = framework.get_backtesting_client().await?;

    // Real backtest request
    let request = tonic::Request::new(BacktestRequest {
        strategy: "MLEnsemble".to_string(),
        symbol: "ES.FUT".to_string(),
        start_date: "2024-01-02".to_string(),
        end_date: "2024-01-10".to_string(),
        initial_capital: 100000.0,
        ml_config: Some(MlConfig {
            models: vec!["DQN", "PPO", "MAMBA2", "TFT"],
            confidence_threshold: 0.6,
            ensemble_method: "weighted_voting",
        }),
    });

    // Real service execution
    let response = client.run_backtest(request).await?;
    let results = response.into_inner();

    // Verify real metrics
    assert!(results.total_trades > 0);
    assert!(results.sharpe_ratio > 1.5);

    framework.stop_services().await?;
    Ok(())
}

---

📊 Success Metrics

Before (Current)

• ❌ 14 files with mock references
• ❌ MLPipelineTestHarness uses mocks
• ❌ Paper trading tests use mocks
• ❌ Backtesting tests use mocks

After (Target)

• ✅ ZERO mock references
• ✅ All tests use real implementations
• ✅ 80/80 E2E tests pass (100%)
• ✅ Real ML inference verified
• ✅ Real database integration verified
• ✅ Real gRPC integration verified

---

⏱️ Timeline

Total: ~6 hours

• Phase 1: 2 hours
• Phase 2: 1.5 hours
• Phase 3: 1 hour
• Phase 4: 0.5 hours
• Verification: 1 hour

---

📚 Key Files

To Modify

1. tests/e2e/src/ml_pipeline.rs - Update to real ML
2. tests/e2e/tests/e2e_ml_paper_trading_test.rs - Use real executor
3. tests/e2e/tests/e2e_ml_backtesting_test.rs - Use real service
4. tests/e2e/src/lib.rs - Remove mocks module

To Delete

1. tests/e2e/src/mocks/mod.rs
2. tests/e2e/src/mocks/dual_provider_mocks.rs

Real Implementations

1. ml/src/inference.rs - RealMLInferenceEngine
2. ml/src/ensemble/coordinator.rs - EnsembleCoordinator
3. services/trading_service/src/paper_trading_executor.rs - PaperTradingExecutor
4. data/src/dbn_data_source.rs - DbnDataSource

---

Status: ⏳ READY FOR IMPLEMENTATION

Next Step: Execute Phase 1 - Update MLPipelineTestHarness