foxhunt/TESTING_PLAN.md

Foxhunt ML/AI Testing Plan: Realistic Crypto Data Integration

Last Updated: 2025-10-06 Status: Ready for Implementation Goal: Validate ML/AI models with realistic crypto tick data using EXISTING infrastructure

---

Executive Summary

Foxhunt has 90% of required testing infrastructure already implemented. This plan leverages existing Parquet replay, backtesting service, and feature engineering to validate ML models with realistic cryptocurrency tick data.

Key Principle: REUSE existing infrastructure, DO NOT rebuild.

---

Existing Infrastructure (REUSE These)

1. Data Replay System ✅

Location: data/src/parquet_persistence.rs

• ParquetMarketDataWriter: Production-ready async batching

  • 10K events/batch, 5s flush interval
  • SNAPPY compression with dictionary encoding
  • Schema: timestamp_ns, symbol, venue, event_type, price, quantity, sequence, latency_ns

• ParquetMarketDataReader: INCOMPLETE (needs implementation)

  • Currently returns empty Vec
  • Required for test data replay

2. Backtesting Service ✅

Location: services/backtesting_service/

• gRPC service on port 50053
• StrategyEngine with model versioning
• PerformanceAnalyzer (Sharpe, drawdown, PnL)
• Real-time progress streaming
• DATABASE: postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt

3. Feature Engineering Pipeline ✅

Location: data/src/training_pipeline.rs

• TechnicalIndicatorsCalculator: MA (5,10,20,50,200), RSI, Bollinger, MACD
• MicrostructureAnalyzer: Bid-ask spread, volume imbalance, price impact
• TLOBProcessor: 20-level order book reconstruction
• RegimeDetector: Volatility regime classification

4. ML Models ✅

Locations:

• ml/src/dqn/: DQN agent with experience replay
• ml/src/mamba/: MAMBA-2 state space model
• ml/src/tft/: Temporal Fusion Transformer
• ml/src/liquid/: Liquid neural networks
• ml/src/ppo/: Proximal Policy Optimization

---

Required Additions (3 Components)

Addition 1: Complete ParquetMarketDataReader (2-4 hours)

File: data/src/parquet_persistence.rs:268

Current State:

pub async fn read_file(&self, filename: &str) -> Result<Vec<MarketDataEvent>> {
    warn!("Parquet reader not fully implemented yet");
    Ok(Vec::new())  // ❌ PLACEHOLDER
}

Implementation:

use parquet::arrow::async_reader::ParquetObjectReader;
use parquet::arrow::ParquetRecordBatchReaderBuilder;
use arrow::array::{TimestampNanosecondType, StringArray, Float64Array, UInt64Array};

pub async fn read_file(&self, filename: &str) -> Result<Vec<MarketDataEvent>> {
    let filepath = Path::new(&self.base_path).join(filename);
    let file = tokio::fs::File::open(filepath).await?;

    let builder = ParquetRecordBatchReaderBuilder::try_new(file).await?;
    let mut reader = builder.build()?;

    let mut events = Vec::new();
    while let Some(Ok(batch)) = reader.next() {
        // Extract 8 columns from Arrow batch
        let timestamps = batch.column(0).as_primitive::<TimestampNanosecondType>();
        let symbols = batch.column(1).as_string::<i32>();
        let venues = batch.column(2).as_string::<i32>();
        let event_types = batch.column(3).as_string::<i32>();
        let prices = batch.column(4).as_primitive::<Float64Array>();
        let quantities = batch.column(5).as_primitive::<Float64Array>();
        let sequences = batch.column(6).as_primitive::<UInt64Array>();
        let latencies = batch.column(7).as_primitive::<UInt64Array>();

        for i in 0..batch.num_rows() {
            events.push(MarketDataEvent {
                timestamp_ns: timestamps.value(i) as u64,
                symbol: symbols.value(i).to_string(),
                venue: venues.value(i).to_string(),
                event_type: event_types.value(i).to_string(),
                price: prices.value(i),
                quantity: quantities.value(i),
                sequence: sequences.value(i),
                latency_ns: Some(latencies.value(i)),
            });
        }
    }
    Ok(events)
}

Test:

#[tokio::test]
async fn test_parquet_roundtrip() {
    let writer = ParquetMarketDataWriter::new(...);
    writer.write(test_events).await?;

    let reader = ParquetMarketDataReader::new(...);
    let read_events = reader.read_file("test.parquet").await?;

    assert_eq!(test_events, read_events);
}

---

Addition 2: Binance WebSocket Client (4-6 hours)

New File: data/src/providers/binance_crypto.rs

Purpose: Free, unlimited crypto tick data from Binance WebSocket API

Implementation:

use tokio_tungstenite::{connect_async, tungstenite::Message};
use serde::{Deserialize, Serialize};

#[derive(Debug, Deserialize)]
struct BinanceTrade {
    #[serde(rename = "E")]
    event_time: u64,  // Milliseconds
    #[serde(rename = "s")]
    symbol: String,   // "BTCUSDT"
    #[serde(rename = "p")]
    price: String,    // "43250.50"
    #[serde(rename = "q")]
    quantity: String, // "0.05"
    #[serde(rename = "T")]
    trade_time: u64,  // Milliseconds
}

pub struct BinanceCryptoClient {
    writer: Arc<ParquetMarketDataWriter>,
    symbols: Vec<String>,
}

impl BinanceCryptoClient {
    pub async fn start_streaming(&self) -> Result<()> {
        let stream = format!(
            "wss://stream.binance.com:9443/stream?streams={}",
            self.symbols.iter()
                .map(|s| format!("{}@trade", s.to_lowercase()))
                .collect::<Vec<_>>()
                .join("/")
        );

        let (ws_stream, _) = connect_async(stream).await?;
        let (_, mut read) = ws_stream.split();

        while let Some(msg) = read.next().await {
            match msg? {
                Message::Text(text) => {
                    let trade: BinanceTrade = serde_json::from_str(&text)?;
                    let event = self.convert_to_market_event(trade);
                    self.writer.write_event(event).await?;
                }
                _ => {}
            }
        }
        Ok(())
    }

    fn convert_to_market_event(&self, trade: BinanceTrade) -> MarketDataEvent {
        MarketDataEvent {
            timestamp_ns: trade.trade_time * 1_000_000,
            symbol: trade.symbol,
            venue: "Binance".to_string(),
            event_type: "trade".to_string(),
            price: trade.price.parse().unwrap_or(0.0),
            quantity: trade.quantity.parse().unwrap_or(0.0),
            sequence: 0,
            latency_ns: None,
        }
    }
}

Usage:

# Generate 1-hour test dataset
cargo run --bin crypto_data_collector -- \
  --symbols BTCUSDT,ETHUSDT,SOLUSDT \
  --duration 3600 \
  --output test_data/crypto/multi_1h.parquet

---

Addition 3: Test Datasets (2-3 hours)

Location: test_data/crypto/

Datasets to Generate:

1. btc_1h.parquet - 1 hour BTC/USDT ticks (~50MB, 100K-500K trades)
2. eth_1h.parquet - 1 hour ETH/USDT ticks (~30MB)
3. sol_1h.parquet - 1 hour SOL/USDT ticks (~20MB)
4. multi_regime_1wk.parquet - 1 week multi-asset for regime testing (~2GB)

Generation Script:

#!/bin/bash
# Generate all test datasets

# 1-hour datasets (fast tests)
cargo run --bin crypto_data_collector -- \
  --symbols BTCUSDT --duration 3600 \
  --output test_data/crypto/btc_1h.parquet

cargo run --bin crypto_data_collector -- \
  --symbols ETHUSDT --duration 3600 \
  --output test_data/crypto/eth_1h.parquet

cargo run --bin crypto_data_collector -- \
  --symbols SOLUSDT --duration 3600 \
  --output test_data/crypto/sol_1h.parquet

# 1-week dataset (comprehensive testing)
cargo run --bin crypto_data_collector -- \
  --symbols BTCUSDT,ETHUSDT,SOLUSDT \
  --duration 604800 \
  --output test_data/crypto/multi_regime_1wk.parquet

---

4-Tier Testing Strategy

Tier 1: Unit Tests (Mocks) - 30 minutes runtime

Purpose: Fast, deterministic validation of ML logic

// ml/tests/dqn_unit_tests.rs
#[test]
fn test_dqn_action_selection() {
    let mock_state = create_mock_market_state();
    let agent = DQNAgent::new(config);
    let action = agent.select_action(&mock_state);
    assert!(action.is_valid());
}

Coverage Target: 60-70% of ML model code Database: None required (pure mocks)

---

Tier 2: Integration Tests (Parquet Replay) - 1 hour dataset

Purpose: Realistic validation with recorded crypto data

// ml/tests/realistic_crypto_tests.rs
#[tokio::test]
async fn test_dqn_convergence_with_crypto_replay() {
    // REUSE ParquetMarketDataReader
    let reader = ParquetMarketDataReader::new("test_data/crypto");
    let events = reader.read_file("btc_1h.parquet").await?;

    let mut agent = DQNAgent::new(config);
    let mut total_reward = 0.0;

    for event in events {
        let reward = agent.step(&event);
        total_reward += reward;
    }

    // Validate convergence
    assert!(total_reward > 0.0, "DQN failed to learn");
}

#[tokio::test]
async fn test_mamba2_order_book_prediction() {
    let reader = ParquetMarketDataReader::new("test_data/crypto");
    let events = reader.read_file("eth_1h.parquet").await?;

    // REUSE TLOBProcessor
    let model = MAMBA2Model::load("models/mamba2_ob_predictor.safetensors")?;
    let mut correct_predictions = 0;

    for window in events.windows(50) {
        let ob_state = reconstruct_order_book(window);
        let prediction = model.predict_mid_price_change(&ob_state);
        let actual = calculate_actual_change(window);

        if prediction.signum() == actual.signum() {
            correct_predictions += 1;
        }
    }

    let accuracy = correct_predictions as f64 / total;
    assert!(accuracy > 0.55, "MAMBA-2 accuracy {:.2}% < 55%", accuracy * 100.0);
}

Coverage Target: 75-85% with realistic data patterns Database: PostgreSQL for feature storage (REUSE docker-compose.yml) Connection: postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt

---

Tier 3: Multi-Regime Backtesting - 1 week dataset

Purpose: Validate across different market conditions

REUSE Existing BacktestingService:

// Integration with existing gRPC service
#[tokio::test]
async fn test_backtest_crypto_multi_regime() {
    // REUSE BacktestingService via gRPC
    let mut client = BacktestingServiceClient::connect("http://localhost:50053").await?;

    let request = StartBacktestRequest {
        strategy_id: "dqn_crypto_v1".to_string(),
        data_source: "parquet://test_data/crypto/multi_regime_1wk.parquet".to_string(),
        config: json!({
            "initial_capital": 100000.0,
            "symbols": ["BTCUSDT", "ETHUSDT", "SOLUSDT"],
        }),
    };

    let response = client.start_backtest(request).await?;

    // Monitor progress
    let mut stream = client.subscribe_progress(response.backtest_id).await?;
    while let Some(event) = stream.next().await {
        println!("Progress: {:.1}%", event.progress);
    }

    // Validate results
    let results = client.get_results(response.backtest_id).await?;
    assert!(results.sharpe_ratio > 1.0);
    assert!(results.max_drawdown < 0.15);
}

Test Scenarios:

1. High Volatility: Crypto crash period - circuit breakers
2. Low Volatility: Sideways consolidation - regime detection
3. Trending: Bull run - momentum strategies

Coverage Target: 90%+ with stress scenarios Database: REUSE PostgreSQL for backtest results

---

Tier 4: Live Simulation - Future Work

Purpose: Paper trading with real WebSocket feeds

REUSE Infrastructure:

• Binance WebSocket → ParquetWriter → ML Models
• API Gateway for auth
• Trading Service for position management
• Redis for real-time caching

---

Infrastructure Setup

Quick Start

# 1. Start all infrastructure (REUSE docker-compose.yml)
docker-compose up -d

# Verify services
docker-compose ps

# 2. Run migrations (REUSE existing migrations)
cargo sqlx migrate run

# 3. Verify PostgreSQL
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c '\dt'

Database Credentials (from docker-compose.yml)

PostgreSQL (TimescaleDB):

Host: localhost:5432
Database: foxhunt
User: foxhunt
Password: foxhunt_dev_password
URL: postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt

Redis:

Host: localhost:6379
URL: redis://localhost:6379

InfluxDB:

Host: localhost:8086
User: foxhunt
Password: foxhunt_dev_password
Org: foxhunt
Bucket: trading_metrics

Vault:

Host: localhost:8200
Token: foxhunt-dev-root
URL: http://localhost:8200

Grafana:

URL: http://localhost:3000
Username: admin
Password: foxhunt123

Prometheus:

URL: http://localhost:9090

---

Implementation Timeline

Phase 1: Core Replay (Week 1)

Goal: Enable Parquet-based test data replay

• Day 1-2: Complete ParquetMarketDataReader (2-4 hours)
• Day 3: Write roundtrip tests (2 hours)
• Day 4: Generate 1-hour test datasets (2 hours)
• Day 5: Tier 2 integration tests (4 hours)

Deliverables:

• ✅ ParquetReader implemented
• ✅ 3 × 1-hour crypto datasets
• ✅ 5 integration tests passing

---

Phase 2: Data Collection (Week 2)

Goal: Continuous crypto data collection

• Day 1-2: Implement BinanceCryptoClient (4-6 hours)
• Day 3: Generate 1-week dataset (overnight run)
• Day 4: Multi-regime backtesting tests (4 hours)
• Day 5: Validation and metrics (2 hours)

Deliverables:

• ✅ Binance WebSocket client
• ✅ 1-week multi-asset dataset
• ✅ Tier 3 backtesting tests

---

Phase 3: ML Validation (Week 3)

Goal: Comprehensive ML model testing

• Day 1: DQN convergence tests (4 hours)
• Day 2: MAMBA-2 prediction accuracy (4 hours)
• Day 3: TFT forecasting tests (4 hours)
• Day 4: Liquid network tests (4 hours)
• Day 5: Performance benchmarks (4 hours)

Deliverables:

• ✅ 15+ ML integration tests
• ✅ Coverage >75% with realistic data
• ✅ Performance benchmarks

---

Success Criteria

Code Coverage

• Tier 1 (Mocks): 60-70% ML code coverage
• Tier 2 (Replay): 75-85% with realistic patterns
• Tier 3 (Backtesting): 90%+ with stress scenarios

Model Validation

• DQN: Positive cumulative reward after 1-hour replay
• MAMBA-2: >55% directional prediction accuracy
• TFT: <5% MAPE on price forecasting
• Liquid: Convergence within 10K training steps

Performance

• Tier 1: <1 minute test suite runtime
• Tier 2: <5 minutes per 1-hour dataset
• Tier 3: <30 minutes per 1-week backtest

---

Maintenance

Dataset Refresh

# Monthly: Regenerate test datasets with latest crypto data
./scripts/refresh_test_data.sh

# Stores in test_data/crypto/YYYY-MM/

Coverage Monitoring

# Weekly: Measure ML test coverage
cargo llvm-cov --html --output-dir coverage_ml -p ml

# Target: Maintain >75% coverage

Infrastructure Health

# Daily: Verify Docker services
docker-compose ps
docker-compose logs -f backtesting_service

# Weekly: Check PostgreSQL migrations
cargo sqlx migrate info

---

Anti-Patterns (DO NOT DO)

❌ DO NOT create new Python-based testing frameworks ❌ DO NOT rebuild Parquet infrastructure ❌ DO NOT create new database schemas ❌ DO NOT spin up new Docker services ❌ DO NOT use CSV files instead of Parquet ❌ DO NOT mock all data (need realistic patterns)

✅ DO reuse ParquetMarketDataWriter/Reader ✅ DO use existing BacktestingService gRPC API ✅ DO leverage FeatureProcessor pipeline ✅ DO connect to PostgreSQL via docker-compose credentials ✅ DO use Binance free WebSocket data

---

References

Code Locations

• Parquet Replay: data/src/parquet_persistence.rs
• Backtesting Service: services/backtesting_service/src/service.rs
• Feature Engineering: data/src/training_pipeline.rs
• ML Models: ml/src/{dqn,mamba,tft,liquid,ppo}/

Documentation

• Architecture: CLAUDE.md
• Docker Setup: docker-compose.yml
• Migrations: migrations/*.sql
• Wave Reports: WAVE114_FINAL_REPORT.md

External APIs

• Binance WebSocket: wss://stream.binance.com:9443/ws/{symbol}@trade
• Binance Docs: https://binance-docs.github.io/apidocs/spot/en/#websocket-market-streams

---

Last Updated: 2025-10-06 Status: Ready for Phase 1 Implementation Next Step: Complete ParquetMarketDataReader (2-4 hours)