# Foxhunt ML/AI Testing Plan: Realistic Crypto Data Integration

**Last Updated**: 2025-10-06
**Status**: Ready for Implementation
**Goal**: Validate ML/AI models with realistic crypto tick data using EXISTING infrastructure

---

## Executive Summary

Foxhunt has **90% of required testing infrastructure already implemented**. This plan leverages existing Parquet replay, backtesting service, and feature engineering to validate ML models with realistic cryptocurrency tick data.

**Key Principle**: **REUSE existing infrastructure, DO NOT rebuild.**

---

## Existing Infrastructure (REUSE These)

### 1. Data Replay System ✅
**Location**: `data/src/parquet_persistence.rs`

- **ParquetMarketDataWriter**: Production-ready async batching
  - 10K events/batch, 5s flush interval
  - SNAPPY compression with dictionary encoding
  - Schema: timestamp_ns, symbol, venue, event_type, price, quantity, sequence, latency_ns

- **ParquetMarketDataReader**: **INCOMPLETE** (needs implementation)
  - Currently returns empty Vec
  - Required for test data replay

### 2. Backtesting Service ✅
**Location**: `services/backtesting_service/`

- gRPC service on port 50053
- StrategyEngine with model versioning
- PerformanceAnalyzer (Sharpe, drawdown, PnL)
- Real-time progress streaming
- **DATABASE**: `postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt`

### 3. Feature Engineering Pipeline ✅
**Location**: `data/src/training_pipeline.rs`

- **TechnicalIndicatorsCalculator**: MA (5,10,20,50,200), RSI, Bollinger, MACD
- **MicrostructureAnalyzer**: Bid-ask spread, volume imbalance, price impact
- **TLOBProcessor**: 20-level order book reconstruction
- **RegimeDetector**: Volatility regime classification

### 4. ML Models ✅
**Locations**:
- `ml/src/dqn/`: DQN agent with experience replay
- `ml/src/mamba/`: MAMBA-2 state space model
- `ml/src/tft/`: Temporal Fusion Transformer
- `ml/src/liquid/`: Liquid neural networks
- `ml/src/ppo/`: Proximal Policy Optimization

---

## Required Additions (3 Components)

### Addition 1: Complete ParquetMarketDataReader (2-4 hours)

**File**: `data/src/parquet_persistence.rs:268`

**Current State**:
```rust
pub async fn read_file(&self, filename: &str) -> Result<Vec<MarketDataEvent>> {
    warn!("Parquet reader not fully implemented yet");
    Ok(Vec::new())  // ❌ PLACEHOLDER
}
```

**Implementation**:
```rust
use parquet::arrow::async_reader::ParquetObjectReader;
use parquet::arrow::ParquetRecordBatchReaderBuilder;
use arrow::array::{TimestampNanosecondType, StringArray, Float64Array, UInt64Array};

pub async fn read_file(&self, filename: &str) -> Result<Vec<MarketDataEvent>> {
    let filepath = Path::new(&self.base_path).join(filename);
    let file = tokio::fs::File::open(filepath).await?;

    let builder = ParquetRecordBatchReaderBuilder::try_new(file).await?;
    let mut reader = builder.build()?;

    let mut events = Vec::new();
    while let Some(Ok(batch)) = reader.next() {
        // Extract 8 columns from Arrow batch
        let timestamps = batch.column(0).as_primitive::<TimestampNanosecondType>();
        let symbols = batch.column(1).as_string::<i32>();
        let venues = batch.column(2).as_string::<i32>();
        let event_types = batch.column(3).as_string::<i32>();
        let prices = batch.column(4).as_primitive::<Float64Array>();
        let quantities = batch.column(5).as_primitive::<Float64Array>();
        let sequences = batch.column(6).as_primitive::<UInt64Array>();
        let latencies = batch.column(7).as_primitive::<UInt64Array>();

        for i in 0..batch.num_rows() {
            events.push(MarketDataEvent {
                timestamp_ns: timestamps.value(i) as u64,
                symbol: symbols.value(i).to_string(),
                venue: venues.value(i).to_string(),
                event_type: event_types.value(i).to_string(),
                price: prices.value(i),
                quantity: quantities.value(i),
                sequence: sequences.value(i),
                latency_ns: Some(latencies.value(i)),
            });
        }
    }
    Ok(events)
}
```

**Test**:
```rust
#[tokio::test]
async fn test_parquet_roundtrip() {
    let writer = ParquetMarketDataWriter::new(...);
    writer.write(test_events).await?;

    let reader = ParquetMarketDataReader::new(...);
    let read_events = reader.read_file("test.parquet").await?;

    assert_eq!(test_events, read_events);
}
```

---

### Addition 2: Binance WebSocket Client (4-6 hours)

**New File**: `data/src/providers/binance_crypto.rs`

**Purpose**: Free, unlimited crypto tick data from Binance WebSocket API

**Implementation**:
```rust
use tokio_tungstenite::{connect_async, tungstenite::Message};
use serde::{Deserialize, Serialize};

#[derive(Debug, Deserialize)]
struct BinanceTrade {
    #[serde(rename = "E")]
    event_time: u64,  // Milliseconds
    #[serde(rename = "s")]
    symbol: String,   // "BTCUSDT"
    #[serde(rename = "p")]
    price: String,    // "43250.50"
    #[serde(rename = "q")]
    quantity: String, // "0.05"
    #[serde(rename = "T")]
    trade_time: u64,  // Milliseconds
}

pub struct BinanceCryptoClient {
    writer: Arc<ParquetMarketDataWriter>,
    symbols: Vec<String>,
}

impl BinanceCryptoClient {
    pub async fn start_streaming(&self) -> Result<()> {
        let stream = format!(
            "wss://stream.binance.com:9443/stream?streams={}",
            self.symbols.iter()
                .map(|s| format!("{}@trade", s.to_lowercase()))
                .collect::<Vec<_>>()
                .join("/")
        );

        let (ws_stream, _) = connect_async(stream).await?;
        let (_, mut read) = ws_stream.split();

        while let Some(msg) = read.next().await {
            match msg? {
                Message::Text(text) => {
                    let trade: BinanceTrade = serde_json::from_str(&text)?;
                    let event = self.convert_to_market_event(trade);
                    self.writer.write_event(event).await?;
                }
                _ => {}
            }
        }
        Ok(())
    }

    fn convert_to_market_event(&self, trade: BinanceTrade) -> MarketDataEvent {
        MarketDataEvent {
            timestamp_ns: trade.trade_time * 1_000_000,
            symbol: trade.symbol,
            venue: "Binance".to_string(),
            event_type: "trade".to_string(),
            price: trade.price.parse().unwrap_or(0.0),
            quantity: trade.quantity.parse().unwrap_or(0.0),
            sequence: 0,
            latency_ns: None,
        }
    }
}
```

**Usage**:
```bash
# Generate 1-hour test dataset
cargo run --bin crypto_data_collector -- \
  --symbols BTCUSDT,ETHUSDT,SOLUSDT \
  --duration 3600 \
  --output test_data/crypto/multi_1h.parquet
```

---

### Addition 3: Test Datasets (2-3 hours)

**Location**: `test_data/crypto/`

**Datasets to Generate**:
1. **btc_1h.parquet** - 1 hour BTC/USDT ticks (~50MB, 100K-500K trades)
2. **eth_1h.parquet** - 1 hour ETH/USDT ticks (~30MB)
3. **sol_1h.parquet** - 1 hour SOL/USDT ticks (~20MB)
4. **multi_regime_1wk.parquet** - 1 week multi-asset for regime testing (~2GB)

**Generation Script**:
```bash
#!/bin/bash
# Generate all test datasets

# 1-hour datasets (fast tests)
cargo run --bin crypto_data_collector -- \
  --symbols BTCUSDT --duration 3600 \
  --output test_data/crypto/btc_1h.parquet

cargo run --bin crypto_data_collector -- \
  --symbols ETHUSDT --duration 3600 \
  --output test_data/crypto/eth_1h.parquet

cargo run --bin crypto_data_collector -- \
  --symbols SOLUSDT --duration 3600 \
  --output test_data/crypto/sol_1h.parquet

# 1-week dataset (comprehensive testing)
cargo run --bin crypto_data_collector -- \
  --symbols BTCUSDT,ETHUSDT,SOLUSDT \
  --duration 604800 \
  --output test_data/crypto/multi_regime_1wk.parquet
```

---

## 4-Tier Testing Strategy

### Tier 1: Unit Tests (Mocks) - 30 minutes runtime
**Purpose**: Fast, deterministic validation of ML logic

```rust
// ml/tests/dqn_unit_tests.rs
#[test]
fn test_dqn_action_selection() {
    let mock_state = create_mock_market_state();
    let agent = DQNAgent::new(config);
    let action = agent.select_action(&mock_state);
    assert!(action.is_valid());
}
```

**Coverage Target**: 60-70% of ML model code
**Database**: None required (pure mocks)

---

### Tier 2: Integration Tests (Parquet Replay) - 1 hour dataset
**Purpose**: Realistic validation with recorded crypto data

```rust
// ml/tests/realistic_crypto_tests.rs
#[tokio::test]
async fn test_dqn_convergence_with_crypto_replay() {
    // REUSE ParquetMarketDataReader
    let reader = ParquetMarketDataReader::new("test_data/crypto");
    let events = reader.read_file("btc_1h.parquet").await?;

    let mut agent = DQNAgent::new(config);
    let mut total_reward = 0.0;

    for event in events {
        let reward = agent.step(&event);
        total_reward += reward;
    }

    // Validate convergence
    assert!(total_reward > 0.0, "DQN failed to learn");
}

#[tokio::test]
async fn test_mamba2_order_book_prediction() {
    let reader = ParquetMarketDataReader::new("test_data/crypto");
    let events = reader.read_file("eth_1h.parquet").await?;

    // REUSE TLOBProcessor
    let model = MAMBA2Model::load("models/mamba2_ob_predictor.safetensors")?;
    let mut correct_predictions = 0;

    for window in events.windows(50) {
        let ob_state = reconstruct_order_book(window);
        let prediction = model.predict_mid_price_change(&ob_state);
        let actual = calculate_actual_change(window);

        if prediction.signum() == actual.signum() {
            correct_predictions += 1;
        }
    }

    let accuracy = correct_predictions as f64 / total;
    assert!(accuracy > 0.55, "MAMBA-2 accuracy {:.2}% < 55%", accuracy * 100.0);
}
```

**Coverage Target**: 75-85% with realistic data patterns
**Database**: PostgreSQL for feature storage (REUSE docker-compose.yml)
**Connection**: `postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt`

---

### Tier 3: Multi-Regime Backtesting - 1 week dataset
**Purpose**: Validate across different market conditions

**REUSE Existing BacktestingService**:
```rust
// Integration with existing gRPC service
#[tokio::test]
async fn test_backtest_crypto_multi_regime() {
    // REUSE BacktestingService via gRPC
    let mut client = BacktestingServiceClient::connect("http://localhost:50053").await?;

    let request = StartBacktestRequest {
        strategy_id: "dqn_crypto_v1".to_string(),
        data_source: "parquet://test_data/crypto/multi_regime_1wk.parquet".to_string(),
        config: json!({
            "initial_capital": 100000.0,
            "symbols": ["BTCUSDT", "ETHUSDT", "SOLUSDT"],
        }),
    };

    let response = client.start_backtest(request).await?;

    // Monitor progress
    let mut stream = client.subscribe_progress(response.backtest_id).await?;
    while let Some(event) = stream.next().await {
        println!("Progress: {:.1}%", event.progress);
    }

    // Validate results
    let results = client.get_results(response.backtest_id).await?;
    assert!(results.sharpe_ratio > 1.0);
    assert!(results.max_drawdown < 0.15);
}
```

**Test Scenarios**:
1. **High Volatility**: Crypto crash period - circuit breakers
2. **Low Volatility**: Sideways consolidation - regime detection
3. **Trending**: Bull run - momentum strategies

**Coverage Target**: 90%+ with stress scenarios
**Database**: REUSE PostgreSQL for backtest results

---

### Tier 4: Live Simulation - Future Work
**Purpose**: Paper trading with real WebSocket feeds

**REUSE Infrastructure**:
- Binance WebSocket → ParquetWriter → ML Models
- API Gateway for auth
- Trading Service for position management
- Redis for real-time caching

---

## Infrastructure Setup

### Quick Start
```bash
# 1. Start all infrastructure (REUSE docker-compose.yml)
docker-compose up -d

# Verify services
docker-compose ps

# 2. Run migrations (REUSE existing migrations)
cargo sqlx migrate run

# 3. Verify PostgreSQL
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c '\dt'
```

### Database Credentials (from docker-compose.yml)

**PostgreSQL** (TimescaleDB):
```
Host: localhost:5432
Database: foxhunt
User: foxhunt
Password: foxhunt_dev_password
URL: postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt
```

**Redis**:
```
Host: localhost:6379
URL: redis://localhost:6379
```

**InfluxDB**:
```
Host: localhost:8086
User: foxhunt
Password: foxhunt_dev_password
Org: foxhunt
Bucket: trading_metrics
```

**Vault**:
```
Host: localhost:8200
Token: foxhunt-dev-root
URL: http://localhost:8200
```

**Grafana**:
```
URL: http://localhost:3000
Username: admin
Password: foxhunt123
```

**Prometheus**:
```
URL: http://localhost:9090
```

---

## Implementation Timeline

### Phase 1: Core Replay (Week 1)
**Goal**: Enable Parquet-based test data replay

- **Day 1-2**: Complete `ParquetMarketDataReader` (2-4 hours)
- **Day 3**: Write roundtrip tests (2 hours)
- **Day 4**: Generate 1-hour test datasets (2 hours)
- **Day 5**: Tier 2 integration tests (4 hours)

**Deliverables**:
- ✅ ParquetReader implemented
- ✅ 3 × 1-hour crypto datasets
- ✅ 5 integration tests passing

---

### Phase 2: Data Collection (Week 2)
**Goal**: Continuous crypto data collection

- **Day 1-2**: Implement `BinanceCryptoClient` (4-6 hours)
- **Day 3**: Generate 1-week dataset (overnight run)
- **Day 4**: Multi-regime backtesting tests (4 hours)
- **Day 5**: Validation and metrics (2 hours)

**Deliverables**:
- ✅ Binance WebSocket client
- ✅ 1-week multi-asset dataset
- ✅ Tier 3 backtesting tests

---

### Phase 3: ML Validation (Week 3)
**Goal**: Comprehensive ML model testing

- **Day 1**: DQN convergence tests (4 hours)
- **Day 2**: MAMBA-2 prediction accuracy (4 hours)
- **Day 3**: TFT forecasting tests (4 hours)
- **Day 4**: Liquid network tests (4 hours)
- **Day 5**: Performance benchmarks (4 hours)

**Deliverables**:
- ✅ 15+ ML integration tests
- ✅ Coverage >75% with realistic data
- ✅ Performance benchmarks

---

## Success Criteria

### Code Coverage
- **Tier 1 (Mocks)**: 60-70% ML code coverage
- **Tier 2 (Replay)**: 75-85% with realistic patterns
- **Tier 3 (Backtesting)**: 90%+ with stress scenarios

### Model Validation
- **DQN**: Positive cumulative reward after 1-hour replay
- **MAMBA-2**: >55% directional prediction accuracy
- **TFT**: <5% MAPE on price forecasting
- **Liquid**: Convergence within 10K training steps

### Performance
- **Tier 1**: <1 minute test suite runtime
- **Tier 2**: <5 minutes per 1-hour dataset
- **Tier 3**: <30 minutes per 1-week backtest

---

## Maintenance

### Dataset Refresh
```bash
# Monthly: Regenerate test datasets with latest crypto data
./scripts/refresh_test_data.sh

# Stores in test_data/crypto/YYYY-MM/
```

### Coverage Monitoring
```bash
# Weekly: Measure ML test coverage
cargo llvm-cov --html --output-dir coverage_ml -p ml

# Target: Maintain >75% coverage
```

### Infrastructure Health
```bash
# Daily: Verify Docker services
docker-compose ps
docker-compose logs -f backtesting_service

# Weekly: Check PostgreSQL migrations
cargo sqlx migrate info
```

---

## Anti-Patterns (DO NOT DO)

❌ **DO NOT** create new Python-based testing frameworks
❌ **DO NOT** rebuild Parquet infrastructure
❌ **DO NOT** create new database schemas
❌ **DO NOT** spin up new Docker services
❌ **DO NOT** use CSV files instead of Parquet
❌ **DO NOT** mock all data (need realistic patterns)

✅ **DO** reuse ParquetMarketDataWriter/Reader
✅ **DO** use existing BacktestingService gRPC API
✅ **DO** leverage FeatureProcessor pipeline
✅ **DO** connect to PostgreSQL via docker-compose credentials
✅ **DO** use Binance free WebSocket data

---

## References

### Code Locations
- Parquet Replay: `data/src/parquet_persistence.rs`
- Backtesting Service: `services/backtesting_service/src/service.rs`
- Feature Engineering: `data/src/training_pipeline.rs`
- ML Models: `ml/src/{dqn,mamba,tft,liquid,ppo}/`

### Documentation
- Architecture: `CLAUDE.md`
- Docker Setup: `docker-compose.yml`
- Migrations: `migrations/*.sql`
- Wave Reports: `WAVE114_FINAL_REPORT.md`

### External APIs
- Binance WebSocket: `wss://stream.binance.com:9443/ws/{symbol}@trade`
- Binance Docs: https://binance-docs.github.io/apidocs/spot/en/#websocket-market-streams

---

**Last Updated**: 2025-10-06
**Status**: Ready for Phase 1 Implementation
**Next Step**: Complete ParquetMarketDataReader (2-4 hours)