# Wave 4 Test Data Management Guide

**Version**: 1.0
**Date**: 2025-10-22
**Author**: Agent W4-5

---

## 1. Test Data Sources

### Real Market Data (Preferred for Integration/E2E)

**DBN Files** (Databento native format):
- `test_data/ES.FUT.dbn` (500MB, 1M bars, ES futures)
- `test_data/NQ.FUT.dbn` (450MB, 900K bars, NASDAQ futures)
- `test_data/CL.FUT.dbn` (380MB, 750K bars, Crude Oil)
- `test_data/6E.FUT.dbn` (320MB, 600K bars, Euro FX)

**Parquet Files** (ML training):
- `test_data/ES_FUT_180d.parquet` (1.2GB, 180 days)
- `test_data/NQ_FUT_90d.parquet` (600MB, 90 days)
- `test_data/ZN_FUT_90d_clean.parquet` (450MB, clean data)
- `test_data/6E_FUT_180d.parquet` (800MB, 180 days)

**Storage**: Gitignored, download on-demand from Databento (~$2-$4 per symbol)

### Synthetic Data (For Unit Tests)

```rust
use common::testing::generate_mock_market_data;

let market_data = generate_mock_market_data(GenerateConfig {
    symbol: "ES.FUT",
    bars: 1000,
    start_price: 4500.0,
    volatility: 0.02,
    seed: Some(42),  // Deterministic
});
```

### Database Fixtures

```sql
-- tests/fixtures/database/seed_orders.sql
INSERT INTO orders (order_id, symbol, side, quantity, price, status)
VALUES
    (1, 'ES.FUT', 'BUY', 10, 4500.0, 'PENDING'),
    (2, 'NQ.FUT', 'SELL', 5, 18000.0, 'FILLED'),
    (3, '6E.FUT', 'BUY', 20, 1.0850, 'CANCELLED');
```

---

## 2. Test Data Versioning

**Naming Convention**:
- DBN: `{symbol}.{date_range}.dbn` (e.g., `ES.FUT.2024-01-01_to_2024-03-31.dbn`)
- Parquet: `{symbol}_{duration}.parquet` (e.g., `NQ_FUT_180d.parquet`)
- Fixtures: `fixtures/migration_{version}/*.sql`

---

## 3. Test Data Cleanup

```rust
struct TestContext {
    db_pool: PgPool,
    redis_client: RedisClient,
}

impl Drop for TestContext {
    fn drop(&mut self) {
        // Cleanup database
        let _ = self.db_pool.execute("DELETE FROM orders WHERE order_id < 1000000");
        // Cleanup Redis
        let _ = self.redis_client.flushdb();
    }
}
```

---

## 4. Download Script

```bash
#!/bin/bash
# scripts/download_test_data.sh

DATABENTO_API_KEY=${DATABENTO_API_KEY:-"your-api-key"}

# Download ES.FUT (180 days)
databento download \
  --dataset GLBX.MDP3 \
  --symbols ES.FUT \
  --start 2024-06-01 \
  --end 2024-12-01 \
  --schema ohlcv-1m \
  --output test_data/ES_FUT_180d.dbn

echo "Test data downloaded to test_data/"
```

---

**Last Updated**: 2025-10-22