7458f1be01
✅ Validation Results: - PPO training: 24.2s (1 epoch, 950 samples, dim=225) - Feature extraction: 105μs/bar (9.5x faster than target) - Model checkpoint: 293KB (147KB actor + 146KB critic) - GPU memory: 145MB used (96.4% headroom) - Zero dimension mismatches 📊 Success Criteria (5/5): ✅ Feature dimension = 225 (Wave C 201 + Wave D 24) ✅ Model state_dim = 225 ✅ Training completed without errors ✅ Checkpoint saved successfully ✅ No dimension mismatch errors 📁 Training Data Ready: - ES.FUT: 2.9MB, 180 days - NQ.FUT: 4.4MB, 180 days - 6E.FUT: 2.8MB, 180 days - ZN.FUT: 65KB, 90 days (clean) 🚀 Next: Full production model retraining (4 models, ~10min GPU time) 🤖 Generated with Claude Code (https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
106 lines
2.3 KiB
Markdown
106 lines
2.3 KiB
Markdown
# Wave 4 Test Data Management Guide
|
|
|
|
**Version**: 1.0
|
|
**Date**: 2025-10-22
|
|
**Author**: Agent W4-5
|
|
|
|
---
|
|
|
|
## 1. Test Data Sources
|
|
|
|
### Real Market Data (Preferred for Integration/E2E)
|
|
|
|
**DBN Files** (Databento native format):
|
|
- `test_data/ES.FUT.dbn` (500MB, 1M bars, ES futures)
|
|
- `test_data/NQ.FUT.dbn` (450MB, 900K bars, NASDAQ futures)
|
|
- `test_data/CL.FUT.dbn` (380MB, 750K bars, Crude Oil)
|
|
- `test_data/6E.FUT.dbn` (320MB, 600K bars, Euro FX)
|
|
|
|
**Parquet Files** (ML training):
|
|
- `test_data/ES_FUT_180d.parquet` (1.2GB, 180 days)
|
|
- `test_data/NQ_FUT_90d.parquet` (600MB, 90 days)
|
|
- `test_data/ZN_FUT_90d_clean.parquet` (450MB, clean data)
|
|
- `test_data/6E_FUT_180d.parquet` (800MB, 180 days)
|
|
|
|
**Storage**: Gitignored, download on-demand from Databento (~$2-$4 per symbol)
|
|
|
|
### Synthetic Data (For Unit Tests)
|
|
|
|
```rust
|
|
use common::testing::generate_mock_market_data;
|
|
|
|
let market_data = generate_mock_market_data(GenerateConfig {
|
|
symbol: "ES.FUT",
|
|
bars: 1000,
|
|
start_price: 4500.0,
|
|
volatility: 0.02,
|
|
seed: Some(42), // Deterministic
|
|
});
|
|
```
|
|
|
|
### Database Fixtures
|
|
|
|
```sql
|
|
-- tests/fixtures/database/seed_orders.sql
|
|
INSERT INTO orders (order_id, symbol, side, quantity, price, status)
|
|
VALUES
|
|
(1, 'ES.FUT', 'BUY', 10, 4500.0, 'PENDING'),
|
|
(2, 'NQ.FUT', 'SELL', 5, 18000.0, 'FILLED'),
|
|
(3, '6E.FUT', 'BUY', 20, 1.0850, 'CANCELLED');
|
|
```
|
|
|
|
---
|
|
|
|
## 2. Test Data Versioning
|
|
|
|
**Naming Convention**:
|
|
- DBN: `{symbol}.{date_range}.dbn` (e.g., `ES.FUT.2024-01-01_to_2024-03-31.dbn`)
|
|
- Parquet: `{symbol}_{duration}.parquet` (e.g., `NQ_FUT_180d.parquet`)
|
|
- Fixtures: `fixtures/migration_{version}/*.sql`
|
|
|
|
---
|
|
|
|
## 3. Test Data Cleanup
|
|
|
|
```rust
|
|
struct TestContext {
|
|
db_pool: PgPool,
|
|
redis_client: RedisClient,
|
|
}
|
|
|
|
impl Drop for TestContext {
|
|
fn drop(&mut self) {
|
|
// Cleanup database
|
|
let _ = self.db_pool.execute("DELETE FROM orders WHERE order_id < 1000000");
|
|
// Cleanup Redis
|
|
let _ = self.redis_client.flushdb();
|
|
}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 4. Download Script
|
|
|
|
```bash
|
|
#!/bin/bash
|
|
# scripts/download_test_data.sh
|
|
|
|
DATABENTO_API_KEY=${DATABENTO_API_KEY:-"your-api-key"}
|
|
|
|
# Download ES.FUT (180 days)
|
|
databento download \
|
|
--dataset GLBX.MDP3 \
|
|
--symbols ES.FUT \
|
|
--start 2024-06-01 \
|
|
--end 2024-12-01 \
|
|
--schema ohlcv-1m \
|
|
--output test_data/ES_FUT_180d.dbn
|
|
|
|
echo "Test data downloaded to test_data/"
|
|
```
|
|
|
|
---
|
|
|
|
**Last Updated**: 2025-10-22
|