Files

jgrusewski 7458f1be01 feat(wave12): E2E validation complete - 225-feature pipeline ready

✅ Validation Results:
- PPO training: 24.2s (1 epoch, 950 samples, dim=225)
- Feature extraction: 105μs/bar (9.5x faster than target)
- Model checkpoint: 293KB (147KB actor + 146KB critic)
- GPU memory: 145MB used (96.4% headroom)
- Zero dimension mismatches

📊 Success Criteria (5/5):
✅ Feature dimension = 225 (Wave C 201 + Wave D 24)
✅ Model state_dim = 225
✅ Training completed without errors
✅ Checkpoint saved successfully
✅ No dimension mismatch errors

📁 Training Data Ready:
- ES.FUT: 2.9MB, 180 days
- NQ.FUT: 4.4MB, 180 days
- 6E.FUT: 2.8MB, 180 days
- ZN.FUT: 65KB, 90 days (clean)

🚀 Next: Full production model retraining (4 models, ~10min GPU time)

🤖 Generated with Claude Code (https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

2025-10-22 22:48:04 +02:00

2.3 KiB

Raw Blame History

Wave 4 Test Data Management Guide

Version: 1.0 Date: 2025-10-22 Author: Agent W4-5

1. Test Data Sources

Real Market Data (Preferred for Integration/E2E)

DBN Files (Databento native format):

test_data/ES.FUT.dbn (500MB, 1M bars, ES futures)
test_data/NQ.FUT.dbn (450MB, 900K bars, NASDAQ futures)
test_data/CL.FUT.dbn (380MB, 750K bars, Crude Oil)
test_data/6E.FUT.dbn (320MB, 600K bars, Euro FX)

Parquet Files (ML training):

test_data/ES_FUT_180d.parquet (1.2GB, 180 days)
test_data/NQ_FUT_90d.parquet (600MB, 90 days)
test_data/ZN_FUT_90d_clean.parquet (450MB, clean data)
test_data/6E_FUT_180d.parquet (800MB, 180 days)

Storage: Gitignored, download on-demand from Databento (~$2-$4 per symbol)

Synthetic Data (For Unit Tests)

use common::testing::generate_mock_market_data;

let market_data = generate_mock_market_data(GenerateConfig {
    symbol: "ES.FUT",
    bars: 1000,
    start_price: 4500.0,
    volatility: 0.02,
    seed: Some(42),  // Deterministic
});

Database Fixtures

-- tests/fixtures/database/seed_orders.sql
INSERT INTO orders (order_id, symbol, side, quantity, price, status)
VALUES
    (1, 'ES.FUT', 'BUY', 10, 4500.0, 'PENDING'),
    (2, 'NQ.FUT', 'SELL', 5, 18000.0, 'FILLED'),
    (3, '6E.FUT', 'BUY', 20, 1.0850, 'CANCELLED');

2. Test Data Versioning

Naming Convention:

DBN: {symbol}.{date_range}.dbn (e.g., ES.FUT.2024-01-01_to_2024-03-31.dbn)
Parquet: {symbol}_{duration}.parquet (e.g., NQ_FUT_180d.parquet)
Fixtures: fixtures/migration_{version}/*.sql

3. Test Data Cleanup

struct TestContext {
    db_pool: PgPool,
    redis_client: RedisClient,
}

impl Drop for TestContext {
    fn drop(&mut self) {
        // Cleanup database
        let _ = self.db_pool.execute("DELETE FROM orders WHERE order_id < 1000000");
        // Cleanup Redis
        let _ = self.redis_client.flushdb();
    }
}

4. Download Script

#!/bin/bash
# scripts/download_test_data.sh

DATABENTO_API_KEY=${DATABENTO_API_KEY:-"your-api-key"}

# Download ES.FUT (180 days)
databento download \
  --dataset GLBX.MDP3 \
  --symbols ES.FUT \
  --start 2024-06-01 \
  --end 2024-12-01 \
  --schema ohlcv-1m \
  --output test_data/ES_FUT_180d.dbn

echo "Test data downloaded to test_data/"

Last Updated: 2025-10-22

2.3 KiB Raw Blame History