Files

jgrusewski 7458f1be01 feat(wave12): E2E validation complete - 225-feature pipeline ready

✅ Validation Results:
- PPO training: 24.2s (1 epoch, 950 samples, dim=225)
- Feature extraction: 105μs/bar (9.5x faster than target)
- Model checkpoint: 293KB (147KB actor + 146KB critic)
- GPU memory: 145MB used (96.4% headroom)
- Zero dimension mismatches

📊 Success Criteria (5/5):
✅ Feature dimension = 225 (Wave C 201 + Wave D 24)
✅ Model state_dim = 225
✅ Training completed without errors
✅ Checkpoint saved successfully
✅ No dimension mismatch errors

📁 Training Data Ready:
- ES.FUT: 2.9MB, 180 days
- NQ.FUT: 4.4MB, 180 days
- 6E.FUT: 2.8MB, 180 days
- ZN.FUT: 65KB, 90 days (clean)

🚀 Next: Full production model retraining (4 models, ~10min GPU time)

🤖 Generated with Claude Code (https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

2025-10-22 22:48:04 +02:00

job_configs

feat(wave12): E2E validation complete - 225-feature pipeline ready

2025-10-22 22:48:04 +02:00

README.md

feat(wave12): E2E validation complete - 225-feature pipeline ready

2025-10-22 22:48:04 +02:00

README.md

Integration Test Fixtures

This directory contains test data fixtures for integration testing.

Directory Structure

fixtures/
├── small_parquet_files/     # Small Parquet files for quick tests (100 bars each)
├── invalid_parquet_files/   # Corrupted files for error testing
├── job_configs/             # Valid and invalid job configurations
└── README.md

Test Data Files

Small Parquet Files

Purpose: Quick training tests without loading large datasets
Size: ~100 OHLCV bars per file
Usage: Real data integration tests

Use the actual small Parquet files from /test_data/ directory:

ES_FUT_small.parquet
NQ_FUT_small.parquet
6E_FUT_small.parquet
ZN_FUT_small.parquet

Invalid Parquet Files

Purpose: Error handling validation
Types: Corrupted headers, missing columns, invalid data types

Job Configurations

Valid Configurations

valid_dqn.json: DQN model with standard hyperparameters
valid_ppo.json: PPO model configuration
valid_mamba2.json: MAMBA-2 configuration
valid_tft.json: TFT configuration

Invalid Configurations

invalid_zero_epochs.json: max_epochs = 0 (should fail validation)
invalid_negative_lr.json: learning_rate < 0 (should fail)
invalid_huge_batch.json: batch_size = 1000000 (resource error)

Creating New Fixtures

Generate Small Parquet Files

# From project root
cargo run --example create_small_parquet_files

This creates 100-bar samples from real market data for fast testing.

Validate Fixtures

# Run integration tests
cd services/ml_training_service
cargo test --test integration -- --test-threads=1

Usage in Tests

use std::path::Path;

// Load valid config
let config_path = Path::new("tests/fixtures/job_configs/valid_dqn.json");
let config = std::fs::read_to_string(config_path)?;

// Load test parquet
let parquet_path = Path::new("test_data/ES_FUT_small.parquet");

Maintenance

Update fixtures when adding new models or features
Keep file sizes small (<100KB for quick CI/CD)
Document any non-obvious test cases