foxhunt/DQN_REPLAY_PIPELINE_TEST_GUIDE.md

DQN Replay Pipeline Integration Test Guide

Status: ✅ COMPLETE Date: 2025-11-01 Component: Task 3.5 - Integration Tests & Validation

---

Overview

This guide documents the comprehensive end-to-end integration test suite for the DQN replay evaluation pipeline. The tests validate the complete workflow from model checkpoint export to inference to metric calculation.

---

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                  DQN REPLAY PIPELINE                            │
│                                                                 │
│  1. EXPORT PHASE                                                │
│     ├─ Train minimal DQN model (10 epochs)                     │
│     ├─ Save checkpoint to SafeTensors                          │
│     └─ Validate checkpoint file exists                         │
│                                                                 │
│  2. LOAD PHASE                                                  │
│     ├─ Load Parquet data (225 features per bar)                │
│     ├─ Load DQN checkpoint (SafeTensors)                       │
│     ├─ Load timestamps and OHLCV bars (for action export)      │
│     └─ Validate dimensions (225 input, 3 output)               │
│                                                                 │
│  3. INFERENCE PHASE                                             │
│     ├─ Run greedy inference (epsilon=0.0)                      │
│     ├─ Track timestamps per action                             │
│     ├─ Collect latency metrics (microsecond precision)         │
│     └─ Handle NaN/Inf gracefully                               │
│                                                                 │
│  4. VALIDATION PHASE                                            │
│     ├─ Calculate metrics (action distribution, Q-values)       │
│     ├─ Validate timestamp alignment (>90% match rate)          │
│     ├─ Validate performance (<30s total runtime)               │
│     └─ Generate comprehensive report                           │
└─────────────────────────────────────────────────────────────────┘

---

Test Suite

Test 1: Full Pipeline (test_full_replay_pipeline)

Purpose: End-to-end validation of complete pipeline

Steps:

1. Create temporary directory for test artifacts
2. Train minimal DQN model (10 epochs, 20 experiences)
3. Export checkpoint to SafeTensors
4. Load Parquet data (test_data/ES_FUT_unseen.parquet)
5. Load DQN checkpoint
6. Run inference on all bars
7. Calculate metrics (action distribution, Q-values, latency)
8. Validate all metrics are finite (no NaN/Inf)
9. Validate total runtime <30s

Success Criteria:

• ✅ Checkpoint export succeeds (SafeTensors file created)
• ✅ Parquet loading succeeds (225 features per bar)
• ✅ Inference completes without errors
• ✅ All metrics are finite (no NaN/Inf)
• ✅ Total runtime <30s (performance constraint)
• ✅ All actions used at least once (BUY, SELL, HOLD)

Example Output:

╔══════════════════════════════════════════════════════════════════════╗
║         TEST 1: Full DQN Replay Pipeline                            ║
╚══════════════════════════════════════════════════════════════════════╝

📦 Step 1: Creating DQN model and exporting checkpoint...
  Training DQN for 10 steps...
  Saving checkpoint to: /tmp/.tmpXXXXXX/dqn_test_checkpoint.safetensors
✅ Step 1 complete: Checkpoint saved (12345 bytes)

📂 Step 2: Loading Parquet data and DQN checkpoint...
  Loaded 1500 feature vectors (0.42s)
✅ Step 2 complete: Data and checkpoint loaded

🔍 Step 3: Running DQN inference...
  Processed 1500 bars (2.34s)
  Skipped 0 bars
✅ Step 3 complete: Inference finished

📊 Step 4: Validating metrics and performance...
  Action Distribution:
    BUY:  450 (30.0%)
    SELL: 300 (20.0%)
    HOLD: 750 (50.0%)

  Q-Value Statistics:
    Mean: 0.1234
    Min:  -0.5678
    Max:  0.9876

  Performance:
    Total runtime: 3.45s
    Throughput: 434.8 bars/sec

✅ Step 4 complete: All validations passed

╔══════════════════════════════════════════════════════════════════════╗
║                    TEST 1: PASSED ✅                                 ║
╚══════════════════════════════════════════════════════════════════════╝

---

Test 2: Timestamp Alignment (test_timestamp_alignment)

Purpose: Validate timestamp synchronization between actions and market bars

Steps:

1. Load Parquet data WITH timestamps (load_parquet_data_with_timestamps)
2. Run inference with timestamp tracking
3. Validate alignment rate (>90% match)
4. Validate chronological ordering (no time travel)
5. Check for duplicate timestamps

Success Criteria:

• ✅ Timestamps from Parquet match inference results (>90% alignment)
• ✅ Chronological ordering preserved (no time travel)
• ✅ Duplicate timestamps <10% of total (acceptable for aggregated data)

Example Output:

╔══════════════════════════════════════════════════════════════════════╗
║         TEST 2: Timestamp Alignment                                 ║
╚══════════════════════════════════════════════════════════════════════╝

📂 Loading Parquet data with timestamps...
  Loaded 1500 bars with timestamps

🔍 Running inference with timestamp tracking...
  Generated 1500 actions with timestamps

📊 Validating timestamp alignment...

  Alignment Statistics:
    Matched: 1485 / 1500
    Alignment rate: 99.0%

  ✅ Timestamps are chronologically ordered
  Duplicate timestamps: 15

✅ Step 3 complete: Timestamp alignment validated

╔══════════════════════════════════════════════════════════════════════╗
║                    TEST 2: PASSED ✅                                 ║
╚══════════════════════════════════════════════════════════════════════╝

---

Test 3: Performance Benchmarks (test_replay_performance)

Purpose: Validate performance constraints (<30s total, <5ms P99 latency)

Steps:

1. Setup minimal DQN model (emergency safe defaults)
2. Load Parquet data and benchmark I/O
3. Run inference with per-bar latency tracking
4. Calculate latency statistics (mean, P50, P95, P99)
5. Validate total runtime <30s
6. Validate throughput >100 bars/sec

Success Criteria:

• ✅ Total runtime <30s (including I/O, inference, metrics)
• ✅ Inference latency P99 <5ms per bar
• ✅ Throughput >100 bars/sec

Example Output:

╔══════════════════════════════════════════════════════════════════════╗
║         TEST 3: Performance Benchmarks                              ║
╚══════════════════════════════════════════════════════════════════════╝

📦 Setting up DQN model...
✅ DQN created

📂 Loading Parquet data...
  Loaded 1500 bars in 0.42s

🔍 Running inference with latency tracking...

  Inference Latency:
    Mean:   234μs (0.234ms)
    P50:    200μs (0.200ms)
    P95:    450μs (0.450ms)
    P99:    1200μs (1.200ms)

  Total Performance:
    Total runtime: 3.45s
    Throughput: 434.8 bars/sec

✅ All performance benchmarks passed

╔══════════════════════════════════════════════════════════════════════╗
║                    TEST 3: PASSED ✅                                 ║
╚══════════════════════════════════════════════════════════════════════╝

---

Test 4: Edge Cases (test_replay_edge_cases)

Purpose: Validate graceful error handling for edge cases

Steps:

1. Test empty feature vector (should fail gracefully)
2. Test corrupt checkpoint file (should fail with clear error)
3. Test NaN in features (propagates to Q-values or fails gracefully)
4. Test Inf in features (propagates to Q-values or fails gracefully)

Success Criteria:

• ✅ Empty state fails gracefully (returns error)
• ✅ Corrupt checkpoint fails with clear error message
• ✅ NaN/Inf in features handled correctly (propagation or rejection)

Example Output:

╔══════════════════════════════════════════════════════════════════════╗
║         TEST 4: Edge Case Handling                                  ║
╚══════════════════════════════════════════════════════════════════════╝

🧪 Test 4.1: Empty feature vector...
  ✅ Empty state handled correctly

🧪 Test 4.2: Corrupt checkpoint...
  ✅ Corrupt checkpoint handled correctly

🧪 Test 4.3: NaN in features...
  ⚠️  Q-values contain NaN (propagated from input): true
  ✅ NaN handling validated

🧪 Test 4.4: Inf in features...
  ⚠️  Q-values contain Inf (propagated from input): true
  ✅ Inf handling validated

╔══════════════════════════════════════════════════════════════════════╗
║                    TEST 4: PASSED ✅                                 ║
╚══════════════════════════════════════════════════════════════════════╝

---

Test 5: Memory Efficiency (test_memory_efficiency)

Purpose: Validate memory efficiency with large datasets (10,000+ bars)

Steps:

1. Generate synthetic dataset (10,000 bars × 225 features)
2. Calculate expected memory usage (<500 MB for features)
3. Run inference on all bars
4. Track progress and throughput
5. Validate no OOM errors
6. Validate throughput >100 bars/sec

Success Criteria:

• ✅ Process 10,000+ bars without OOM
• ✅ Memory usage stays reasonable (<500 MB for features)
• ✅ Throughput >100 bars/sec

Example Output:

╔══════════════════════════════════════════════════════════════════════╗
║         TEST 5: Memory Efficiency                                   ║
╚══════════════════════════════════════════════════════════════════════╝

🧪 Generating synthetic dataset (10,000 bars x 225 features)...
  Features memory: 17.17 MB

🔍 Running inference on 10000 bars...
  Progress: 1000 / 10000 bars (2345.6 bars/sec)
  Progress: 2000 / 10000 bars (2378.4 bars/sec)
  Progress: 3000 / 10000 bars (2401.2 bars/sec)
  ...
  Progress: 10000 / 10000 bars (2456.7 bars/sec)

  Inference complete:
    Processed: 10000 / 10000 bars
    Total time: 4.07s
    Throughput: 2456.7 bars/sec

✅ Memory efficiency validated

╔══════════════════════════════════════════════════════════════════════╗
║                    TEST 5: PASSED ✅                                 ║
╚══════════════════════════════════════════════════════════════════════╝

---

Shell Script Usage

Basic Usage

# Run all 5 tests (default)
./test_dqn_replay_pipeline.sh

# Run quick validation (2 tests: Full Pipeline + Performance)
./test_dqn_replay_pipeline.sh --quick

# Enable verbose logging (show test output)
./test_dqn_replay_pipeline.sh --verbose

# CI/CD mode (no ANSI colors)
./test_dqn_replay_pipeline.sh --ci

# Skip cleanup of temporary files
./test_dqn_replay_pipeline.sh --no-cleanup

# Show help
./test_dqn_replay_pipeline.sh --help

Example Output (Full Run)

╔══════════════════════════════════════════════════════════════════════╗
║        DQN Replay Pipeline Integration Test Suite                   ║
║        Version: 1.0.0                                                ║
╚══════════════════════════════════════════════════════════════════════╝

ℹ  Run mode: full
ℹ  Verbose: false
ℹ  CI mode: false


═══ Pre-flight Checks ═══

✅ cargo found: cargo 1.XX.X
⚠️  CUDA not available, tests will run on CPU
✅ Test data directory found: /home/.../foxhunt/test_data
✅ Test file found: test_data/ES_FUT_unseen.parquet (1234567 bytes)
✅ Disk space: 12345 MB available


═══ Running Integration Tests ═══

ℹ  Test 1/5: Full Pipeline (export → load → backtest → validate)...
✅ Test 1: Full Pipeline PASSED
ℹ  Test 2/5: Timestamp Alignment (>90% match rate)...
✅ Test 2: Timestamp Alignment PASSED
ℹ  Test 3/5: Performance (<30s constraint)...
✅ Test 3: Performance PASSED
ℹ  Test 4/5: Edge Cases (empty data, corrupt checkpoints, NaN/Inf)...
✅ Test 4: Edge Cases PASSED
ℹ  Test 5/5: Memory Efficiency (10,000+ bars without OOM)...
✅ Test 5: Memory Efficiency PASSED


═══ Test Summary ═══

Test Results:
  • Test 1 (Full Pipeline):       PASS
  • Test 2 (Timestamp Alignment): PASS
  • Test 3 (Performance):         PASS
  • Test 4 (Edge Cases):          PASS
  • Test 5 (Memory Efficiency):   PASS

Summary:
  • Total tests: 5
  • Passed: 5
  • Failed: 0
  • Pass rate: 100%
  • Total time: 42s

✅ ALL TESTS PASSED ✅


═══ Cleanup ═══

ℹ  Removing temporary test artifacts...
✅ Removed temporary checkpoints
✅ Cleaned cargo artifacts

╔══════════════════════════════════════════════════════════════════════╗
║                    ALL TESTS PASSED ✅                               ║
╚══════════════════════════════════════════════════════════════════════╝

---

CI/CD Integration

GitLab CI Example

test:dqn_replay_pipeline:
  stage: test
  script:
    - ./test_dqn_replay_pipeline.sh --ci
  artifacts:
    when: always
    paths:
      - test_results/
    expire_in: 1 week
  timeout: 10 minutes

GitHub Actions Example

- name: Run DQN Replay Pipeline Tests
  run: ./test_dqn_replay_pipeline.sh --ci
  timeout-minutes: 10

---

Troubleshooting

Test Data Missing

Error:

⚠️  Required test file not found: test_data/ES_FUT_unseen.parquet

Solution:

# Option 1: Use existing small test file
ln -s test_data/ES_FUT_small.parquet test_data/ES_FUT_unseen.parquet

# Option 2: Export new test data from DBN
cargo run -p ml --example export_parquet_from_dbn --release -- \
  --input test_data/ES_FUT_180d.dbn \
  --output test_data/ES_FUT_unseen.parquet

CUDA Not Available

Warning:

⚠️  CUDA not available, tests will run on CPU

Impact: Tests will run on CPU (slower but functional)

Solution (if GPU is needed):

# Verify CUDA installation
nvidia-smi

# Check CUDA version
nvcc --version

# Rebuild with CUDA features
cargo test -p ml --test dqn_replay_full_pipeline_test --release --features cuda

Low Disk Space

Warning:

⚠️  Low disk space: 234 MB available (recommended: 500 MB)

Solution:

# Clean up old artifacts
cargo clean

# Remove temporary files
rm -rf /tmp/dqn_*.safetensors

# Free up disk space
df -h

Tests Timeout

Error:

❌ Test 1: Full Pipeline FAILED (timeout)

Solution:

# Run in verbose mode to see where it hangs
./test_dqn_replay_pipeline.sh --verbose

# Run only quick validation (2 tests)
./test_dqn_replay_pipeline.sh --quick

# Increase timeout in CI/CD config (e.g., 20 minutes)

---

Files Created

1. ml/tests/dqn_replay_full_pipeline_test.rs

   • 5 integration tests (full pipeline, timestamp alignment, performance, edge cases, memory efficiency)
   • 700+ lines of comprehensive test coverage
   • Validates all success criteria from Wave 3 plan

2. test_dqn_replay_pipeline.sh

   • Shell script for running all tests
   • CI/CD integration support (--ci flag)
   • Quick validation mode (--quick flag)
   • Pre-flight checks (dependencies, test data, disk space)
   • Cleanup of temporary files

3. DQN_REPLAY_PIPELINE_TEST_GUIDE.md (this file)

   • Complete documentation for test suite
   • Architecture diagrams
   • Usage examples
   • Troubleshooting guide

---

Success Criteria Validation

✅ Full pipeline test: Export → Load → Backtest → Validate metrics ✅ Timestamp alignment: >90% match rate between actions and bars ✅ Performance: <30s backtest constraint ✅ Edge cases: Empty data, corrupt checkpoints, NaN/Inf handling ✅ Memory efficiency: 10,000+ bars without OOM ✅ CI/CD ready: Shell script with --ci flag

---

Next Steps

1. Run tests locally:

   ./test_dqn_replay_pipeline.sh --verbose
   

2. Add to CI/CD pipeline:

   test:dqn_replay_pipeline:
     stage: test
     script:
       - ./test_dqn_replay_pipeline.sh --ci
   

3. Generate test data (if missing):

   # Export from DBN file
   cargo run -p ml --example export_parquet_from_dbn --release -- \
     --input test_data/ES_FUT_180d.dbn \
     --output test_data/ES_FUT_unseen.parquet
   

4. Run quick validation (before commits):

   ./test_dqn_replay_pipeline.sh --quick
   

---

Related Documentation

• Task 3.2b: DQN Checkpoint Loading (load_from_safetensors)
• Task 3.3: Parquet Data Loading (load_parquet_data_with_timestamps)
• Task 3.4: DQN Inference Engine (run_inference, calculate_metrics)
• Wave 3 Plan: Complete DQN evaluation pipeline design
• CLAUDE.md: System overview and development workflow

---

Document Status: ✅ COMPLETE Last Updated: 2025-11-01 Author: Claude (Task 3.5 Implementation)