Files
foxhunt/docs/test_fixtures_migration_guide.md
jgrusewski 2df1ea92e1 feat(ml): WAVE 29 DQN Codebase Cleanup & Refactoring Campaign
BREAKING CHANGES:
- Removed orphaned dqn.rs monolithic trainer (4,975 lines)
- Removed orphaned dqn_ensemble.rs module (816 lines)
- Removed orphaned tft.rs and tft_complete_int8_integration_test.rs
- TFT trainer split into modular directory structure

DQN Module Refactoring:
- Split trainers/dqn.rs into modular structure (config.rs, statistics.rs, trainer.rs)
- Fixed hyperopt 39D search space (continuous params only)
- Boolean flags (use_dueling, use_double_dqn, use_per, use_noisy_nets) are now FIXED architectural decisions
- use_distributional defaults to false (Candle BUG #36 - scatter_add gradient issues)

Clean Module Structure:
- ml/src/trainers/dqn/ directory with proper mod.rs exports
- ml/src/trainers/tft/ directory with config.rs, types.rs, model.rs, trainer.rs, tests.rs
- All P0 features validated: TD-error clamping, batch diversity, LR scheduler, priority staleness

Documentation:
- Added comprehensive docs in docs/codebase-cleanup/
- ADR-001 for DQN refactoring decisions
- Rainbow DQN component matrix and quick reference guides

Build Status: Compiles with zero errors

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-27 23:46:13 +01:00

6.6 KiB

Test Fixtures Migration Guide

Quick reference for migrating test files to use the new test_common crate.

Quick Start

1. Add Dependency

In your test crate's Cargo.toml:

[dev-dependencies]
test_common = { path = "../../tests/test_common" }

2. Common Migrations

Database Setup

Before:

async fn setup_test_db() -> PgPool {
    let database_url = std::env::var("DATABASE_URL")
        .unwrap_or_else(|_| "postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt".to_string());
    PgPool::connect(&database_url).await.unwrap()
}

#[tokio::test]
async fn my_test() {
    let pool = setup_test_db().await;
    // ...
}

After:

use test_common::TestDb;

#[tokio::test]
async fn my_test() {
    let db = TestDb::with_migrations().await;
    let pool = db.pool();
    // ...
}

Mock Orders

Before:

fn create_mock_order() -> Order {
    Order {
        id: uuid::Uuid::new_v4().to_string(),
        symbol: "AAPL".to_string(),
        quantity: 100.0,
        price: Some(150.0),
        side: Side::Buy,
        order_type: OrderType::Limit,
        status: OrderStatus::Pending,
        timestamp: Utc::now(),
        filled_quantity: 0.0,
        average_fill_price: None,
    }
}

After:

use test_common::MockOrderBuilder;

let order = MockOrderBuilder::new()
    .symbol("AAPL")
    .buy()
    .quantity(100.0)
    .limit_price(150.0)
    .build();

OHLCV Bars

Before:

fn create_mock_bars(count: usize) -> Vec<OHLCVBar> {
    let mut bars = Vec::with_capacity(count);
    let base_price = 100.0;
    let base_time = chrono::Utc::now();

    for i in 0..count {
        let change = ((i as f64 / 100.0).sin() * 0.02);
        let price = base_price * (1.0 + change);

        bars.push(OHLCVBar {
            timestamp: base_time + chrono::Duration::minutes((i * 5) as i64),
            open: price,
            high: price * 1.01,
            low: price * 0.99,
            close: price,
            volume: 1000 + (i * 10) as u64,
        });
    }

    bars
}

After:

use test_common::generate_ohlcv_bars;

let bars = generate_ohlcv_bars(100, 100.0);

Crisis Scenarios (VaR Testing)

Before:

let crisis_returns = vec![
    dec!(-0.08), dec!(-0.10), dec!(-0.07), dec!(-0.12),
    // ... 16 more lines
];

After:

use test_common::fixtures::market_data::generate_crisis_returns;

let crisis_returns = generate_crisis_returns();

Migration Checklist

For each test file:

  • Add test_common to Cargo.toml
  • Replace setup_test_db() with TestDb
  • Replace create_mock_order() with MockOrderBuilder
  • Replace create_mock_bars() with generate_ohlcv_bars()
  • Replace crisis scenarios with generate_*_returns()
  • Replace custom assertions with test_common::assert_*!()
  • Remove old helper functions
  • Run tests: cargo test
  • Count LOC saved

Examples by Test Type

ML Tests

// Before: 60+ LOC of setup
async fn setup_test_orchestrator() -> Arc<TrainingOrchestrator> {
    let ml_config = MLConfig::default();
    let db_config = create_test_database_config().await;
    // ... 50 more lines
}

// After: 8 LOC
use test_common::{TestDb, fixtures::config::MLTestConfig};

let db = TestDb::with_migrations().await;
let config = MLTestConfig::default_test();

Risk Tests

// Before: 40+ LOC
fn create_crisis_scenario() -> Vec<Decimal> {
    vec![
        dec!(-0.08), dec!(-0.10), // ... 18 more
    ]
}

fn assert_var_valid(returns: &[Decimal], var: Decimal) {
    let exceedances = returns.iter()
        .filter(|&&r| r.abs() > var)
        .count();
    assert!(exceedances <= 2);
}

// After: 5 LOC
use test_common::{
    fixtures::market_data::generate_crisis_returns,
    assertions::assert_var_exceedances,
};

let returns = generate_crisis_returns();
assert_var_exceedances(&returns, var, 0.99, 0.1);

Integration Tests

// Before: 80+ LOC of database + orchestrator setup

// After: 15 LOC
use test_common::{TestDb, generate_ohlcv_bars, MockOrderBuilder};

#[tokio::test]
async fn end_to_end_workflow() {
    let db = TestDb::with_migrations().await;
    let bars = generate_ohlcv_bars(100, 150.0);
    let order = MockOrderBuilder::new()
        .symbol("AAPL")
        .buy()
        .build();

    // Test logic...
}

Common Patterns

Builder Pattern

All builders follow the same pattern:

let result = Builder::new()
    .property1(value1)
    .property2(value2)
    .build();

Assertions

Domain-specific assertions:

// Percentage tolerance
assert_approx_eq!(actual, expected, 0.02); // 2% tolerance

// OHLC validation
assert_ohlc_valid(open, high, low, close);

// VaR exceedances
assert_var_exceedances(&returns, var, confidence, tolerance);

// P&L validation
assert_pnl(entry_price, current_price, quantity, expected, tolerance);

Troubleshooting

Import Errors

// If you get "can't find crate test_common"
// Check Cargo.toml has correct path:
[dev-dependencies]
test_common = { path = "../../tests/test_common" }

Type Mismatches

// MockOrder vs actual Order type
// The mock is for testing only, map to your domain type:
let mock = MockOrderBuilder::new().build();
let domain_order = Order {
    id: mock.id,
    symbol: Symbol::from_str(&mock.symbol).unwrap(),
    // ... map other fields
};

Database Connection Issues

// If tests fail with connection errors:
// 1. Ensure DATABASE_URL is set
// 2. Ensure PostgreSQL is running
// 3. Check connection pool limits

// Each TestDb creates isolated schema
let db = TestDb::with_migrations().await;
// Schema is automatically cleaned up on drop

Migration Progress Tracking

Create a tracking document:

## Test Migration Progress

### ML Training Service (18 files)
- [ ] checkpoint_manager_tests.rs (~30 LOC saved)
- [ ] job_tracker_test.rs (~25 LOC saved)
- [ ] integration_tests.rs (~40 LOC saved)
... (15 more)

### Risk Tests (18 files)
- [ ] risk_comprehensive_tests.rs (~80 LOC saved)
- [ ] var_calculations_tests.rs (~60 LOC saved)
... (16 more)

Total Estimated Savings: XXX LOC

Best Practices

  1. Migrate by directory: Complete one directory before moving to next
  2. Run tests frequently: After each migration
  3. Commit often: One file or small group per commit
  4. Document savings: Track LOC removed in commit messages
  5. Update imports: Remove unused imports after migration

Need Help?

  • See /tests/test_common/README.md for full documentation
  • Check /docs/test_fixtures_consolidation_report.md for analysis
  • Review existing tests in test_common/src/*/tests for examples