Files
foxhunt/docs/test_fixtures_migration_guide.md
jgrusewski 2df1ea92e1 feat(ml): WAVE 29 DQN Codebase Cleanup & Refactoring Campaign
BREAKING CHANGES:
- Removed orphaned dqn.rs monolithic trainer (4,975 lines)
- Removed orphaned dqn_ensemble.rs module (816 lines)
- Removed orphaned tft.rs and tft_complete_int8_integration_test.rs
- TFT trainer split into modular directory structure

DQN Module Refactoring:
- Split trainers/dqn.rs into modular structure (config.rs, statistics.rs, trainer.rs)
- Fixed hyperopt 39D search space (continuous params only)
- Boolean flags (use_dueling, use_double_dqn, use_per, use_noisy_nets) are now FIXED architectural decisions
- use_distributional defaults to false (Candle BUG #36 - scatter_add gradient issues)

Clean Module Structure:
- ml/src/trainers/dqn/ directory with proper mod.rs exports
- ml/src/trainers/tft/ directory with config.rs, types.rs, model.rs, trainer.rs, tests.rs
- All P0 features validated: TD-error clamping, batch diversity, LR scheduler, priority staleness

Documentation:
- Added comprehensive docs in docs/codebase-cleanup/
- ADR-001 for DQN refactoring decisions
- Rainbow DQN component matrix and quick reference guides

Build Status: Compiles with zero errors

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-27 23:46:13 +01:00

313 lines
6.6 KiB
Markdown

# Test Fixtures Migration Guide
Quick reference for migrating test files to use the new `test_common` crate.
## Quick Start
### 1. Add Dependency
In your test crate's `Cargo.toml`:
```toml
[dev-dependencies]
test_common = { path = "../../tests/test_common" }
```
### 2. Common Migrations
#### Database Setup
**Before:**
```rust
async fn setup_test_db() -> PgPool {
let database_url = std::env::var("DATABASE_URL")
.unwrap_or_else(|_| "postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt".to_string());
PgPool::connect(&database_url).await.unwrap()
}
#[tokio::test]
async fn my_test() {
let pool = setup_test_db().await;
// ...
}
```
**After:**
```rust
use test_common::TestDb;
#[tokio::test]
async fn my_test() {
let db = TestDb::with_migrations().await;
let pool = db.pool();
// ...
}
```
#### Mock Orders
**Before:**
```rust
fn create_mock_order() -> Order {
Order {
id: uuid::Uuid::new_v4().to_string(),
symbol: "AAPL".to_string(),
quantity: 100.0,
price: Some(150.0),
side: Side::Buy,
order_type: OrderType::Limit,
status: OrderStatus::Pending,
timestamp: Utc::now(),
filled_quantity: 0.0,
average_fill_price: None,
}
}
```
**After:**
```rust
use test_common::MockOrderBuilder;
let order = MockOrderBuilder::new()
.symbol("AAPL")
.buy()
.quantity(100.0)
.limit_price(150.0)
.build();
```
#### OHLCV Bars
**Before:**
```rust
fn create_mock_bars(count: usize) -> Vec<OHLCVBar> {
let mut bars = Vec::with_capacity(count);
let base_price = 100.0;
let base_time = chrono::Utc::now();
for i in 0..count {
let change = ((i as f64 / 100.0).sin() * 0.02);
let price = base_price * (1.0 + change);
bars.push(OHLCVBar {
timestamp: base_time + chrono::Duration::minutes((i * 5) as i64),
open: price,
high: price * 1.01,
low: price * 0.99,
close: price,
volume: 1000 + (i * 10) as u64,
});
}
bars
}
```
**After:**
```rust
use test_common::generate_ohlcv_bars;
let bars = generate_ohlcv_bars(100, 100.0);
```
#### Crisis Scenarios (VaR Testing)
**Before:**
```rust
let crisis_returns = vec![
dec!(-0.08), dec!(-0.10), dec!(-0.07), dec!(-0.12),
// ... 16 more lines
];
```
**After:**
```rust
use test_common::fixtures::market_data::generate_crisis_returns;
let crisis_returns = generate_crisis_returns();
```
## Migration Checklist
For each test file:
- [ ] Add `test_common` to `Cargo.toml`
- [ ] Replace `setup_test_db()` with `TestDb`
- [ ] Replace `create_mock_order()` with `MockOrderBuilder`
- [ ] Replace `create_mock_bars()` with `generate_ohlcv_bars()`
- [ ] Replace crisis scenarios with `generate_*_returns()`
- [ ] Replace custom assertions with `test_common::assert_*!()`
- [ ] Remove old helper functions
- [ ] Run tests: `cargo test`
- [ ] Count LOC saved
## Examples by Test Type
### ML Tests
```rust
// Before: 60+ LOC of setup
async fn setup_test_orchestrator() -> Arc<TrainingOrchestrator> {
let ml_config = MLConfig::default();
let db_config = create_test_database_config().await;
// ... 50 more lines
}
// After: 8 LOC
use test_common::{TestDb, fixtures::config::MLTestConfig};
let db = TestDb::with_migrations().await;
let config = MLTestConfig::default_test();
```
### Risk Tests
```rust
// Before: 40+ LOC
fn create_crisis_scenario() -> Vec<Decimal> {
vec![
dec!(-0.08), dec!(-0.10), // ... 18 more
]
}
fn assert_var_valid(returns: &[Decimal], var: Decimal) {
let exceedances = returns.iter()
.filter(|&&r| r.abs() > var)
.count();
assert!(exceedances <= 2);
}
// After: 5 LOC
use test_common::{
fixtures::market_data::generate_crisis_returns,
assertions::assert_var_exceedances,
};
let returns = generate_crisis_returns();
assert_var_exceedances(&returns, var, 0.99, 0.1);
```
### Integration Tests
```rust
// Before: 80+ LOC of database + orchestrator setup
// After: 15 LOC
use test_common::{TestDb, generate_ohlcv_bars, MockOrderBuilder};
#[tokio::test]
async fn end_to_end_workflow() {
let db = TestDb::with_migrations().await;
let bars = generate_ohlcv_bars(100, 150.0);
let order = MockOrderBuilder::new()
.symbol("AAPL")
.buy()
.build();
// Test logic...
}
```
## Common Patterns
### Builder Pattern
All builders follow the same pattern:
```rust
let result = Builder::new()
.property1(value1)
.property2(value2)
.build();
```
### Assertions
Domain-specific assertions:
```rust
// Percentage tolerance
assert_approx_eq!(actual, expected, 0.02); // 2% tolerance
// OHLC validation
assert_ohlc_valid(open, high, low, close);
// VaR exceedances
assert_var_exceedances(&returns, var, confidence, tolerance);
// P&L validation
assert_pnl(entry_price, current_price, quantity, expected, tolerance);
```
## Troubleshooting
### Import Errors
```rust
// If you get "can't find crate test_common"
// Check Cargo.toml has correct path:
[dev-dependencies]
test_common = { path = "../../tests/test_common" }
```
### Type Mismatches
```rust
// MockOrder vs actual Order type
// The mock is for testing only, map to your domain type:
let mock = MockOrderBuilder::new().build();
let domain_order = Order {
id: mock.id,
symbol: Symbol::from_str(&mock.symbol).unwrap(),
// ... map other fields
};
```
### Database Connection Issues
```rust
// If tests fail with connection errors:
// 1. Ensure DATABASE_URL is set
// 2. Ensure PostgreSQL is running
// 3. Check connection pool limits
// Each TestDb creates isolated schema
let db = TestDb::with_migrations().await;
// Schema is automatically cleaned up on drop
```
## Migration Progress Tracking
Create a tracking document:
```markdown
## Test Migration Progress
### ML Training Service (18 files)
- [ ] checkpoint_manager_tests.rs (~30 LOC saved)
- [ ] job_tracker_test.rs (~25 LOC saved)
- [ ] integration_tests.rs (~40 LOC saved)
... (15 more)
### Risk Tests (18 files)
- [ ] risk_comprehensive_tests.rs (~80 LOC saved)
- [ ] var_calculations_tests.rs (~60 LOC saved)
... (16 more)
Total Estimated Savings: XXX LOC
```
## Best Practices
1. **Migrate by directory**: Complete one directory before moving to next
2. **Run tests frequently**: After each migration
3. **Commit often**: One file or small group per commit
4. **Document savings**: Track LOC removed in commit messages
5. **Update imports**: Remove unused imports after migration
## Need Help?
- See `/tests/test_common/README.md` for full documentation
- Check `/docs/test_fixtures_consolidation_report.md` for analysis
- Review existing tests in `test_common/src/*/tests` for examples