BREAKING CHANGES: - Removed orphaned dqn.rs monolithic trainer (4,975 lines) - Removed orphaned dqn_ensemble.rs module (816 lines) - Removed orphaned tft.rs and tft_complete_int8_integration_test.rs - TFT trainer split into modular directory structure DQN Module Refactoring: - Split trainers/dqn.rs into modular structure (config.rs, statistics.rs, trainer.rs) - Fixed hyperopt 39D search space (continuous params only) - Boolean flags (use_dueling, use_double_dqn, use_per, use_noisy_nets) are now FIXED architectural decisions - use_distributional defaults to false (Candle BUG #36 - scatter_add gradient issues) Clean Module Structure: - ml/src/trainers/dqn/ directory with proper mod.rs exports - ml/src/trainers/tft/ directory with config.rs, types.rs, model.rs, trainer.rs, tests.rs - All P0 features validated: TD-error clamping, batch diversity, LR scheduler, priority staleness Documentation: - Added comprehensive docs in docs/codebase-cleanup/ - ADR-001 for DQN refactoring decisions - Rainbow DQN component matrix and quick reference guides Build Status: Compiles with zero errors 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
313 lines
6.6 KiB
Markdown
313 lines
6.6 KiB
Markdown
# Test Fixtures Migration Guide
|
|
|
|
Quick reference for migrating test files to use the new `test_common` crate.
|
|
|
|
## Quick Start
|
|
|
|
### 1. Add Dependency
|
|
|
|
In your test crate's `Cargo.toml`:
|
|
|
|
```toml
|
|
[dev-dependencies]
|
|
test_common = { path = "../../tests/test_common" }
|
|
```
|
|
|
|
### 2. Common Migrations
|
|
|
|
#### Database Setup
|
|
|
|
**Before:**
|
|
```rust
|
|
async fn setup_test_db() -> PgPool {
|
|
let database_url = std::env::var("DATABASE_URL")
|
|
.unwrap_or_else(|_| "postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt".to_string());
|
|
PgPool::connect(&database_url).await.unwrap()
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn my_test() {
|
|
let pool = setup_test_db().await;
|
|
// ...
|
|
}
|
|
```
|
|
|
|
**After:**
|
|
```rust
|
|
use test_common::TestDb;
|
|
|
|
#[tokio::test]
|
|
async fn my_test() {
|
|
let db = TestDb::with_migrations().await;
|
|
let pool = db.pool();
|
|
// ...
|
|
}
|
|
```
|
|
|
|
#### Mock Orders
|
|
|
|
**Before:**
|
|
```rust
|
|
fn create_mock_order() -> Order {
|
|
Order {
|
|
id: uuid::Uuid::new_v4().to_string(),
|
|
symbol: "AAPL".to_string(),
|
|
quantity: 100.0,
|
|
price: Some(150.0),
|
|
side: Side::Buy,
|
|
order_type: OrderType::Limit,
|
|
status: OrderStatus::Pending,
|
|
timestamp: Utc::now(),
|
|
filled_quantity: 0.0,
|
|
average_fill_price: None,
|
|
}
|
|
}
|
|
```
|
|
|
|
**After:**
|
|
```rust
|
|
use test_common::MockOrderBuilder;
|
|
|
|
let order = MockOrderBuilder::new()
|
|
.symbol("AAPL")
|
|
.buy()
|
|
.quantity(100.0)
|
|
.limit_price(150.0)
|
|
.build();
|
|
```
|
|
|
|
#### OHLCV Bars
|
|
|
|
**Before:**
|
|
```rust
|
|
fn create_mock_bars(count: usize) -> Vec<OHLCVBar> {
|
|
let mut bars = Vec::with_capacity(count);
|
|
let base_price = 100.0;
|
|
let base_time = chrono::Utc::now();
|
|
|
|
for i in 0..count {
|
|
let change = ((i as f64 / 100.0).sin() * 0.02);
|
|
let price = base_price * (1.0 + change);
|
|
|
|
bars.push(OHLCVBar {
|
|
timestamp: base_time + chrono::Duration::minutes((i * 5) as i64),
|
|
open: price,
|
|
high: price * 1.01,
|
|
low: price * 0.99,
|
|
close: price,
|
|
volume: 1000 + (i * 10) as u64,
|
|
});
|
|
}
|
|
|
|
bars
|
|
}
|
|
```
|
|
|
|
**After:**
|
|
```rust
|
|
use test_common::generate_ohlcv_bars;
|
|
|
|
let bars = generate_ohlcv_bars(100, 100.0);
|
|
```
|
|
|
|
#### Crisis Scenarios (VaR Testing)
|
|
|
|
**Before:**
|
|
```rust
|
|
let crisis_returns = vec![
|
|
dec!(-0.08), dec!(-0.10), dec!(-0.07), dec!(-0.12),
|
|
// ... 16 more lines
|
|
];
|
|
```
|
|
|
|
**After:**
|
|
```rust
|
|
use test_common::fixtures::market_data::generate_crisis_returns;
|
|
|
|
let crisis_returns = generate_crisis_returns();
|
|
```
|
|
|
|
## Migration Checklist
|
|
|
|
For each test file:
|
|
|
|
- [ ] Add `test_common` to `Cargo.toml`
|
|
- [ ] Replace `setup_test_db()` with `TestDb`
|
|
- [ ] Replace `create_mock_order()` with `MockOrderBuilder`
|
|
- [ ] Replace `create_mock_bars()` with `generate_ohlcv_bars()`
|
|
- [ ] Replace crisis scenarios with `generate_*_returns()`
|
|
- [ ] Replace custom assertions with `test_common::assert_*!()`
|
|
- [ ] Remove old helper functions
|
|
- [ ] Run tests: `cargo test`
|
|
- [ ] Count LOC saved
|
|
|
|
## Examples by Test Type
|
|
|
|
### ML Tests
|
|
|
|
```rust
|
|
// Before: 60+ LOC of setup
|
|
async fn setup_test_orchestrator() -> Arc<TrainingOrchestrator> {
|
|
let ml_config = MLConfig::default();
|
|
let db_config = create_test_database_config().await;
|
|
// ... 50 more lines
|
|
}
|
|
|
|
// After: 8 LOC
|
|
use test_common::{TestDb, fixtures::config::MLTestConfig};
|
|
|
|
let db = TestDb::with_migrations().await;
|
|
let config = MLTestConfig::default_test();
|
|
```
|
|
|
|
### Risk Tests
|
|
|
|
```rust
|
|
// Before: 40+ LOC
|
|
fn create_crisis_scenario() -> Vec<Decimal> {
|
|
vec![
|
|
dec!(-0.08), dec!(-0.10), // ... 18 more
|
|
]
|
|
}
|
|
|
|
fn assert_var_valid(returns: &[Decimal], var: Decimal) {
|
|
let exceedances = returns.iter()
|
|
.filter(|&&r| r.abs() > var)
|
|
.count();
|
|
assert!(exceedances <= 2);
|
|
}
|
|
|
|
// After: 5 LOC
|
|
use test_common::{
|
|
fixtures::market_data::generate_crisis_returns,
|
|
assertions::assert_var_exceedances,
|
|
};
|
|
|
|
let returns = generate_crisis_returns();
|
|
assert_var_exceedances(&returns, var, 0.99, 0.1);
|
|
```
|
|
|
|
### Integration Tests
|
|
|
|
```rust
|
|
// Before: 80+ LOC of database + orchestrator setup
|
|
|
|
// After: 15 LOC
|
|
use test_common::{TestDb, generate_ohlcv_bars, MockOrderBuilder};
|
|
|
|
#[tokio::test]
|
|
async fn end_to_end_workflow() {
|
|
let db = TestDb::with_migrations().await;
|
|
let bars = generate_ohlcv_bars(100, 150.0);
|
|
let order = MockOrderBuilder::new()
|
|
.symbol("AAPL")
|
|
.buy()
|
|
.build();
|
|
|
|
// Test logic...
|
|
}
|
|
```
|
|
|
|
## Common Patterns
|
|
|
|
### Builder Pattern
|
|
|
|
All builders follow the same pattern:
|
|
|
|
```rust
|
|
let result = Builder::new()
|
|
.property1(value1)
|
|
.property2(value2)
|
|
.build();
|
|
```
|
|
|
|
### Assertions
|
|
|
|
Domain-specific assertions:
|
|
|
|
```rust
|
|
// Percentage tolerance
|
|
assert_approx_eq!(actual, expected, 0.02); // 2% tolerance
|
|
|
|
// OHLC validation
|
|
assert_ohlc_valid(open, high, low, close);
|
|
|
|
// VaR exceedances
|
|
assert_var_exceedances(&returns, var, confidence, tolerance);
|
|
|
|
// P&L validation
|
|
assert_pnl(entry_price, current_price, quantity, expected, tolerance);
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Import Errors
|
|
|
|
```rust
|
|
// If you get "can't find crate test_common"
|
|
// Check Cargo.toml has correct path:
|
|
[dev-dependencies]
|
|
test_common = { path = "../../tests/test_common" }
|
|
```
|
|
|
|
### Type Mismatches
|
|
|
|
```rust
|
|
// MockOrder vs actual Order type
|
|
// The mock is for testing only, map to your domain type:
|
|
let mock = MockOrderBuilder::new().build();
|
|
let domain_order = Order {
|
|
id: mock.id,
|
|
symbol: Symbol::from_str(&mock.symbol).unwrap(),
|
|
// ... map other fields
|
|
};
|
|
```
|
|
|
|
### Database Connection Issues
|
|
|
|
```rust
|
|
// If tests fail with connection errors:
|
|
// 1. Ensure DATABASE_URL is set
|
|
// 2. Ensure PostgreSQL is running
|
|
// 3. Check connection pool limits
|
|
|
|
// Each TestDb creates isolated schema
|
|
let db = TestDb::with_migrations().await;
|
|
// Schema is automatically cleaned up on drop
|
|
```
|
|
|
|
## Migration Progress Tracking
|
|
|
|
Create a tracking document:
|
|
|
|
```markdown
|
|
## Test Migration Progress
|
|
|
|
### ML Training Service (18 files)
|
|
- [ ] checkpoint_manager_tests.rs (~30 LOC saved)
|
|
- [ ] job_tracker_test.rs (~25 LOC saved)
|
|
- [ ] integration_tests.rs (~40 LOC saved)
|
|
... (15 more)
|
|
|
|
### Risk Tests (18 files)
|
|
- [ ] risk_comprehensive_tests.rs (~80 LOC saved)
|
|
- [ ] var_calculations_tests.rs (~60 LOC saved)
|
|
... (16 more)
|
|
|
|
Total Estimated Savings: XXX LOC
|
|
```
|
|
|
|
## Best Practices
|
|
|
|
1. **Migrate by directory**: Complete one directory before moving to next
|
|
2. **Run tests frequently**: After each migration
|
|
3. **Commit often**: One file or small group per commit
|
|
4. **Document savings**: Track LOC removed in commit messages
|
|
5. **Update imports**: Remove unused imports after migration
|
|
|
|
## Need Help?
|
|
|
|
- See `/tests/test_common/README.md` for full documentation
|
|
- Check `/docs/test_fixtures_consolidation_report.md` for analysis
|
|
- Review existing tests in `test_common/src/*/tests` for examples
|