feat(ml): WAVE 29 DQN Codebase Cleanup & Refactoring Campaign

BREAKING CHANGES: - Removed orphaned dqn.rs monolithic trainer (4,975 lines) - Removed orphaned dqn_ensemble.rs module (816 lines) - Removed orphaned tft.rs and tft_complete_int8_integration_test.rs - TFT trainer split into modular directory structure DQN Module Refactoring: - Split trainers/dqn.rs into modular structure (config.rs, statistics.rs, trainer.rs) - Fixed hyperopt 39D search space (continuous params only) - Boolean flags (use_dueling, use_double_dqn, use_per, use_noisy_nets) are now FIXED architectural decisions - use_distributional defaults to false (Candle BUG #36 - scatter_add gradient issues) Clean Module Structure: - ml/src/trainers/dqn/ directory with proper mod.rs exports - ml/src/trainers/tft/ directory with config.rs, types.rs, model.rs, trainer.rs, tests.rs - All P0 features validated: TD-error clamping, batch diversity, LR scheduler, priority staleness Documentation: - Added comprehensive docs in docs/codebase-cleanup/ - ADR-001 for DQN refactoring decisions - Rainbow DQN component matrix and quick reference guides Build Status: Compiles with zero errors 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-27 23:46:13 +01:00
parent 2c1acda2f3
commit 2df1ea92e1
763 changed files with 247870 additions and 1714 deletions
--- a/docs/test_fixtures_migration_guide.md
+++ b/docs/test_fixtures_migration_guide.md
@@ -0,0 +1,312 @@
+# Test Fixtures Migration Guide
+
+Quick reference for migrating test files to use the new `test_common` crate.
+
+## Quick Start
+
+### 1. Add Dependency
+
+In your test crate's `Cargo.toml`:
+
+```toml
+[dev-dependencies]
+test_common = { path = "../../tests/test_common" }
+```
+
+### 2. Common Migrations
+
+#### Database Setup
+
+**Before:**
+```rust
+async fn setup_test_db() -> PgPool {
+    let database_url = std::env::var("DATABASE_URL")
+        .unwrap_or_else(|_| "postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt".to_string());
+    PgPool::connect(&database_url).await.unwrap()
+}
+
+#[tokio::test]
+async fn my_test() {
+    let pool = setup_test_db().await;
+    // ...
+}
+```
+
+**After:**
+```rust
+use test_common::TestDb;
+
+#[tokio::test]
+async fn my_test() {
+    let db = TestDb::with_migrations().await;
+    let pool = db.pool();
+    // ...
+}
+```
+
+#### Mock Orders
+
+**Before:**
+```rust
+fn create_mock_order() -> Order {
+    Order {
+        id: uuid::Uuid::new_v4().to_string(),
+        symbol: "AAPL".to_string(),
+        quantity: 100.0,
+        price: Some(150.0),
+        side: Side::Buy,
+        order_type: OrderType::Limit,
+        status: OrderStatus::Pending,
+        timestamp: Utc::now(),
+        filled_quantity: 0.0,
+        average_fill_price: None,
+    }
+}
+```
+
+**After:**
+```rust
+use test_common::MockOrderBuilder;
+
+let order = MockOrderBuilder::new()
+    .symbol("AAPL")
+    .buy()
+    .quantity(100.0)
+    .limit_price(150.0)
+    .build();
+```
+
+#### OHLCV Bars
+
+**Before:**
+```rust
+fn create_mock_bars(count: usize) -> Vec<OHLCVBar> {
+    let mut bars = Vec::with_capacity(count);
+    let base_price = 100.0;
+    let base_time = chrono::Utc::now();
+
+    for i in 0..count {
+        let change = ((i as f64 / 100.0).sin() * 0.02);
+        let price = base_price * (1.0 + change);
+
+        bars.push(OHLCVBar {
+            timestamp: base_time + chrono::Duration::minutes((i * 5) as i64),
+            open: price,
+            high: price * 1.01,
+            low: price * 0.99,
+            close: price,
+            volume: 1000 + (i * 10) as u64,
+        });
+    }
+
+    bars
+}
+```
+
+**After:**
+```rust
+use test_common::generate_ohlcv_bars;
+
+let bars = generate_ohlcv_bars(100, 100.0);
+```
+
+#### Crisis Scenarios (VaR Testing)
+
+**Before:**
+```rust
+let crisis_returns = vec![
+    dec!(-0.08), dec!(-0.10), dec!(-0.07), dec!(-0.12),
+    // ... 16 more lines
+];
+```
+
+**After:**
+```rust
+use test_common::fixtures::market_data::generate_crisis_returns;
+
+let crisis_returns = generate_crisis_returns();
+```
+
+## Migration Checklist
+
+For each test file:
+
+- [ ] Add `test_common` to `Cargo.toml`
+- [ ] Replace `setup_test_db()` with `TestDb`
+- [ ] Replace `create_mock_order()` with `MockOrderBuilder`
+- [ ] Replace `create_mock_bars()` with `generate_ohlcv_bars()`
+- [ ] Replace crisis scenarios with `generate_*_returns()`
+- [ ] Replace custom assertions with `test_common::assert_*!()`
+- [ ] Remove old helper functions
+- [ ] Run tests: `cargo test`
+- [ ] Count LOC saved
+
+## Examples by Test Type
+
+### ML Tests
+
+```rust
+// Before: 60+ LOC of setup
+async fn setup_test_orchestrator() -> Arc<TrainingOrchestrator> {
+    let ml_config = MLConfig::default();
+    let db_config = create_test_database_config().await;
+    // ... 50 more lines
+}
+
+// After: 8 LOC
+use test_common::{TestDb, fixtures::config::MLTestConfig};
+
+let db = TestDb::with_migrations().await;
+let config = MLTestConfig::default_test();
+```
+
+### Risk Tests
+
+```rust
+// Before: 40+ LOC
+fn create_crisis_scenario() -> Vec<Decimal> {
+    vec![
+        dec!(-0.08), dec!(-0.10), // ... 18 more
+    ]
+}
+
+fn assert_var_valid(returns: &[Decimal], var: Decimal) {
+    let exceedances = returns.iter()
+        .filter(|&&r| r.abs() > var)
+        .count();
+    assert!(exceedances <= 2);
+}
+
+// After: 5 LOC
+use test_common::{
+    fixtures::market_data::generate_crisis_returns,
+    assertions::assert_var_exceedances,
+};
+
+let returns = generate_crisis_returns();
+assert_var_exceedances(&returns, var, 0.99, 0.1);
+```
+
+### Integration Tests
+
+```rust
+// Before: 80+ LOC of database + orchestrator setup
+
+// After: 15 LOC
+use test_common::{TestDb, generate_ohlcv_bars, MockOrderBuilder};
+
+#[tokio::test]
+async fn end_to_end_workflow() {
+    let db = TestDb::with_migrations().await;
+    let bars = generate_ohlcv_bars(100, 150.0);
+    let order = MockOrderBuilder::new()
+        .symbol("AAPL")
+        .buy()
+        .build();
+
+    // Test logic...
+}
+```
+
+## Common Patterns
+
+### Builder Pattern
+
+All builders follow the same pattern:
+
+```rust
+let result = Builder::new()
+    .property1(value1)
+    .property2(value2)
+    .build();
+```
+
+### Assertions
+
+Domain-specific assertions:
+
+```rust
+// Percentage tolerance
+assert_approx_eq!(actual, expected, 0.02); // 2% tolerance
+
+// OHLC validation
+assert_ohlc_valid(open, high, low, close);
+
+// VaR exceedances
+assert_var_exceedances(&returns, var, confidence, tolerance);
+
+// P&L validation
+assert_pnl(entry_price, current_price, quantity, expected, tolerance);
+```
+
+## Troubleshooting
+
+### Import Errors
+
+```rust
+// If you get "can't find crate test_common"
+// Check Cargo.toml has correct path:
+[dev-dependencies]
+test_common = { path = "../../tests/test_common" }
+```
+
+### Type Mismatches
+
+```rust
+// MockOrder vs actual Order type
+// The mock is for testing only, map to your domain type:
+let mock = MockOrderBuilder::new().build();
+let domain_order = Order {
+    id: mock.id,
+    symbol: Symbol::from_str(&mock.symbol).unwrap(),
+    // ... map other fields
+};
+```
+
+### Database Connection Issues
+
+```rust
+// If tests fail with connection errors:
+// 1. Ensure DATABASE_URL is set
+// 2. Ensure PostgreSQL is running
+// 3. Check connection pool limits
+
+// Each TestDb creates isolated schema
+let db = TestDb::with_migrations().await;
+// Schema is automatically cleaned up on drop
+```
+
+## Migration Progress Tracking
+
+Create a tracking document:
+
+```markdown
+## Test Migration Progress
+
+### ML Training Service (18 files)
+- [ ] checkpoint_manager_tests.rs (~30 LOC saved)
+- [ ] job_tracker_test.rs (~25 LOC saved)
+- [ ] integration_tests.rs (~40 LOC saved)
+... (15 more)
+
+### Risk Tests (18 files)
+- [ ] risk_comprehensive_tests.rs (~80 LOC saved)
+- [ ] var_calculations_tests.rs (~60 LOC saved)
+... (16 more)
+
+Total Estimated Savings: XXX LOC
+```
+
+## Best Practices
+
+1. **Migrate by directory**: Complete one directory before moving to next
+2. **Run tests frequently**: After each migration
+3. **Commit often**: One file or small group per commit
+4. **Document savings**: Track LOC removed in commit messages
+5. **Update imports**: Remove unused imports after migration
+
+## Need Help?
+
+- See `/tests/test_common/README.md` for full documentation
+- Check `/docs/test_fixtures_consolidation_report.md` for analysis
+- Review existing tests in `test_common/src/*/tests` for examples