diff --git a/docs/plans/2026-02-23-production-hardening-phase2-implementation.md b/docs/plans/2026-02-23-production-hardening-phase2-implementation.md new file mode 100644 index 000000000..f2dd31cb2 --- /dev/null +++ b/docs/plans/2026-02-23-production-hardening-phase2-implementation.md @@ -0,0 +1,1144 @@ +# Production Hardening Phase 2 — Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Eliminate all remaining safety gaps, wire real data into placeholder paths, verify correctness with integration tests, and establish training infrastructure — ready for $100K live trading. + +**Architecture:** Risk-prioritized 4-layer approach executed sequentially: Safety Net → Correctness → Verification → Training Infrastructure. Each task is TDD with explicit test-first steps. + +**Tech Stack:** Rust 1.83+, Candle v0.9.1, nalgebra, sha2, Databento API (Python scripts), safetensors, tokio, tonic (gRPC) + +**Build:** `SQLX_OFFLINE=true cargo check --workspace` (no PostgreSQL required) +**Test:** `SQLX_OFFLINE=true cargo test -p --lib` +**Clippy deny:** `#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::indexing_slicing)]` + +--- + +## Layer 1 — Safety Net (Tasks 1–6) + +### Task 1: Replace `std::process::exit()` in temporal_guard.rs + +**Files:** +- Modify: `ml/src/validation/temporal_guard.rs:294-395` (7 exit calls in test helpers) + +**Context:** The `TemporalGuard` struct itself is correct — it uses `Result`-based error handling. The 7 `std::process::exit(1)` calls are all in test helper functions (`make_test_data()` and test bodies) where `unwrap()` is denied by clippy. They use `unwrap_or_else(|_| std::process::exit(1))` as a workaround. + +**Step 1: Write a test verifying the exit calls are gone** + +```bash +SQLX_OFFLINE=true cargo test -p ml --lib validation::temporal_guard -- --nocapture 2>&1 | head -20 +``` + +Verify current tests pass (baseline). + +**Step 2: Replace exit calls with safe test helpers** + +Replace the `make_test_data()` helper and all test `unwrap_or_else(|_| std::process::exit(1))` patterns with a safe `expect_test_result()` helper that uses `match` + `unreachable!()` (allowed in test-only code), or restructure tests to propagate `Result`: + +```rust +// Instead of: +// TemporalGuard::new(&data, 5).unwrap_or_else(|e| { std::process::exit(1) }); +// Use: +// let guard = TemporalGuard::new(&data, 5).map_err(|e| format!("Guard creation failed: {e}"))?; +// By making test functions return Result<(), Box> +``` + +For each test function in the `tests` module (~6 tests): +1. Change signature to `fn test_name() -> Result<(), Box>` +2. Replace `unwrap_or_else(|_| std::process::exit(1))` with `?` operator +3. Replace `make_test_data()` inner fallback with `?` propagation + +For `make_test_data()`: +```rust +fn make_test_data(n: usize) -> Result> { + let prices: Vec = (0..n).map(|i| 100.0 + i as f64).collect(); + Ok(TimeSeriesData::new(make_timestamps(n), make_features(n, 3), prices)?) +} +``` + +**Step 3: Run tests to verify** + +```bash +SQLX_OFFLINE=true cargo test -p ml --lib validation::temporal_guard -- --nocapture +``` +Expected: All 5 tests pass, 0 `std::process::exit` calls remain. + +**Step 4: Verify no exit calls remain in ml/src/** + +```bash +grep -rn "std::process::exit" ml/src/ +``` +Expected: 0 matches (examples/ and tests/ outside src/ are acceptable for CLI binaries). + +**Step 5: Commit** + +```bash +git add ml/src/validation/temporal_guard.rs +git commit -m "fix(ml): replace std::process::exit with Result propagation in temporal_guard tests" +``` + +--- + +### Task 2: Wire real position data into trading agent + +**Files:** +- Modify: `services/trading_agent_service/src/service.rs:804-809` +- Ref: `services/trading_service/src/core/position_manager.rs` (PositionManager API) + +**Context:** In `service.rs`, the `AssetAllocation` is built with `current_weight: 0.0, current_quantity: 0.0` hardcoded. The TODO comment says "Fetch from live position service / positions table." The service already has a `TradingServiceClient` or position tracking capability elsewhere. + +**Step 1: Read position manager API** + +Read `services/trading_service/src/core/position_manager.rs` to understand the `PositionManager::get_position(symbol)` signature and return type. + +**Step 2: Write a test for real position data wiring** + +In the test module of `service.rs`, add a test that verifies `AssetAllocation` fields reflect a non-zero position when one exists. If no test module exists, add one. + +**Step 3: Wire real position lookup** + +Replace the hardcoded zeros at lines 804-809 with a lookup from whatever position tracking is available in the service's state. The service's `self` should have access to position data. If position data is not available in the service state: +- Add a `positions: HashMap` field (weight, quantity) to the service state +- Populate it from the gRPC `GetPositions` call before allocation +- Use it in the allocation loop + +```rust +// Replace: +current_weight: 0.0, +current_quantity: 0.0, +rebalance_delta: target_quantity, + +// With: +current_weight: positions.get(symbol).map(|p| p.weight).unwrap_or(0.0), +current_quantity: positions.get(symbol).map(|p| p.quantity).unwrap_or(0.0), +rebalance_delta: target_quantity - positions.get(symbol).map(|p| p.quantity).unwrap_or(0.0), +``` + +**Step 4: Run tests** + +```bash +SQLX_OFFLINE=true cargo test -p trading_agent_service --lib -- --nocapture +``` + +**Step 5: Commit** + +```bash +git add services/trading_agent_service/src/service.rs +git commit -m "fix(trading_agent): wire real position data into allocation calculations" +``` + +--- + +### Task 3: Wire real VaR calculation + +**Files:** +- Modify: `services/trading_service/src/services/risk.rs:315-321,415-427` +- Ref: `risk/src/var/` (real VaR calculators) + +**Context:** Line 321 passes `confidence_level * 1_000_000.0` as the notional value to `calculate_marginal_var()`. This is a placeholder — the notional should come from real portfolio value. Line 418 has the same pattern. + +**Step 1: Understand the RiskEngine API** + +Read `risk/src/var/` to understand what `calculate_marginal_var` expects as its `notional` parameter. + +**Step 2: Fix the notional value calculation** + +The `get_va_r` method already calls `self.fetch_positions()` at line 343 and builds a `position_map`. Use the sum of absolute position values as the portfolio notional: + +```rust +// Replace line 321: +// confidence_level * 1_000_000.0, // notional proxy scaled by confidence +// With: +let portfolio_notional: f64 = positions.iter().map(|p| (p.quantity * p.avg_price).abs()).sum(); +let portfolio_notional = if portfolio_notional > 0.0 { portfolio_notional } else { 100_000.0 }; // fallback for empty portfolio +``` + +Similarly fix lines 417-418 for `get_risk_metrics`: +```rust +// Fetch positions before the VaR call +let positions = self.fetch_positions().await; +let portfolio_notional: f64 = positions.iter().map(|p| (p.quantity * p.avg_price).abs()).sum(); +let portfolio_notional = if portfolio_notional > 0.0 { portfolio_notional } else { 100_000.0 }; +``` + +**Step 3: Run tests** + +```bash +SQLX_OFFLINE=true cargo test -p trading_service --lib -- --nocapture +``` + +**Step 4: Commit** + +```bash +git add services/trading_service/src/services/risk.rs +git commit -m "fix(trading_service): wire real portfolio notional into VaR calculations" +``` + +--- + +### Task 4: Kill switch Redis monitoring — document deferral + +**Files:** +- Modify: `risk/src/safety/kill_switch.rs:378-379` + +**Context:** The TODO says to spawn a tokio background task for Redis health checking. However, the local kill switch (AtomicBool) already works for single-process trading. Redis monitoring is only needed for multi-service distributed kill switch, which is a Phase 2 broker integration concern. + +**Step 1: Replace TODO with architectural decision doc comment** + +```rust +// Replace the TODO at line 378 with: +// Note: Redis health monitoring background task is deferred to multi-service deployment. +// The local AtomicBool kill switch provides immediate process-level protection. +// For distributed trading across multiple services, implement: +// 1. Periodic Redis PING with configurable interval (config.health_check_interval) +// 2. On Redis disconnect: log warning, increment failure_count, continue with local-only mode +// 3. On reconnect: re-sync kill switch state from Redis +// See risk/src/safety/unix_socket_kill_switch.rs for the Unix socket alternative. +``` + +**Step 2: Run tests** + +```bash +SQLX_OFFLINE=true cargo test -p risk --lib -- --nocapture +``` + +**Step 3: Commit** + +```bash +git add risk/src/safety/kill_switch.rs +git commit -m "docs(risk): document Redis monitoring deferral for kill switch" +``` + +--- + +### Task 5: GPU OOM detection and CPU fallback + +**Files:** +- Modify: `ml/src/inference.rs:538-579` (RealMLInferenceEngine::load_model) +- Ref: `ml/src/memory_optimization/auto_batch_size.rs:397-448` (detect_gpu_memory) +- Ref: `ml/src/safety/memory_manager.rs:130-190` (MemorySafetyManager) + +**Context:** The `detect_gpu_memory()` function already exists in `auto_batch_size.rs` and returns `(total_mb, free_mb, device_name)` via nvidia-smi. The `MemorySafetyManager` in `safety/memory_manager.rs` already checks GPU memory limits. The issue is that `load_model()` in `inference.rs` tries GPU and hard-fails with `GpuRequired` error if it can't use CUDA. We need it to fall back to CPU instead. + +**Step 1: Write a test for GPU fallback behavior** + +In the inference.rs test module, add: +```rust +#[tokio::test] +async fn test_load_model_falls_back_to_cpu_when_gpu_unavailable() { + // Create engine with "gpu" preference + let config = RealInferenceConfig { + device_preference: "gpu".to_string(), + ..Default::default() + }; + let safety = Arc::new(MLSafetyManager::default()); + let engine = RealMLInferenceEngine::new(config, safety); + + // On CI (no GPU), this should succeed with CPU fallback, not error + // The test validates the fallback path works +} +``` + +**Step 2: Modify load_model to fallback to CPU** + +In `inference.rs:544-558`, change the GPU path: + +```rust +"cuda" | "gpu" => match Device::new_cuda(0) { + Ok(cuda_device) => { + // Check VRAM before committing to GPU + match ml::memory_optimization::auto_batch_size::detect_gpu_memory() { + Ok((_, free_mb, _)) if free_mb > 500.0 => { + info!("Using CUDA device for model: {} (free VRAM: {:.0}MB)", model_id, free_mb); + cuda_device + } + Ok((_, free_mb, _)) => { + warn!( + "GPU VRAM too low ({:.0}MB free), falling back to CPU for model: {}", + free_mb, model_id + ); + Device::Cpu + } + Err(e) => { + warn!("Cannot detect GPU memory ({}), falling back to CPU for model: {}", e, model_id); + Device::Cpu + } + } + } + Err(e) => { + warn!("CUDA not available ({}), falling back to CPU for model: {}", e, model_id); + Device::Cpu + } +}, +``` + +**Step 3: Run tests** + +```bash +SQLX_OFFLINE=true cargo test -p ml --lib inference -- --nocapture +``` + +**Step 4: Verify compilation** + +```bash +SQLX_OFFLINE=true cargo check --workspace +``` + +**Step 5: Commit** + +```bash +git add ml/src/inference.rs +git commit -m "fix(ml): GPU OOM detection with automatic CPU fallback in inference engine" +``` + +--- + +### Task 6: Per-model circuit breakers in ensemble + +**Files:** +- Modify: `ml/src/ensemble/inference_ensemble.rs` +- Test: Same file, `mod tests` + +**Context:** The `InferenceEnsemble` currently skips models that fail `predict()` (line 91-98) but doesn't protect against models returning garbage (NaN direction, NaN confidence, extreme values). A model could return `direction: NaN` or `confidence: -1.0` and it would silently corrupt the weighted average. + +**Step 1: Write failing tests for NaN protection** + +In `inference_ensemble.rs` tests module, add: + +```rust +#[test] +fn test_ensemble_filters_nan_predictions() { + struct NaNAdapter; + impl ModelInferenceAdapter for NaNAdapter { + fn model_name(&self) -> &str { "NaN-model" } + fn predict(&self, _: &FeatureVector) -> MLResult { + Ok(EnsemblePrediction { + model_name: "NaN-model".to_string(), + direction: f64::NAN, + confidence: 0.8, + metadata: PredictionMeta::default(), + }) + } + fn is_ready(&self) -> bool { true } + } + + let adapters: Vec> = vec![ + Box::new(NaNAdapter), + Box::new(DummyAdapter { + name: "Good".to_string(), + direction: 1.0, + confidence: 0.9, + ready: true, + }), + ]; + + let ensemble = InferenceEnsemble::new(adapters); + let pred = ensemble.predict(&make_features()).expect("should succeed with good model"); + // NaN model should be filtered out, only good model contributes + assert!(pred.direction.is_finite(), "direction must be finite"); + assert!(pred.confidence.is_finite(), "confidence must be finite"); +} + +#[test] +fn test_ensemble_filters_extreme_confidence() { + // Model with confidence > 1.0 or < 0.0 should be clamped or skipped + let adapters: Vec> = vec![ + Box::new(DummyAdapter { + name: "Overconfident".to_string(), + direction: 1.0, + confidence: 5.0, // invalid + ready: true, + }), + Box::new(DummyAdapter { + name: "Normal".to_string(), + direction: -1.0, + confidence: 0.7, + ready: true, + }), + ]; + + let ensemble = InferenceEnsemble::new(adapters); + let pred = ensemble.predict(&make_features()).expect("should succeed"); + assert!(pred.confidence <= 1.0, "ensemble confidence should be <= 1.0"); + assert!(pred.confidence >= 0.0, "ensemble confidence should be >= 0.0"); +} +``` + +**Step 2: Run tests to verify they fail** + +```bash +SQLX_OFFLINE=true cargo test -p ml --lib ensemble::inference_ensemble -- --nocapture +``` + +**Step 3: Add prediction validation in the predict loop** + +In `inference_ensemble.rs`, inside the `Ok(pred)` arm of the match (line 78), add validation: + +```rust +Ok(pred) => { + // Circuit breaker: skip predictions with NaN/Inf or out-of-range values + if !pred.direction.is_finite() || !pred.confidence.is_finite() { + tracing::warn!( + model = %model_name, + direction = %pred.direction, + confidence = %pred.confidence, + "Model returned NaN/Inf prediction, skipping (circuit breaker)" + ); + continue; + } + + // Clamp confidence to [0.0, 1.0] + let confidence = pred.confidence.clamp(0.0, 1.0); + + let w = self.weights.get(&model_name).copied().unwrap_or(1.0); + let wc = w * confidence; + weighted_direction_sum += pred.direction * wc; + weight_confidence_sum += wc; + confidence_sum += confidence; + successful_count += 1; + model_names.push(model_name); +} +``` + +**Step 4: Run tests to verify they pass** + +```bash +SQLX_OFFLINE=true cargo test -p ml --lib ensemble::inference_ensemble -- --nocapture +``` +Expected: All tests pass including new ones. + +**Step 5: Commit** + +```bash +git add ml/src/ensemble/inference_ensemble.rs +git commit -m "fix(ml): per-model circuit breakers in ensemble — filter NaN/Inf, clamp confidence" +``` + +--- + +## Layer 2 — Correctness (Tasks 7–12) + +### Task 7: Feature extraction NaN/Inf guards + +**Files:** +- Modify: `services/trading_service/src/services/enhanced_ml.rs` (FeaturePreprocessor::normalize) + +**Context:** The `normalize()` function at line 116-127 does `(value - mean) / std_dev`. If `value` is already NaN/Inf (from upstream feature extraction), the NaN propagates silently through inference. Also, division by zero is possible if `std_dev` becomes 0. + +**Step 1: Write a failing test** + +```rust +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_normalize_nan_returns_zero() { + let preprocessor = FeaturePreprocessor::new(); + let result = preprocessor.normalize("price_momentum", f64::NAN); + assert!(result.is_finite(), "NaN input should produce finite output"); + assert_eq!(result, 0.0); + } + + #[test] + fn test_normalize_inf_returns_zero() { + let preprocessor = FeaturePreprocessor::new(); + let result = preprocessor.normalize("price_momentum", f64::INFINITY); + assert!(result.is_finite(), "Inf input should produce finite output"); + } +} +``` + +**Step 2: Add NaN/Inf guard to normalize()** + +```rust +pub fn normalize(&self, feature_name: &str, value: f64) -> f64 { + // Guard: NaN/Inf inputs produce 0.0 (neutral) + if !value.is_finite() { + tracing::warn!(feature = %feature_name, value = %value, "NaN/Inf feature detected, replacing with 0.0"); + return 0.0; + } + + if let Some(stats) = self.stats.get(feature_name) { + if stats.std_dev > 0.0 { + let normalized = (value - stats.mean) / stats.std_dev; + // Clamp to prevent extreme values from dominating + normalized.clamp(-10.0, 10.0) + } else { + value + } + } else { + value.tanh() + } +} +``` + +**Step 3: Run tests** + +```bash +SQLX_OFFLINE=true cargo test -p trading_service --lib services::enhanced_ml -- --nocapture +``` + +**Step 4: Commit** + +```bash +git add services/trading_service/src/services/enhanced_ml.rs +git commit -m "fix(trading_service): NaN/Inf guards in feature normalization" +``` + +--- + +### Task 8: Model file integrity validation (SHA-256 checksum) + +**Files:** +- Modify: `ml/src/checkpoint/validation.rs` (if exists) or create checksum logic +- Modify: `ml/src/checkpoint/mod.rs` (add checksum on save/load) +- Ref: `ml/Cargo.toml` (sha2 already a dependency) + +**Context:** `ml/Cargo.toml` already has `sha2` as a dependency. The `checkpoint/validation.rs` and `checkpoint/signer.rs` already exist. We need to add a checksum file (`.sha256`) alongside each safetensors file, written during save and verified during load. + +**Step 1: Read existing checkpoint validation and signer modules** + +Read `ml/src/checkpoint/validation.rs` and `ml/src/checkpoint/signer.rs` to understand existing infrastructure. + +**Step 2: Add checksum generation on save** + +In the checkpoint save path, after writing the safetensors file, compute SHA-256 of the file bytes and write a `.sha256` sidecar file: + +```rust +use sha2::{Sha256, Digest}; +use std::io::Write; + +pub fn write_checksum(safetensors_path: &Path) -> Result<(), MLError> { + let bytes = std::fs::read(safetensors_path).map_err(|e| MLError::CheckpointError { + reason: format!("Failed to read file for checksum: {}", e), + })?; + let hash = Sha256::digest(&bytes); + let hex = format!("{:x}", hash); + let checksum_path = safetensors_path.with_extension("sha256"); + std::fs::write(&checksum_path, hex.as_bytes()).map_err(|e| MLError::CheckpointError { + reason: format!("Failed to write checksum: {}", e), + })?; + Ok(()) +} + +pub fn verify_checksum(safetensors_path: &Path) -> Result { + let checksum_path = safetensors_path.with_extension("sha256"); + if !checksum_path.exists() { + // No checksum file = skip verification (backwards compatible) + tracing::warn!("No checksum file for {}, skipping integrity check", safetensors_path.display()); + return Ok(true); + } + let expected = std::fs::read_to_string(&checksum_path).map_err(|e| MLError::CheckpointError { + reason: format!("Failed to read checksum: {}", e), + })?; + let bytes = std::fs::read(safetensors_path).map_err(|e| MLError::CheckpointError { + reason: format!("Failed to read file for verification: {}", e), + })?; + let actual = format!("{:x}", Sha256::digest(&bytes)); + Ok(actual.trim() == expected.trim()) +} +``` + +**Step 3: Write tests** + +```rust +#[test] +fn test_checksum_roundtrip() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("test_model.safetensors"); + std::fs::write(&path, b"fake model data").unwrap(); + + write_checksum(&path).unwrap(); + assert!(verify_checksum(&path).unwrap()); + + // Corrupt the file + std::fs::write(&path, b"corrupted data").unwrap(); + assert!(!verify_checksum(&path).unwrap()); +} +``` + +**Step 4: Run tests** + +```bash +SQLX_OFFLINE=true cargo test -p ml --lib checkpoint -- --nocapture +``` + +**Step 5: Commit** + +```bash +git add ml/src/checkpoint/ +git commit -m "feat(ml): SHA-256 checksum validation for model checkpoint integrity" +``` + +--- + +### Task 9: Model versioning and rollback + +**Files:** +- Modify: `ml/src/model_registry.rs` +- Ref: `ml/src/model_registry/checkpoint_loader.rs` +- Ref: `migrations/021_ml_model_versioning.sql` + +**Context:** A model registry already exists at `ml/src/model_registry.rs` with `checkpoint_loader.rs`. There's also a SQL migration for model versioning. The key gap is: no CLI/API to roll back to a previous version, and no A/B comparison tracking. + +**Step 1: Read existing model registry** + +Read `ml/src/model_registry.rs` and `ml/src/model_registry/checkpoint_loader.rs` to understand current API. + +**Step 2: Add rollback capability** + +Add a `rollback_to_version(model_name: &str, version: &semver::Version)` method that: +1. Looks up the specified version in the registry +2. Loads its checkpoint path +3. Sets it as the "active" version +4. Logs the rollback event + +**Step 3: Write tests and verify** + +```bash +SQLX_OFFLINE=true cargo test -p ml --lib model_registry -- --nocapture +``` + +**Step 4: Commit** + +```bash +git add ml/src/model_registry.rs ml/src/model_registry/ +git commit -m "feat(ml): model version rollback capability in registry" +``` + +--- + +### Task 10: Liquid CfC v2 ensemble integration + +**Files:** +- Merge: `worktree-liquid-cfc-v2` branch to `feat/production-hardening` +- Verify: `ml/src/liquid/adapter.rs` (LiquidInferenceAdapter) +- Modify: `services/trading_service/src/services/enhanced_ml.rs:238-251` (add CfC model type) + +**Context:** The `worktree-liquid-cfc-v2` branch has 10 commits (+15,014/-2,666 lines) adding CfC v2 with `LiquidInferenceAdapter` implementing `ModelInferenceAdapter` and `LiquidTrainableAdapter` implementing `UnifiedTrainable`. The adapter files are in `ml/src/liquid/`. + +**Step 1: Merge the liquid-cfc-v2 branch** + +```bash +cd /home/jgrusewski/Work/foxhunt/.claude/worktrees/production-hardening +git merge worktree-liquid-cfc-v2 --no-ff -m "feat(ml): merge liquid CfC v2 ensemble model" +``` + +Resolve any conflicts (likely in `ml/src/lib.rs` module declarations and `Cargo.toml`). + +**Step 2: Verify compilation** + +```bash +SQLX_OFFLINE=true cargo check --workspace +``` + +**Step 3: Add CfC to model type detection in enhanced_ml.rs** + +In `services/trading_service/src/services/enhanced_ml.rs`, around line 238-251, add: + +```rust +} else if model_id.contains("CFC") || model_id.contains("cfc") || model_id.contains("liquid") { + "CFC" +``` + +And add the corresponding match arm: +```rust +"CFC" | "LIQUID" => { + let cfc_model = RealCfCModel::from_checkpoint(model_id.to_string(), checkpoint_path) + .map_err(|e| Status::internal(format!("Failed to load CfC model: {}", e)))?; + Arc::new(cfc_model) as Arc +}, +``` + +**Step 4: Run tests** + +```bash +SQLX_OFFLINE=true cargo test -p ml --lib liquid -- --nocapture +SQLX_OFFLINE=true cargo test -p trading_service --lib -- --nocapture +``` + +**Step 5: Commit** + +```bash +git add -A +git commit -m "feat(trading_service): register CfC v2 model in enhanced ML service" +``` + +--- + +### Task 11: Portfolio correlation matrix for Markowitz + +**Files:** +- Modify: `services/trading_agent_service/src/allocation.rs:129-178` + +**Context:** The `mean_variance()` method builds a diagonal covariance matrix (line 154-156). The doc comment at lines 140-153 already describes the implementation plan. `nalgebra::DMatrix` is already imported. + +**Step 1: Write a test for correlation-aware allocation** + +```rust +#[test] +fn test_mean_variance_with_correlations_differs_from_diagonal() { + let allocator = PortfolioAllocator::new(AllocationMethod::MeanVariance { lambda: 1.0 }); + let assets = vec![ + AssetInfo { symbol: "A".into(), expected_return: 0.10, volatility: 0.20, ..Default::default() }, + AssetInfo { symbol: "B".into(), expected_return: 0.08, volatility: 0.15, ..Default::default() }, + ]; + + // With correlation = 0.0 (diagonal) vs correlation = 0.9 (highly correlated) + // allocations should differ + let diagonal_alloc = allocator.allocate(&assets, Decimal::from(100_000))?; + // ... compare with correlated version +} +``` + +**Step 2: Add correlations parameter to mean_variance** + +Extend `AssetInfo` or add an optional `correlations: Option<&DMatrix>` parameter. Build full covariance matrix when correlations are provided. + +**Step 3: Run tests** + +```bash +SQLX_OFFLINE=true cargo test -p trading_agent_service --lib allocation -- --nocapture +``` + +**Step 4: Commit** + +```bash +git add services/trading_agent_service/src/allocation.rs +git commit -m "feat(trading_agent): correlation matrix support in Markowitz allocation" +``` + +--- + +### Task 12: Hyperopt Phase B — ensemble-level optimization + +**Files:** +- Modify: `ml/src/hyperopt/optimizer.rs` +- Ref: `ml/src/hyperopt/adapters/` (existing per-model adapters) + +**Context:** The `ArgminOptimizer` works for individual models. Phase B requires optimizing ensemble weights alongside per-model hyperparameters. This means treating ensemble weights as additional continuous parameters in the PSO search space. + +**Step 1: Read existing optimizer traits** + +Read `ml/src/hyperopt/traits.rs` to understand `ParameterSpace` and `HyperparameterOptimizable`. + +**Step 2: Add ensemble weight parameters to the search space** + +Create an `EnsembleHyperoptAdapter` that wraps N model adapters and adds N weight parameters (one per model) to the continuous parameter space: + +```rust +pub struct EnsembleHyperoptAdapter { + model_adapters: Vec>, + model_names: Vec, +} +``` + +The combined parameter space = concat of all model parameter spaces + N weight params in [0, 1]. + +**Step 3: Write tests** + +```rust +#[test] +fn test_ensemble_hyperopt_parameter_space() { + // Verify combined parameter space has correct dimension +} +``` + +**Step 4: Run tests** + +```bash +SQLX_OFFLINE=true cargo test -p ml --lib hyperopt -- --nocapture +``` + +**Step 5: Commit** + +```bash +git add ml/src/hyperopt/ +git commit -m "feat(ml): ensemble-level hyperopt with joint model weight optimization" +``` + +--- + +## Layer 3 — Verification (Tasks 13–16) + +### Task 13: Critical path tests — risk.rs + +**Files:** +- Create: `services/trading_service/tests/risk_service_tests.rs` +- Ref: `services/trading_service/src/services/risk.rs` + +**Context:** `risk.rs` has 0 unit tests. It implements `get_va_r`, `get_risk_metrics`, `get_risk_limits`, `update_risk_limits`. These are critical for $100K trading — VaR must be correctly calculated. + +**Step 1: Write tests for VaR calculation** + +```rust +#[tokio::test] +async fn test_get_var_returns_positive_value() { ... } + +#[tokio::test] +async fn test_get_var_with_symbols_returns_per_symbol_breakdown() { ... } + +#[tokio::test] +async fn test_risk_metrics_returns_var_1d_5d_10d() { ... } + +#[tokio::test] +async fn test_risk_limits_enforcement() { ... } +``` + +Write at least 8 tests covering: +- VaR is positive for non-empty portfolio +- Per-symbol VaR contributions sum to ~100% +- Risk limits reject orders exceeding VaR threshold +- Fallback path works when RiskEngine returns error +- Empty portfolio returns 0 VaR + +**Step 2: Run tests** + +```bash +SQLX_OFFLINE=true cargo test -p trading_service -- risk_service_tests --nocapture +``` + +**Step 3: Commit** + +```bash +git add services/trading_service/tests/risk_service_tests.rs +git commit -m "test(trading_service): risk service tests for VaR and risk limits" +``` + +--- + +### Task 14: Critical path tests — enhanced_ml.rs and allocation.rs + +**Files:** +- Create: `services/trading_service/tests/enhanced_ml_tests.rs` +- Create: `services/trading_agent_service/tests/allocation_tests.rs` + +**Context:** Both files have 0 dedicated tests. `enhanced_ml.rs` handles model loading and inference. `allocation.rs` handles portfolio allocation with 5 strategies. + +**Step 1: Write tests for enhanced_ml.rs** + +```rust +#[test] +fn test_feature_preprocessor_normalizes_known_features() { ... } +#[test] +fn test_feature_preprocessor_handles_unknown_features() { ... } +#[test] +fn test_feature_type_classification() { ... } +#[test] +fn test_ensemble_config_defaults() { ... } +``` + +**Step 2: Write tests for allocation.rs** + +```rust +#[test] +fn test_equal_weight_allocates_evenly() { ... } +#[test] +fn test_risk_parity_favors_low_volatility() { ... } +#[test] +fn test_mean_variance_respects_lambda() { ... } +#[test] +fn test_kelly_criterion_with_half_kelly() { ... } +#[test] +fn test_allocation_empty_assets_returns_empty() { ... } +#[test] +fn test_allocation_single_asset_gets_full_capital() { ... } +``` + +**Step 3: Run tests** + +```bash +SQLX_OFFLINE=true cargo test -p trading_service -- enhanced_ml_tests --nocapture +SQLX_OFFLINE=true cargo test -p trading_agent_service -- allocation_tests --nocapture +``` + +**Step 4: Commit** + +```bash +git add services/trading_service/tests/enhanced_ml_tests.rs services/trading_agent_service/tests/allocation_tests.rs +git commit -m "test: critical path tests for enhanced ML and portfolio allocation" +``` + +--- + +### Task 15: Replace vacuous assert!(true) tests in critical paths + +**Files:** +- Modify: `trading_engine/src/tests/performance_validation.rs` (4 instances) +- Modify: `trading_engine/src/trading/engine.rs` (2 instances) +- Modify: `risk/src/tests/risk_tests.rs` (2 instances) + +**Context:** These files have tests like: +```rust +assert!(true); // If we get here, creation succeeded +``` +These should test the actual created value's properties. + +**Step 1: Replace each assert!(true) with meaningful assertions** + +For example, in `performance_validation.rs:74`: +```rust +// Replace: assert!(true); // If we get here, creation succeeded +// With: assert!(thing.some_field > 0); or assert_eq!(thing.state, Expected); +``` + +Read each test to understand what was created and what properties should be verified. + +**Step 2: Run tests** + +```bash +SQLX_OFFLINE=true cargo test -p trading_engine --lib -- --nocapture +SQLX_OFFLINE=true cargo test -p risk --lib -- --nocapture +``` + +**Step 3: Commit** + +```bash +git add trading_engine/ risk/ +git commit -m "test: replace vacuous assert!(true) with meaningful assertions in critical paths" +``` + +--- + +### Task 16: Integration tests — ML→Order→Fill and Risk→KillSwitch + +**Files:** +- Create: `tests/integration/ml_order_fill_test.rs` +- Create: `tests/integration/risk_cascade_killswitch_test.rs` + +**Context:** These end-to-end integration tests verify the two most critical paths: +1. Market data → Feature extraction → Model inference → Ensemble vote → Order generation → Simulated fill +2. Risk limit violation → Circuit breaker trip → Kill switch activation → Trading halt + +**Step 1: Write ML→Order→Fill integration test** + +```rust +#[tokio::test] +#[ignore = "Integration test requiring model checkpoints"] +async fn test_ml_to_order_pipeline() { + // 1. Create mock market data (OHLCV bars) + // 2. Extract features (51-dim FeatureVector) + // 3. Create ensemble with DummyAdapters + // 4. Get ensemble prediction + // 5. Generate order from prediction (direction → Buy/Sell, confidence → size) + // 6. Validate order fields (symbol, side, quantity > 0, valid price) +} +``` + +**Step 2: Write Risk→KillSwitch integration test** + +```rust +#[tokio::test] +async fn test_risk_cascade_to_killswitch() { + // 1. Create KillSwitch (local, no Redis) + // 2. Create RiskManager with tight limits + // 3. Submit order exceeding VaR limit + // 4. Verify circuit breaker trips + // 5. Verify kill switch activates + // 6. Verify subsequent orders rejected +} +``` + +**Step 3: Run tests** + +```bash +SQLX_OFFLINE=true cargo test --test ml_order_fill_test -- --nocapture +SQLX_OFFLINE=true cargo test --test risk_cascade_killswitch_test -- --nocapture +``` + +**Step 4: Commit** + +```bash +git add tests/integration/ +git commit -m "test: end-to-end integration tests for ML→Order and Risk→KillSwitch pipelines" +``` + +--- + +## Layer 4 — Training Infrastructure (Tasks 17–19) + +### Task 17: Training pipeline safety — NaN gradient detection + +**Files:** +- Modify: `ml/src/trainers/dqn/trainer.rs` (DqnTrainer training loop) +- Modify: `ml/src/ppo/ppo.rs` (PPO training loop) +- Modify: `ml/src/trainers/tft/trainer.rs` (TFT training loop) + +**Context:** During training, if a gradient becomes NaN (from exploding gradients, bad data, or numerical instability), the model silently becomes garbage. We need NaN detection after each backward pass, auto-halting training and restoring the last good checkpoint. + +**Step 1: Create a gradient NaN check utility** + +In `ml/src/training/mod.rs` or appropriate location: + +```rust +/// Check if any gradient in the GradStore contains NaN or Inf. +/// Returns the name of the first offending parameter, if any. +pub fn check_gradients_finite(grads: &candle_nn::var_map::GradStore) -> Option { + // Iterate through gradient tensors and check for NaN/Inf + // Return parameter name if found +} +``` + +**Step 2: Add gradient check after loss.backward() in DQN trainer** + +In the DQN training loop, after `loss.backward()`: +```rust +let grads = loss.backward()?; +if let Some(bad_param) = check_gradients_finite(&grads) { + warn!("NaN gradient detected in parameter: {}. Halting training, restoring last checkpoint.", bad_param); + // Restore last good checkpoint + // Return early with error + return Err(MLError::TrainingError { reason: format!("NaN gradient in {}", bad_param) }); +} +``` + +**Step 3: Write tests** + +```rust +#[test] +fn test_nan_gradient_detection() { + // Create a tensor with NaN gradient + // Verify check_gradients_finite returns the parameter name +} +``` + +**Step 4: Run tests** + +```bash +SQLX_OFFLINE=true cargo test -p ml --lib training -- --nocapture +``` + +**Step 5: Commit** + +```bash +git add ml/src/training/ ml/src/trainers/ +git commit -m "feat(ml): NaN gradient detection with auto-halt during training" +``` + +--- + +### Task 18: Databento data download pipeline + +**Files:** +- Modify: `scripts/python/data/download_ml_training_data.py` (or create new) +- Ref: `data/src/providers/databento/client.rs` (Rust Databento client) +- Ref: `scripts/python/data/download_es_databento.py` (existing download script) + +**Context:** Multiple Databento download scripts already exist in `scripts/python/data/`. The Rust `data` crate has a full Databento client. The gap is a unified download script that: +1. Downloads OHLCV (1-min) for specified symbols and date ranges +2. Downloads MBP-10 for specified symbols and date ranges +3. Stores in DBN format in `data/databento/` directory +4. Validates downloaded data quality + +**Step 1: Read existing download scripts** + +Read `scripts/python/data/download_es_databento.py` and `scripts/python/data/download_ml_training_data.py` to understand the existing pattern. + +**Step 2: Create unified download script** + +Create `scripts/python/data/download_training_dataset.py`: +- Accept CLI args: `--symbols ES.FUT,NQ.FUT,GC.FUT --start 2021-01-01 --end 2025-12-31 --schemas ohlcv-1m,mbp-10` +- Use `databento` Python client library +- Save to `data/databento/{symbol}/{schema}/` directory structure +- Print data size and bar counts after download +- Validate: no gaps > 1 trading day, no negative prices, no zero-volume bars + +**Step 3: Document data requirements** + +Add a comment block at the top documenting: +- Minimum: 2 years OHLCV for regime diversity +- Recommended: 5 years OHLCV + 1 year MBP-10 +- Symbols: ES, NQ, CL, GC, 6E (5 liquid futures) +- Expected sizes: ~50MB/symbol/year OHLCV, ~2-5GB/symbol/year MBP-10 + +**Step 4: Test with dry-run** + +```bash +python scripts/python/data/download_training_dataset.py --symbols ES.FUT --start 2024-01-01 --end 2024-01-07 --schemas ohlcv-1m --dry-run +``` + +**Step 5: Commit** + +```bash +git add scripts/python/data/download_training_dataset.py +git commit -m "feat(data): unified Databento training data download pipeline" +``` + +--- + +### Task 19: GPU training launcher and Scaleway config + +**Files:** +- Create: `scripts/train_launcher.sh` +- Create: `terraform/scaleway/training_instance.tf` (or `docs/infra/scaleway-gpu-training.md`) + +**Context:** Local GPU is RTX 3050 Ti (4GB VRAM, max batch 230 for PPO). For production training, need Scaleway GPU instances (e.g., GPU-3070-S or L4 instances). The launcher should detect available GPU and route to local or cloud training. + +**Step 1: Create training launcher script** + +`scripts/train_launcher.sh`: +```bash +#!/bin/bash +# Training launcher — detects GPU and selects local or cloud path +# +# Usage: ./train_launcher.sh --model dqn --data data/databento/ES.FUT/ohlcv-1m/ [--cloud] + +# Detect GPU +if nvidia-smi &>/dev/null; then + VRAM_MB=$(nvidia-smi --query-gpu=memory.total --format=csv,noheader,nounits | head -1) + echo "Local GPU detected: ${VRAM_MB}MB VRAM" + + if [[ "${1}" == "--cloud" ]] || [[ "${VRAM_MB}" -lt 8000 ]]; then + echo "Routing to cloud training (VRAM < 8GB or --cloud flag)" + # SSH to Scaleway instance and run training + # ssh training@gpu.fxhnt.ai "cd /opt/foxhunt && cargo run --release -p ml --example train_${MODEL}" + else + echo "Running local training" + SQLX_OFFLINE=true cargo run --release -p ml --example "train_${MODEL}" -- "$@" + fi +else + echo "No GPU detected, routing to cloud training" +fi +``` + +**Step 2: Document Scaleway GPU instance configuration** + +Create `docs/infra/scaleway-gpu-training.md` documenting: +- Instance type: GPU-3070-S (8GB VRAM) or L4 (24GB VRAM) +- Setup: Rust toolchain, CUDA, Foxhunt checkout +- Model artifact sync: scp/rsync safetensors files between local and cloud +- Cost estimate per training run + +**Step 3: Commit** + +```bash +git add scripts/train_launcher.sh docs/infra/scaleway-gpu-training.md +git commit -m "feat(infra): GPU training launcher with local/cloud routing" +``` + +--- + +## Summary + +| Task | Layer | Description | Risk | +|------|-------|-------------|------| +| 1 | Safety | Remove std::process::exit from temporal_guard | HIGH | +| 2 | Safety | Wire real position data | CRITICAL | +| 3 | Safety | Wire real VaR calculation | CRITICAL | +| 4 | Safety | Document kill switch Redis deferral | HIGH | +| 5 | Safety | GPU OOM detection + CPU fallback | HIGH | +| 6 | Safety | Per-model circuit breakers in ensemble | HIGH | +| 7 | Correctness | Feature NaN/Inf guards | HIGH | +| 8 | Correctness | Model file SHA-256 checksum | MEDIUM | +| 9 | Correctness | Model version rollback | MEDIUM | +| 10 | Correctness | Liquid CfC v2 ensemble merge | LOW | +| 11 | Correctness | Portfolio correlation matrix | MEDIUM | +| 12 | Correctness | Ensemble-level hyperopt | MEDIUM | +| 13 | Verification | Risk service tests | HIGH | +| 14 | Verification | Enhanced ML + allocation tests | HIGH | +| 15 | Verification | Replace assert!(true) tests | MEDIUM | +| 16 | Verification | E2E integration tests | HIGH | +| 17 | Training | NaN gradient detection | HIGH | +| 18 | Training | Databento download pipeline | MEDIUM | +| 19 | Training | GPU launcher + Scaleway config | MEDIUM |