diff --git a/docs/plans/2026-02-23-production-hardening-phase2-design.md b/docs/plans/2026-02-23-production-hardening-phase2-design.md new file mode 100644 index 000000000..f78e4ceb4 --- /dev/null +++ b/docs/plans/2026-02-23-production-hardening-phase2-design.md @@ -0,0 +1,199 @@ +# Production Hardening Phase 2 — $100K Live Trading Readiness + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Eliminate all remaining safety gaps, wire real data into placeholder paths, verify correctness with integration tests, and establish training infrastructure — making the system ready for $100K live trading. + +**Architecture:** Risk-prioritized 4-layer approach: Safety Net (crash prevention) → Correctness (accurate calculations) → Verification (test coverage) → Training Infrastructure (GPU strategy + data pipeline). The liquid-cfc-v2 ensemble extension is integrated as part of Layer 2 correctness work. + +**Tech Stack:** Rust (37+ crate workspace), Candle v0.9.1 (ML), Databento (market data), Scaleway (cloud GPU), safetensors (model checkpoints) + +--- + +## Audit Context + +### What Was Done (Phase 1) +- 53-task production hardening merged to main (+2,952/-1,023 lines across 52 files) +- 80+ TODO/FIXME items resolved, all clippy deny rules enforced +- 4,234+ tests passing across 30 crate targets, 0 warnings + +### What the Audit Found +Four parallel agents audited the codebase and identified: +- **92 remaining TODOs** — 4 CRITICAL, 3 HIGH, 8 MEDIUM +- **7 `std::process::exit()` calls** in temporal_guard.rs (panic-equivalent) +- **28 critical files with 0 tests**, 49 vacuous `assert!(true)` tests, 107 ignored tests +- **Model loading has no integrity checks**, ensemble has no per-model circuit breakers +- **GPU OOM = hard crash** (no runtime VRAM monitoring) + +--- + +## Layer 1 — Safety Net (Crash Prevention) + +**Rationale:** These issues can crash the system or cause unrecoverable failures during live trading. Fix first. + +### 1.1 Replace `std::process::exit()` in temporal_guard.rs +- **File:** `ml/src/validation/temporal_guard.rs` +- **Problem:** 7 calls to `std::process::exit(1)` — kills the entire process without cleanup +- **Fix:** Replace with `Result`-based error propagation using a new `TemporalGuardError` enum +- **Risk:** HIGH — process exit during trading = lost positions, no graceful shutdown + +### 1.2 Wire real position data into trading agent +- **File:** `services/trading_agent_service/src/service.rs:804-809` +- **Problem:** Position data hardcoded to `current_weight: 0.0, current_quantity: 0.0` +- **Fix:** Query position manager for real portfolio weights and quantities +- **Risk:** CRITICAL — ensemble allocations are meaningless without real position data + +### 1.3 Wire real VaR calculation +- **Files:** `services/trading_service/src/services/risk.rs:315,415` +- **Problem:** VaR uses placeholder formula `confidence_level * 1_000_000.0` +- **Fix:** Wire to the real VaR calculator in `risk/src/var/` which already has parametric, historical, and Monte Carlo methods +- **Risk:** CRITICAL — risk limits are not enforced if VaR is fake + +### 1.4 Kill switch Redis monitoring +- **File:** `risk/src/safety/kill_switch.rs:378` +- **Problem:** Redis monitoring task comment says it should be spawned but never is +- **Fix:** Spawn the monitoring task in `start()`, or document that Redis monitoring is deferred and the local kill switch is sufficient for Phase 1 +- **Risk:** HIGH — distributed kill switch won't propagate across services + +### 1.5 GPU OOM detection and CPU fallback +- **Files:** `ml/src/inference/inference.rs`, `ml/src/inference/inference_engine.rs` +- **Problem:** GPU out-of-memory = hard crash, no runtime VRAM monitoring +- **Fix:** Add VRAM usage check before GPU inference, fall back to CPU with warning log if VRAM > 80% threshold +- **Risk:** HIGH — GPU OOM during live trading = system crash + +### 1.6 Per-model circuit breakers in ensemble +- **File:** `ml/src/inference/inference_ensemble.rs` +- **Problem:** If one model returns garbage (NaN, extreme values), it contaminates the ensemble vote +- **Fix:** Add per-model circuit breaker that trips on NaN, repeated identical outputs, or extreme value divergence; ensemble continues with remaining healthy models +- **Risk:** HIGH — one bad model can cause the entire ensemble to generate bad trades + +--- + +## Layer 2 — Correctness (Accurate Calculations) + +### 2.1 Portfolio correlation matrix for Markowitz allocation +- **File:** `services/trading_agent_service/src/allocation.rs` +- **Problem:** Uses diagonal covariance (ignores correlations between assets) +- **Fix:** Implement rolling correlation matrix from historical returns, use in Markowitz optimization +- **Risk:** MEDIUM — suboptimal allocation but not dangerous (diagonal is conservative) + +### 2.2 Feature extraction NaN/Inf guards +- **File:** `services/trading_service/src/services/enhanced_ml.rs` +- **Problem:** Feature extraction can produce NaN/Inf from division by zero (e.g., zero volume, zero range), which propagates through model inference +- **Fix:** Add NaN/Inf check after feature extraction, before normalization. Replace with 0.0 and log warning. +- **Risk:** HIGH — NaN in model input = NaN in output = unpredictable trades + +### 2.3 Model file integrity validation +- **Problem:** Model files loaded from disk with no integrity verification — corrupted file = silent bad predictions +- **Fix:** SHA-256 checksum stored alongside safetensors files, verified on load. Schema validation ensures tensor shapes match expected architecture. +- **Risk:** MEDIUM — unlikely but catastrophic if it happens + +### 2.4 Model versioning and rollback +- **Problem:** No way to roll back to a previous model version if a new one performs poorly +- **Fix:** Model registry with version tracking, A/B comparison metrics, and one-command rollback +- **Risk:** MEDIUM — operational risk during model updates + +### 2.5 Liquid CfC v2 ensemble integration +- **Branch:** `worktree-liquid-cfc-v2` (10 commits, +15,014/-2,666 lines) +- **Status:** LiquidInferenceAdapter and LiquidTrainableAdapter already implemented +- **Work needed:** + - Merge liquid-cfc-v2 branch to main + - Register CfC adapter in EnsembleCoordinator (alongside DQN/PPO/TFT/Mamba2) + - Add CfC to hyperopt adapter registry + - Verify CfC inference latency is within ensemble budget (<10ms) + - Add CfC-specific circuit breaker configuration +- **Risk:** LOW — additive change, ensemble already handles N models + +### 2.6 Hyperopt Phase B unblocking +- **File:** `ml/src/hyperopt/optimizer.rs` +- **Problem:** Hyperopt works for individual models but multi-model orchestration (Phase B) is blocked +- **Fix:** Wire ensemble-level hyperopt that optimizes model weights and per-model hyperparameters jointly +- **Risk:** MEDIUM — training without hyperopt = suboptimal model configurations + +--- + +## Layer 3 — Verification (Test Coverage) + +### 3.1 Execution path integration tests (~53 tests) +- **Target files (0-test critical files):** + - `services/trading_service/src/services/risk.rs` (VaR, risk limits) + - `services/trading_service/src/services/enhanced_ml.rs` (feature extraction, inference) + - `services/trading_agent_service/src/service.rs` (allocation, order generation) + - `services/trading_agent_service/src/allocation.rs` (Markowitz optimization) + - `services/trading_service/src/core/execution_engine.rs` (order execution) +- **Approach:** Property-based tests for numerical code, scenario tests for trading logic + +### 3.2 Replace vacuous `assert!(true)` tests +- **Count:** 49 tests that just `assert!(true)` or test trivial construction +- **Fix:** Replace each with meaningful assertions testing actual behavior +- **Priority:** Focus on tests in critical paths first (risk, ML, execution) + +### 3.3 Integration test: ML → Order → Fill +- **Scope:** End-to-end test from feature extraction through model inference, ensemble voting, order generation, and simulated fill +- **Purpose:** Verify the complete trading pipeline produces valid orders from market data + +### 3.4 Integration test: Risk cascade → Kill switch +- **Scope:** Test that risk limit violations properly cascade through circuit breakers to kill switch activation +- **Purpose:** Verify safety mechanisms actually trigger under stress conditions + +--- + +## Layer 4 — Training Infrastructure + +### 4.1 GPU training strategy +- **Local (RTX 3050 Ti, 4GB VRAM):** + - Dev/debug training, small batch sizes (max 230 for PPO) + - Rapid iteration on model architecture changes + - Feature extraction and data preprocessing +- **Cloud (Scaleway GPU instances):** + - Production training runs with full datasets + - Hyperparameter optimization (parallel trials) + - Large batch training for all 5 model types +- **Deliverables:** + - Training launcher script that detects GPU and selects local/cloud path + - Scaleway instance provisioning configuration + - Model artifact sync between local and cloud storage + +### 4.2 Data pipeline — Databento OHLCV + MBP-10 +- **Data source:** Databento API for historical market data +- **Schemas needed:** + - **OHLCV (ohlcv-1m, ohlcv-1h):** Primary training data for all models. ~50MB/symbol/year at 1-min bars + - **MBP-10 (market-by-price, 10 levels):** Microstructure features, order book depth. ~2-5GB/symbol/year compressed +- **Data quantity research:** + - Minimum: 2 years OHLCV for regime diversity (bull, bear, sideways, high-vol) + - Recommended: 5 years OHLCV + 1 year MBP-10 for microstructure features + - Symbols: Start with 5-10 liquid instruments (ES, NQ, CL, GC, EUR/USD equivalent) +- **Storage:** Databento DBN format, already supported by `dbn_sequence_loader.rs` and `dbn_data_source.rs` +- **Pipeline:** + - Databento API client for historical data download + - DBN file management (metadata caching already implemented) + - Train/validation/test temporal splits (temporal_guard.rs enforces no leakage) + +### 4.3 Training pipeline safety +- **Checkpointing:** Already implemented (safetensors), verify all 5 model types checkpoint correctly +- **NaN detection:** Add gradient NaN checks during training, auto-halt with last good checkpoint +- **LR scheduling:** Implement cosine annealing with warmup for production training runs +- **Metric logging:** Training metrics to structured logs for monitoring dashboards +- **Reproducibility:** Seed management for all random operations (data shuffling, model init, exploration) + +--- + +## Scope Exclusions + +- **Broker integration:** Separate workstream, not blocked by this work +- **FIX protocol production config:** Phase 2 of broker_gateway_service +- **Web dashboard enhancements:** Already functional, not in scope +- **TLI replacement:** Already completed (deleted in Phase 1) + +## Success Criteria + +1. Zero `std::process::exit()` calls in the codebase +2. All placeholder data replaced with real calculations (VaR, positions, features) +3. GPU OOM handled gracefully with CPU fallback +4. Ensemble survives individual model failures (circuit breakers) +5. 5 model types (DQN, PPO, TFT, Mamba2, CfC) all in ensemble +6. >90% code coverage on critical trading paths +7. End-to-end integration tests pass (ML→Order→Fill, Risk→KillSwitch) +8. Training runs complete successfully on both local GPU and Scaleway cloud +9. Databento data pipeline downloads, processes, and feeds into training +10. Model checksums verified on every load