docs: production hardening phase 2 design — $100K live trading readiness
4-layer risk-prioritized approach: safety net (crash prevention), correctness (accurate calculations + liquid CfC ensemble integration), verification (test coverage), training infrastructure (Databento + GPU). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
199
docs/plans/2026-02-23-production-hardening-phase2-design.md
Normal file
199
docs/plans/2026-02-23-production-hardening-phase2-design.md
Normal file
@@ -0,0 +1,199 @@
|
||||
# Production Hardening Phase 2 — $100K Live Trading Readiness
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Eliminate all remaining safety gaps, wire real data into placeholder paths, verify correctness with integration tests, and establish training infrastructure — making the system ready for $100K live trading.
|
||||
|
||||
**Architecture:** Risk-prioritized 4-layer approach: Safety Net (crash prevention) → Correctness (accurate calculations) → Verification (test coverage) → Training Infrastructure (GPU strategy + data pipeline). The liquid-cfc-v2 ensemble extension is integrated as part of Layer 2 correctness work.
|
||||
|
||||
**Tech Stack:** Rust (37+ crate workspace), Candle v0.9.1 (ML), Databento (market data), Scaleway (cloud GPU), safetensors (model checkpoints)
|
||||
|
||||
---
|
||||
|
||||
## Audit Context
|
||||
|
||||
### What Was Done (Phase 1)
|
||||
- 53-task production hardening merged to main (+2,952/-1,023 lines across 52 files)
|
||||
- 80+ TODO/FIXME items resolved, all clippy deny rules enforced
|
||||
- 4,234+ tests passing across 30 crate targets, 0 warnings
|
||||
|
||||
### What the Audit Found
|
||||
Four parallel agents audited the codebase and identified:
|
||||
- **92 remaining TODOs** — 4 CRITICAL, 3 HIGH, 8 MEDIUM
|
||||
- **7 `std::process::exit()` calls** in temporal_guard.rs (panic-equivalent)
|
||||
- **28 critical files with 0 tests**, 49 vacuous `assert!(true)` tests, 107 ignored tests
|
||||
- **Model loading has no integrity checks**, ensemble has no per-model circuit breakers
|
||||
- **GPU OOM = hard crash** (no runtime VRAM monitoring)
|
||||
|
||||
---
|
||||
|
||||
## Layer 1 — Safety Net (Crash Prevention)
|
||||
|
||||
**Rationale:** These issues can crash the system or cause unrecoverable failures during live trading. Fix first.
|
||||
|
||||
### 1.1 Replace `std::process::exit()` in temporal_guard.rs
|
||||
- **File:** `ml/src/validation/temporal_guard.rs`
|
||||
- **Problem:** 7 calls to `std::process::exit(1)` — kills the entire process without cleanup
|
||||
- **Fix:** Replace with `Result`-based error propagation using a new `TemporalGuardError` enum
|
||||
- **Risk:** HIGH — process exit during trading = lost positions, no graceful shutdown
|
||||
|
||||
### 1.2 Wire real position data into trading agent
|
||||
- **File:** `services/trading_agent_service/src/service.rs:804-809`
|
||||
- **Problem:** Position data hardcoded to `current_weight: 0.0, current_quantity: 0.0`
|
||||
- **Fix:** Query position manager for real portfolio weights and quantities
|
||||
- **Risk:** CRITICAL — ensemble allocations are meaningless without real position data
|
||||
|
||||
### 1.3 Wire real VaR calculation
|
||||
- **Files:** `services/trading_service/src/services/risk.rs:315,415`
|
||||
- **Problem:** VaR uses placeholder formula `confidence_level * 1_000_000.0`
|
||||
- **Fix:** Wire to the real VaR calculator in `risk/src/var/` which already has parametric, historical, and Monte Carlo methods
|
||||
- **Risk:** CRITICAL — risk limits are not enforced if VaR is fake
|
||||
|
||||
### 1.4 Kill switch Redis monitoring
|
||||
- **File:** `risk/src/safety/kill_switch.rs:378`
|
||||
- **Problem:** Redis monitoring task comment says it should be spawned but never is
|
||||
- **Fix:** Spawn the monitoring task in `start()`, or document that Redis monitoring is deferred and the local kill switch is sufficient for Phase 1
|
||||
- **Risk:** HIGH — distributed kill switch won't propagate across services
|
||||
|
||||
### 1.5 GPU OOM detection and CPU fallback
|
||||
- **Files:** `ml/src/inference/inference.rs`, `ml/src/inference/inference_engine.rs`
|
||||
- **Problem:** GPU out-of-memory = hard crash, no runtime VRAM monitoring
|
||||
- **Fix:** Add VRAM usage check before GPU inference, fall back to CPU with warning log if VRAM > 80% threshold
|
||||
- **Risk:** HIGH — GPU OOM during live trading = system crash
|
||||
|
||||
### 1.6 Per-model circuit breakers in ensemble
|
||||
- **File:** `ml/src/inference/inference_ensemble.rs`
|
||||
- **Problem:** If one model returns garbage (NaN, extreme values), it contaminates the ensemble vote
|
||||
- **Fix:** Add per-model circuit breaker that trips on NaN, repeated identical outputs, or extreme value divergence; ensemble continues with remaining healthy models
|
||||
- **Risk:** HIGH — one bad model can cause the entire ensemble to generate bad trades
|
||||
|
||||
---
|
||||
|
||||
## Layer 2 — Correctness (Accurate Calculations)
|
||||
|
||||
### 2.1 Portfolio correlation matrix for Markowitz allocation
|
||||
- **File:** `services/trading_agent_service/src/allocation.rs`
|
||||
- **Problem:** Uses diagonal covariance (ignores correlations between assets)
|
||||
- **Fix:** Implement rolling correlation matrix from historical returns, use in Markowitz optimization
|
||||
- **Risk:** MEDIUM — suboptimal allocation but not dangerous (diagonal is conservative)
|
||||
|
||||
### 2.2 Feature extraction NaN/Inf guards
|
||||
- **File:** `services/trading_service/src/services/enhanced_ml.rs`
|
||||
- **Problem:** Feature extraction can produce NaN/Inf from division by zero (e.g., zero volume, zero range), which propagates through model inference
|
||||
- **Fix:** Add NaN/Inf check after feature extraction, before normalization. Replace with 0.0 and log warning.
|
||||
- **Risk:** HIGH — NaN in model input = NaN in output = unpredictable trades
|
||||
|
||||
### 2.3 Model file integrity validation
|
||||
- **Problem:** Model files loaded from disk with no integrity verification — corrupted file = silent bad predictions
|
||||
- **Fix:** SHA-256 checksum stored alongside safetensors files, verified on load. Schema validation ensures tensor shapes match expected architecture.
|
||||
- **Risk:** MEDIUM — unlikely but catastrophic if it happens
|
||||
|
||||
### 2.4 Model versioning and rollback
|
||||
- **Problem:** No way to roll back to a previous model version if a new one performs poorly
|
||||
- **Fix:** Model registry with version tracking, A/B comparison metrics, and one-command rollback
|
||||
- **Risk:** MEDIUM — operational risk during model updates
|
||||
|
||||
### 2.5 Liquid CfC v2 ensemble integration
|
||||
- **Branch:** `worktree-liquid-cfc-v2` (10 commits, +15,014/-2,666 lines)
|
||||
- **Status:** LiquidInferenceAdapter and LiquidTrainableAdapter already implemented
|
||||
- **Work needed:**
|
||||
- Merge liquid-cfc-v2 branch to main
|
||||
- Register CfC adapter in EnsembleCoordinator (alongside DQN/PPO/TFT/Mamba2)
|
||||
- Add CfC to hyperopt adapter registry
|
||||
- Verify CfC inference latency is within ensemble budget (<10ms)
|
||||
- Add CfC-specific circuit breaker configuration
|
||||
- **Risk:** LOW — additive change, ensemble already handles N models
|
||||
|
||||
### 2.6 Hyperopt Phase B unblocking
|
||||
- **File:** `ml/src/hyperopt/optimizer.rs`
|
||||
- **Problem:** Hyperopt works for individual models but multi-model orchestration (Phase B) is blocked
|
||||
- **Fix:** Wire ensemble-level hyperopt that optimizes model weights and per-model hyperparameters jointly
|
||||
- **Risk:** MEDIUM — training without hyperopt = suboptimal model configurations
|
||||
|
||||
---
|
||||
|
||||
## Layer 3 — Verification (Test Coverage)
|
||||
|
||||
### 3.1 Execution path integration tests (~53 tests)
|
||||
- **Target files (0-test critical files):**
|
||||
- `services/trading_service/src/services/risk.rs` (VaR, risk limits)
|
||||
- `services/trading_service/src/services/enhanced_ml.rs` (feature extraction, inference)
|
||||
- `services/trading_agent_service/src/service.rs` (allocation, order generation)
|
||||
- `services/trading_agent_service/src/allocation.rs` (Markowitz optimization)
|
||||
- `services/trading_service/src/core/execution_engine.rs` (order execution)
|
||||
- **Approach:** Property-based tests for numerical code, scenario tests for trading logic
|
||||
|
||||
### 3.2 Replace vacuous `assert!(true)` tests
|
||||
- **Count:** 49 tests that just `assert!(true)` or test trivial construction
|
||||
- **Fix:** Replace each with meaningful assertions testing actual behavior
|
||||
- **Priority:** Focus on tests in critical paths first (risk, ML, execution)
|
||||
|
||||
### 3.3 Integration test: ML → Order → Fill
|
||||
- **Scope:** End-to-end test from feature extraction through model inference, ensemble voting, order generation, and simulated fill
|
||||
- **Purpose:** Verify the complete trading pipeline produces valid orders from market data
|
||||
|
||||
### 3.4 Integration test: Risk cascade → Kill switch
|
||||
- **Scope:** Test that risk limit violations properly cascade through circuit breakers to kill switch activation
|
||||
- **Purpose:** Verify safety mechanisms actually trigger under stress conditions
|
||||
|
||||
---
|
||||
|
||||
## Layer 4 — Training Infrastructure
|
||||
|
||||
### 4.1 GPU training strategy
|
||||
- **Local (RTX 3050 Ti, 4GB VRAM):**
|
||||
- Dev/debug training, small batch sizes (max 230 for PPO)
|
||||
- Rapid iteration on model architecture changes
|
||||
- Feature extraction and data preprocessing
|
||||
- **Cloud (Scaleway GPU instances):**
|
||||
- Production training runs with full datasets
|
||||
- Hyperparameter optimization (parallel trials)
|
||||
- Large batch training for all 5 model types
|
||||
- **Deliverables:**
|
||||
- Training launcher script that detects GPU and selects local/cloud path
|
||||
- Scaleway instance provisioning configuration
|
||||
- Model artifact sync between local and cloud storage
|
||||
|
||||
### 4.2 Data pipeline — Databento OHLCV + MBP-10
|
||||
- **Data source:** Databento API for historical market data
|
||||
- **Schemas needed:**
|
||||
- **OHLCV (ohlcv-1m, ohlcv-1h):** Primary training data for all models. ~50MB/symbol/year at 1-min bars
|
||||
- **MBP-10 (market-by-price, 10 levels):** Microstructure features, order book depth. ~2-5GB/symbol/year compressed
|
||||
- **Data quantity research:**
|
||||
- Minimum: 2 years OHLCV for regime diversity (bull, bear, sideways, high-vol)
|
||||
- Recommended: 5 years OHLCV + 1 year MBP-10 for microstructure features
|
||||
- Symbols: Start with 5-10 liquid instruments (ES, NQ, CL, GC, EUR/USD equivalent)
|
||||
- **Storage:** Databento DBN format, already supported by `dbn_sequence_loader.rs` and `dbn_data_source.rs`
|
||||
- **Pipeline:**
|
||||
- Databento API client for historical data download
|
||||
- DBN file management (metadata caching already implemented)
|
||||
- Train/validation/test temporal splits (temporal_guard.rs enforces no leakage)
|
||||
|
||||
### 4.3 Training pipeline safety
|
||||
- **Checkpointing:** Already implemented (safetensors), verify all 5 model types checkpoint correctly
|
||||
- **NaN detection:** Add gradient NaN checks during training, auto-halt with last good checkpoint
|
||||
- **LR scheduling:** Implement cosine annealing with warmup for production training runs
|
||||
- **Metric logging:** Training metrics to structured logs for monitoring dashboards
|
||||
- **Reproducibility:** Seed management for all random operations (data shuffling, model init, exploration)
|
||||
|
||||
---
|
||||
|
||||
## Scope Exclusions
|
||||
|
||||
- **Broker integration:** Separate workstream, not blocked by this work
|
||||
- **FIX protocol production config:** Phase 2 of broker_gateway_service
|
||||
- **Web dashboard enhancements:** Already functional, not in scope
|
||||
- **TLI replacement:** Already completed (deleted in Phase 1)
|
||||
|
||||
## Success Criteria
|
||||
|
||||
1. Zero `std::process::exit()` calls in the codebase
|
||||
2. All placeholder data replaced with real calculations (VaR, positions, features)
|
||||
3. GPU OOM handled gracefully with CPU fallback
|
||||
4. Ensemble survives individual model failures (circuit breakers)
|
||||
5. 5 model types (DQN, PPO, TFT, Mamba2, CfC) all in ensemble
|
||||
6. >90% code coverage on critical trading paths
|
||||
7. End-to-end integration tests pass (ML→Order→Fill, Risk→KillSwitch)
|
||||
8. Training runs complete successfully on both local GPU and Scaleway cloud
|
||||
9. Databento data pipeline downloads, processes, and feeds into training
|
||||
10. Model checksums verified on every load
|
||||
Reference in New Issue
Block a user