docs: production readiness phase 3 design — 18 tasks across 4 pillars

Safety, live trading, security, and observability gaps identified
from comprehensive codebase audit. Organized for parallel swarm execution.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

docs/plans/2026-02-21-production-readiness-phase3-design.md

@@ -0,0 +1,135 @@
+# Production Readiness Phase 3 — Design
+
+**Goal**: Eliminate all CRITICAL and HIGH-severity production gaps across safety, live trading, security, and observability — organized as 4 parallel work streams for swarm execution.
+
+**Architecture**: 18 independent tasks grouped into 4 pillars. Each pillar targets a different concern (safety, trading, security, observability) and operates on non-overlapping files/crates, enabling full parallel execution.
+
+**Tech Stack**: Rust, Candle v0.9.1, object_store, ring/rustls (TLS), sysinfo, InfluxDB client
+
+---
+
+## Pillar 1: Safety & Crash Elimination (6 tasks)
+
+Removes all production crash vectors — panic!, unwrap(), and silent failures.
+
+### Task 1: Remove panic! from strategy_runner data processing
+- **File**: `backtesting/src/strategy_runner.rs:1374,1379`
+- **Fix**: Replace `panic!()` and `unwrap_or_else(|e| panic!(...))` with `Result` propagation or graceful skip-and-log
+- **Verify**: `SQLX_OFFLINE=true cargo check -p backtesting`
+
+### Task 2: Fix IQN unwrap() violations in DQN
+- **File**: `ml/src/dqn/dqn.rs:1165,1581,1582`
+- **Fix**: Replace `.unwrap()` with `.ok_or(MLError::...)` propagation. IQN networks are `Option<>` — should return error if called without IQN enabled.
+- **Verify**: `SQLX_OFFLINE=true cargo check -p ml`
+
+### Task 3: Implement kill switch in compliance service
+- **File**: `services/trading_service/src/compliance_service.rs:323`
+- **Fix**: Wire actual kill switch that halts all order submission and cancels open orders
+- **Verify**: `SQLX_OFFLINE=true cargo check -p trading_service`
+
+### Task 4: Remove ensemble mock prediction fallback
+- **File**: `ml/src/ensemble/coordinator.rs:110,130,187-195`
+- **Fix**: When no models are loaded, return `Err(MLError::NoModelsLoaded)` instead of generating mock predictions. Add a `has_models()` guard method.
+- **Verify**: `SQLX_OFFLINE=true cargo test -p ml --lib ensemble`
+
+### Task 5: Fix zeroed features in PPO position sizer
+- **File**: `adaptive-strategy/src/risk/ppo_position_sizer.rs:1144-1149`
+- **Fix**: Replace `vec![0.0; state_dim / 3]` with real feature extraction from market state
+- **Verify**: `SQLX_OFFLINE=true cargo check -p adaptive-strategy`
+
+### Task 6: Add clippy deny attributes to unprotected crates
+- **Files**: `ml/src/lib.rs`, `services/api_gateway/src/lib.rs`, `services/backtesting_service/src/lib.rs`, `common/src/lib.rs`
+- **Fix**: Add `#![deny(clippy::unwrap_used, clippy::expect_used)]` and fix any violations
+- **Verify**: `SQLX_OFFLINE=true cargo clippy --workspace`
+
+## Pillar 2: Live Trading Path (5 tasks)
+
+Wires real market data and model training into the execution pipeline.
+
+### Task 7: Wire real market price into execution engine
+- **File**: `services/trading_service/src/core/execution_engine.rs:283`
+- **Fix**: Replace TODO placeholder with actual market data feed price lookup from `MarketDataManager`
+- **Verify**: `SQLX_OFFLINE=true cargo check -p trading_service`
+
+### Task 8: Wire participation rate in execution engine
+- **File**: `services/trading_service/src/core/execution_engine.rs:451`
+- **Fix**: Use `participation_rate` parameter for VWAP/TWAP execution when market data is available
+- **Verify**: `SQLX_OFFLINE=true cargo check -p trading_service`
+
+### Task 9: Implement data acquisition validator
+- **File**: `services/data_acquisition_service/src/validator.rs:73,84`
+- **Fix**: Implement `validate()` and `quick_validate()` — check file format, date ranges, symbol coverage, data gaps
+- **Verify**: `SQLX_OFFLINE=true cargo test -p data_acquisition_service --lib validator`
+
+### Task 10: Wire TFT safetensors checkpoint save/load
+- **File**: `ml/src/tft/trainable_adapter.rs:449,483`
+- **Fix**: Implement save/load using safetensors format (same pattern as DQN/PPO checkpointing)
+- **Verify**: `SQLX_OFFLINE=true cargo test -p ml --lib tft::trainable_adapter`
+
+### Task 11: Wire retrain_all_models PPO/TFT/Mamba2 training
+- **File**: `ml/examples/retrain_all_models.rs:856,868,880`
+- **Fix**: Replace TODOs with actual trainer invocations using existing `PpoTrainer`, `TFTTrainer`, `Mamba2Trainable`
+- **Verify**: `SQLX_OFFLINE=true cargo check --example retrain_all_models`
+
+## Pillar 3: Security Hardening (3 tasks)
+
+Completes TLS/mTLS authentication and certificate validation.
+
+### Task 12: Implement mTLS signature verification
+- **File**: `services/api_gateway/src/auth/mtls/validator.rs:391`
+- **Fix**: Implement X.509 signature verification using `ring` or `rustls` — verify certificate chain, check CN/SAN
+- **Verify**: `SQLX_OFFLINE=true cargo check -p api_gateway`
+
+### Task 13: Implement OCSP checking
+- **File**: `services/trading_service/src/tls_config.rs:582`
+- **Fix**: Add OCSP stapling/checking for certificate revocation status
+- **Verify**: `SQLX_OFFLINE=true cargo check -p trading_service`
+
+### Task 14: Implement CRL signature validation
+- **File**: `services/trading_service/src/auth/mtls/revocation.rs:282`
+- **Fix**: Validate CRL issuer signature and check validity period before trusting revocation list
+- **Verify**: `SQLX_OFFLINE=true cargo check -p trading_service`
+
+## Pillar 4: Observability & Testing (4 tasks)
+
+Wires monitoring, storage, and adds missing test coverage.
+
+### Task 15: Implement sysinfo process monitoring
+- **Files**: `ml/src/deployment/monitoring.rs:1131,1139`, `ml/src/deployment/validation.rs:1318`
+- **Fix**: Use `sysinfo` crate to report real process memory and CPU usage
+- **Verify**: `SQLX_OFFLINE=true cargo check -p ml`
+
+### Task 16: Wire InfluxDB backtest storage
+- **File**: `services/backtesting_service/src/storage.rs:79,447`
+- **Fix**: Initialize InfluxDB client and implement write/query for backtest performance metrics
+- **Verify**: `SQLX_OFFLINE=true cargo check -p backtesting_service`
+
+### Task 17: Fix common observability module
+- **Files**: `common/src/lib.rs:34,106`
+- **Fix**: Resolve type errors in observability module and re-enable the module
+- **Verify**: `SQLX_OFFLINE=true cargo check -p common`
+
+### Task 18: Add data_acquisition_service integration tests
+- **File**: `services/data_acquisition_service/tests/` (new test files)
+- **Fix**: Add integration tests for downloader→validator→uploader pipeline (currently only 3 unit tests)
+- **Verify**: `SQLX_OFFLINE=true cargo test -p data_acquisition_service`
+
+---
+
+## Swarm Execution Strategy
+
+All 4 pillars run in parallel. Within each pillar, tasks are independent (different files/crates):
+
+```
+Pillar 1 (Safety):     Tasks 1-6   [backtesting, ml, trading_service, adaptive-strategy, common]
+Pillar 2 (Trading):    Tasks 7-11  [trading_service, data_acquisition, ml]
+Pillar 3 (Security):   Tasks 12-14 [api_gateway, trading_service]
+Pillar 4 (Observability): Tasks 15-18 [ml, backtesting_service, common, data_acquisition]
+```
+
+**Conflict zones** (same file touched by multiple tasks):
+- `trading_service` execution_engine.rs: Tasks 7 & 8 (sequential within pillar)
+- `common/src/lib.rs`: Tasks 6 & 17 (sequential across pillars)
+- `ml/src/deployment/`: Tasks 15 only (no conflict)
+
+Maximum parallelism: **14 agents** (18 tasks minus 4 sequential dependencies).