From aa6669ede022e5782962005ea17681793e71d856 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Sat, 21 Feb 2026 18:53:38 +0100 Subject: [PATCH] =?UTF-8?q?docs:=20production=20readiness=20phase=203=20de?= =?UTF-8?q?sign=20=E2=80=94=2018=20tasks=20across=204=20pillars?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Safety, live trading, security, and observability gaps identified from comprehensive codebase audit. Organized for parallel swarm execution. Co-Authored-By: Claude Opus 4.6 --- ...2-21-production-readiness-phase3-design.md | 135 ++++++++++++++++++ 1 file changed, 135 insertions(+) create mode 100644 docs/plans/2026-02-21-production-readiness-phase3-design.md diff --git a/docs/plans/2026-02-21-production-readiness-phase3-design.md b/docs/plans/2026-02-21-production-readiness-phase3-design.md new file mode 100644 index 000000000..8ccb7661e --- /dev/null +++ b/docs/plans/2026-02-21-production-readiness-phase3-design.md @@ -0,0 +1,135 @@ +# Production Readiness Phase 3 — Design + +**Goal**: Eliminate all CRITICAL and HIGH-severity production gaps across safety, live trading, security, and observability — organized as 4 parallel work streams for swarm execution. + +**Architecture**: 18 independent tasks grouped into 4 pillars. Each pillar targets a different concern (safety, trading, security, observability) and operates on non-overlapping files/crates, enabling full parallel execution. + +**Tech Stack**: Rust, Candle v0.9.1, object_store, ring/rustls (TLS), sysinfo, InfluxDB client + +--- + +## Pillar 1: Safety & Crash Elimination (6 tasks) + +Removes all production crash vectors — panic!, unwrap(), and silent failures. + +### Task 1: Remove panic! from strategy_runner data processing +- **File**: `backtesting/src/strategy_runner.rs:1374,1379` +- **Fix**: Replace `panic!()` and `unwrap_or_else(|e| panic!(...))` with `Result` propagation or graceful skip-and-log +- **Verify**: `SQLX_OFFLINE=true cargo check -p backtesting` + +### Task 2: Fix IQN unwrap() violations in DQN +- **File**: `ml/src/dqn/dqn.rs:1165,1581,1582` +- **Fix**: Replace `.unwrap()` with `.ok_or(MLError::...)` propagation. IQN networks are `Option<>` — should return error if called without IQN enabled. +- **Verify**: `SQLX_OFFLINE=true cargo check -p ml` + +### Task 3: Implement kill switch in compliance service +- **File**: `services/trading_service/src/compliance_service.rs:323` +- **Fix**: Wire actual kill switch that halts all order submission and cancels open orders +- **Verify**: `SQLX_OFFLINE=true cargo check -p trading_service` + +### Task 4: Remove ensemble mock prediction fallback +- **File**: `ml/src/ensemble/coordinator.rs:110,130,187-195` +- **Fix**: When no models are loaded, return `Err(MLError::NoModelsLoaded)` instead of generating mock predictions. Add a `has_models()` guard method. +- **Verify**: `SQLX_OFFLINE=true cargo test -p ml --lib ensemble` + +### Task 5: Fix zeroed features in PPO position sizer +- **File**: `adaptive-strategy/src/risk/ppo_position_sizer.rs:1144-1149` +- **Fix**: Replace `vec![0.0; state_dim / 3]` with real feature extraction from market state +- **Verify**: `SQLX_OFFLINE=true cargo check -p adaptive-strategy` + +### Task 6: Add clippy deny attributes to unprotected crates +- **Files**: `ml/src/lib.rs`, `services/api_gateway/src/lib.rs`, `services/backtesting_service/src/lib.rs`, `common/src/lib.rs` +- **Fix**: Add `#![deny(clippy::unwrap_used, clippy::expect_used)]` and fix any violations +- **Verify**: `SQLX_OFFLINE=true cargo clippy --workspace` + +## Pillar 2: Live Trading Path (5 tasks) + +Wires real market data and model training into the execution pipeline. + +### Task 7: Wire real market price into execution engine +- **File**: `services/trading_service/src/core/execution_engine.rs:283` +- **Fix**: Replace TODO placeholder with actual market data feed price lookup from `MarketDataManager` +- **Verify**: `SQLX_OFFLINE=true cargo check -p trading_service` + +### Task 8: Wire participation rate in execution engine +- **File**: `services/trading_service/src/core/execution_engine.rs:451` +- **Fix**: Use `participation_rate` parameter for VWAP/TWAP execution when market data is available +- **Verify**: `SQLX_OFFLINE=true cargo check -p trading_service` + +### Task 9: Implement data acquisition validator +- **File**: `services/data_acquisition_service/src/validator.rs:73,84` +- **Fix**: Implement `validate()` and `quick_validate()` — check file format, date ranges, symbol coverage, data gaps +- **Verify**: `SQLX_OFFLINE=true cargo test -p data_acquisition_service --lib validator` + +### Task 10: Wire TFT safetensors checkpoint save/load +- **File**: `ml/src/tft/trainable_adapter.rs:449,483` +- **Fix**: Implement save/load using safetensors format (same pattern as DQN/PPO checkpointing) +- **Verify**: `SQLX_OFFLINE=true cargo test -p ml --lib tft::trainable_adapter` + +### Task 11: Wire retrain_all_models PPO/TFT/Mamba2 training +- **File**: `ml/examples/retrain_all_models.rs:856,868,880` +- **Fix**: Replace TODOs with actual trainer invocations using existing `PpoTrainer`, `TFTTrainer`, `Mamba2Trainable` +- **Verify**: `SQLX_OFFLINE=true cargo check --example retrain_all_models` + +## Pillar 3: Security Hardening (3 tasks) + +Completes TLS/mTLS authentication and certificate validation. + +### Task 12: Implement mTLS signature verification +- **File**: `services/api_gateway/src/auth/mtls/validator.rs:391` +- **Fix**: Implement X.509 signature verification using `ring` or `rustls` — verify certificate chain, check CN/SAN +- **Verify**: `SQLX_OFFLINE=true cargo check -p api_gateway` + +### Task 13: Implement OCSP checking +- **File**: `services/trading_service/src/tls_config.rs:582` +- **Fix**: Add OCSP stapling/checking for certificate revocation status +- **Verify**: `SQLX_OFFLINE=true cargo check -p trading_service` + +### Task 14: Implement CRL signature validation +- **File**: `services/trading_service/src/auth/mtls/revocation.rs:282` +- **Fix**: Validate CRL issuer signature and check validity period before trusting revocation list +- **Verify**: `SQLX_OFFLINE=true cargo check -p trading_service` + +## Pillar 4: Observability & Testing (4 tasks) + +Wires monitoring, storage, and adds missing test coverage. + +### Task 15: Implement sysinfo process monitoring +- **Files**: `ml/src/deployment/monitoring.rs:1131,1139`, `ml/src/deployment/validation.rs:1318` +- **Fix**: Use `sysinfo` crate to report real process memory and CPU usage +- **Verify**: `SQLX_OFFLINE=true cargo check -p ml` + +### Task 16: Wire InfluxDB backtest storage +- **File**: `services/backtesting_service/src/storage.rs:79,447` +- **Fix**: Initialize InfluxDB client and implement write/query for backtest performance metrics +- **Verify**: `SQLX_OFFLINE=true cargo check -p backtesting_service` + +### Task 17: Fix common observability module +- **Files**: `common/src/lib.rs:34,106` +- **Fix**: Resolve type errors in observability module and re-enable the module +- **Verify**: `SQLX_OFFLINE=true cargo check -p common` + +### Task 18: Add data_acquisition_service integration tests +- **File**: `services/data_acquisition_service/tests/` (new test files) +- **Fix**: Add integration tests for downloader→validator→uploader pipeline (currently only 3 unit tests) +- **Verify**: `SQLX_OFFLINE=true cargo test -p data_acquisition_service` + +--- + +## Swarm Execution Strategy + +All 4 pillars run in parallel. Within each pillar, tasks are independent (different files/crates): + +``` +Pillar 1 (Safety): Tasks 1-6 [backtesting, ml, trading_service, adaptive-strategy, common] +Pillar 2 (Trading): Tasks 7-11 [trading_service, data_acquisition, ml] +Pillar 3 (Security): Tasks 12-14 [api_gateway, trading_service] +Pillar 4 (Observability): Tasks 15-18 [ml, backtesting_service, common, data_acquisition] +``` + +**Conflict zones** (same file touched by multiple tasks): +- `trading_service` execution_engine.rs: Tasks 7 & 8 (sequential within pillar) +- `common/src/lib.rs`: Tasks 6 & 17 (sequential across pillars) +- `ml/src/deployment/`: Tasks 15 only (no conflict) + +Maximum parallelism: **14 agents** (18 tasks minus 4 sequential dependencies).