From e2a712593a447538d82b26b44c5e4c4bfac18694 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Sat, 21 Feb 2026 15:18:19 +0100 Subject: [PATCH] docs: full system production readiness design (6-phase plan) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Covers data pipeline, trading service core, broker connectivity (FIX 4.4 + TWS), execution algorithms, security/compliance, and observability — based on comprehensive non-ML audit. Co-Authored-By: Claude Opus 4.6 --- ...full-system-production-readiness-design.md | 349 ++++++++++++++++++ 1 file changed, 349 insertions(+) create mode 100644 docs/plans/2026-02-21-full-system-production-readiness-design.md diff --git a/docs/plans/2026-02-21-full-system-production-readiness-design.md b/docs/plans/2026-02-21-full-system-production-readiness-design.md new file mode 100644 index 000000000..83fb4d47d --- /dev/null +++ b/docs/plans/2026-02-21-full-system-production-readiness-design.md @@ -0,0 +1,349 @@ +# Full System Production Readiness Design + +**Date**: 2026-02-21 +**Status**: Approved +**Scope**: Production-grade hardening of all non-ML crates — data pipeline, trading service, broker connectivity, execution algorithms, security/compliance, observability. + +## Problem Statement + +The ML crate has been validated and hardened (ensemble real inference, validation harness, hyperopt). However, the rest of the system has significant production gaps: +- 9 critical issues (broker stubs, data stalls, hardcoded features, empty risk engine) +- 10 high-priority issues (execution stubs, compliance gaps, auth bypasses) +- ~18 medium/low issues (tech debt, orphaned files, placeholder storage) + +## Architecture: 6-Phase Dependency Chain + +``` +Phase 1: Data Pipeline ──→ real data flows +Phase 2: Trading Service Core ──→ real features, risk, order matching +Phase 3: Broker Connectivity ──→ real orders execute (FIX 4.4 + TWS) +Phase 4: Execution Algorithms ──→ TWAP, VWAP, Iceberg, Sniper +Phase 5: Security & Compliance──→ mTLS, OCSP, MiFID II, MAR +Phase 6: Observability ──→ OpenTelemetry, metrics, kill switch +``` + +Each phase is independently testable and deployable. + +--- + +## Phase 1: Data Pipeline + +**Goal**: Real market data flows from Databento through the system. + +### 1.1 Databento Stream (CRITICAL) + +**File**: `data/src/providers/databento/stream.rs:839` +**Current**: `poll_next()` always returns `Poll::Pending` — stream never delivers data. + +**Fix**: Implement real async stream using `databento` crate's `LiveClient`: +- `DatabentoMarketDataStream` wraps `databento::LiveClient` +- `poll_next` reads from the client's async receiver +- Proper `Waker` registration for async task notification +- Parse incoming DBN records into `MarketEvent` variants +- Handle reconnection on disconnect with exponential backoff + +### 1.2 Data Acquisition Service (CRITICAL) + +**File**: `services/data_acquisition_service/src/downloader.rs`, `uploader.rs` +**Current**: Both `download()` and `upload()` return hard errors. + +**Fix**: +- **Downloader**: Use `databento::HistoricalClient::timeseries_get_range()` for historical data + - Accept `DownloadJob` with dataset, schema, date range, symbols + - Stream to local temp file, validate checksum, return path +- **Uploader**: Use `storage::ObjectStoreBackend` (already exists in storage/ crate) + - Check existence via HEAD request (dedup) + - Upload with proper content-type and metadata tags + - Return `UploadResult` with object key and size + +### 1.3 DBN Uploader MinIO Integration + +**File**: `data/src/dbn_uploader.rs:211-264` +**Current**: TODO comments for existence check and actual upload. + +**Fix**: Wire the existing `ObjectStoreBackend` for dedup check + upload. The storage crate already has the MinIO client — this is plumbing. + +### 1.4 Placeholder Trade Events + +**File**: `data/src/providers/databento/mod.rs:450-461` +**Current**: Unsupported event types create synthetic `trade_id: Some("placeholder")` events. + +**Fix**: Return `None` for unsupported event types. Log at `debug!` level. Do not create synthetic data that corrupts downstream analytics. + +--- + +## Phase 2: Trading Service Core + +**Goal**: Real features, real risk enforcement, real order matching. + +### 2.1 Feature Extraction (CRITICAL) + +**File**: `services/trading_service/src/ensemble_coordinator.rs:569` +**Current**: `fetch_features_for_symbol()` returns `vec![0.5; 16]` constant. + +**Fix**: +- Query market data cache (Redis or in-memory ring buffer) for latest OHLCV +- Use `data` crate's feature extractors for technical indicators +- Build 51-dim `FeatureVector` matching the canonical layout (price 0-9, technical 10-19, order book 20-29, microstructure 30-39, position/risk 40-50) +- Fall back to last-known features if cache miss (not zeros) + +### 2.2 Risk Engine Wiring (CRITICAL) + +**File**: `services/trading_service/src/state.rs:539-629` +**Current**: `RiskEngine` and `MLEngine` are empty structs. + +**Fix**: +- `RiskEngine` wraps `risk::RiskEngine` (41k lines, production-ready) +- `MLEngine` wraps `EnsembleCoordinator` with loaded model adapters +- Both initialized during service startup with config from `config/` crate +- Connected to the trading pipeline via `AppState` + +### 2.3 VaR Calculation (CRITICAL) + +**File**: `services/trading_service/src/core/risk_manager.rs:899-919` +**Current**: Placeholder percentile calculation. + +**Fix**: Use `risk::VarCalculator` which implements: +- Monte Carlo VaR (10,000 simulations, configurable) +- Historical VaR (250-day window) +- Parametric VaR (variance-covariance) +- All three are already implemented and tested in `risk/` crate (632 tests) + +### 2.4 Order Book Matching (HIGH) + +**File**: `services/trading_service/src/core/order_manager.rs:544-552` +**Current**: `consume_entry` and `reduce_quantity` methods don't exist on `SmallBatchRing`. + +**Fix**: +- Add `consume_entry()` to `SmallBatchRing` — remove fully filled entry +- Add `reduce_quantity()` to `SmallBatchRing` — partial fill, reduce remaining +- Wire into the order matching loop for both full and partial fills + +### 2.5 API Gateway ML Proxy (CRITICAL) + +**File**: `services/api_gateway/src/handlers/ml.rs:217-226` +**Current**: Returns `prediction = 0.5`, `confidence = 0.75` hardcoded. + +**Fix**: gRPC proxy to `ml_training_service` inference endpoint: +- Forward feature vector from request body +- Call `ml_training_service::Predict` gRPC method +- Return real prediction and confidence +- Add circuit breaker: if ML service unavailable, return error (not fake data) + +--- + +## Phase 3: Broker Connectivity + +**Goal**: Both AMP Futures (FIX 4.4) and Interactive Brokers (TWS) execute real orders. + +### 3.1 FIX 4.4 Protocol (AMP Futures) + +**File**: `trading_engine/src/brokers/fix.rs` (47 lines → ~2000 lines) +**Current**: `FixMessage` struct with HashMap fields only. + +**Implementation**: + +``` +FixEngine +├── FixSession (state machine) +│ ├── States: Disconnected → Connected → LoggedIn → Active → LoggingOut +│ ├── Heartbeat/TestRequest (tag 35=0/1) +│ ├── Sequence numbers (tag 34) with persistence +│ └── Gap fill / resend request (tag 35=2/4) +├── FixTransport (tokio TCP + optional TLS) +│ ├── Framed reader/writer on SOH delimiter +│ ├── Async read loop with Waker +│ └── Reconnect with exponential backoff +├── FixCodec +│ ├── Serialize: BeginString(8) + BodyLength(9) + ... + Checksum(10) +│ ├── Parse: Split on SOH, validate checksum, extract tag=value pairs +│ └── Checksum: sum of all bytes mod 256, formatted as 3-digit string +└── FixSequenceStore (file-backed) + ├── Persist outgoing sequence number + └── Recovery after crash: resend from last persisted +``` + +**Application messages**: +- `NewOrderSingle` (35=D): ClOrdID, Symbol, Side, OrderQty, OrdType, Price, TimeInForce +- `ExecutionReport` (35=8): Parse fills, partial fills, rejects, cancels +- `OrderCancelRequest` (35=F): Cancel by ClOrdID +- `OrderCancelReplaceRequest` (35=G): Modify price/quantity + +### 3.2 Interactive Brokers (TWS API) + +**File**: `trading_engine/src/brokers/interactive_brokers.rs` (~150 lines → ~800 lines) +**Current**: Stub returning hardcoded "IB123456". + +**Implementation**: +- Real `ibapi::Client` connection to TWS/IB Gateway (port 7496/7497) +- `submit_order()`: `client.place_order()` with proper `Contract` + `Order` structs +- `cancel_order()`: `client.cancel_order()` +- `get_positions()`: `client.req_positions()` with position callback +- `subscribe_executions()`: `client.req_executions()` → channel of `Execution` events +- Account monitoring: `client.req_account_updates()` for real-time P&L + +### 3.3 ICMarkets Connector + +**File**: `trading_engine/src/brokers/icmarkets.rs` +**Current**: Copy-paste of IB stub. + +**Implementation**: Uses FIX 4.4 (same engine as AMP Futures, different session config): +- ICMarkets FIX endpoint configuration +- Symbol mapping (ICM uses different symbology) +- Session credentials from config/Vault + +### 3.4 Order Router + +**File**: `trading_engine/src/routing.rs` (48 lines → ~200 lines) +**Current**: String matching on `config.default_broker`. + +**Fix**: +- Type-safe `BrokerType` enum (AMP, IB, ICMarkets) +- Latency-aware routing: measure round-trip per broker, route to fastest for asset class +- Failover: if primary broker disconnected, route to secondary +- Smart routing: futures → AMP, equities → IB, forex → ICMarkets + +--- + +## Phase 4: Execution Algorithms + +**Goal**: Production-grade algorithmic execution beyond market/limit orders. + +### 4.1 TWAP (Time-Weighted Average Price) + +- Slice parent order into N child orders over time window +- Random jitter ±10% of slice interval to avoid detection +- Cancel remaining slices on complete fill +- Track execution quality: actual avg price vs TWAP benchmark + +### 4.2 VWAP (Volume-Weighted Average Price) + +- Fetch historical volume profile for symbol+time window +- Allocate child order sizes proportional to expected volume per bucket +- Track participation rate (our volume / total market volume) +- Dynamic adjustment: if behind schedule, increase next slice; if ahead, reduce + +### 4.3 Iceberg + +- Show `visible_qty` on book (e.g., 10% of total) +- On fill notification, replenish from hidden `total_qty` +- Randomize visible quantity ±20% per replenish to avoid detection +- Track total filled vs remaining hidden + +### 4.4 Sniper (Liquidity Detection) + +- Monitor L2 order book for large resting orders at target price levels +- When detected: send aggressive IOC (Immediate-or-Cancel) to hit the resting liquidity +- Requires sub-millisecond reaction time → integrate with SIMD order book processing +- Configurable minimum quantity threshold for "large" detection + +--- + +## Phase 5: Security & Compliance + +### 5.1 mTLS Certificate Verification (CRITICAL) + +**File**: `services/api_gateway/src/auth/mtls/validator.rs:391` +**Current**: `Ok(())` — accepts any certificate. + +**Fix**: Implement X.509 signature chain verification: +- Verify client cert signed by trusted CA (from CA bundle) +- Check validity period (not expired, not yet valid) +- Check key usage extensions (clientAuth) +- Use `rustls` (already a dependency) for chain validation + +### 5.2 OCSP Revocation (HIGH) + +**File**: `services/api_gateway/src/auth/mtls/revocation.rs:303` +**Current**: Logs warning, returns "not revoked". + +**Fix**: +- Extract OCSP responder URL from certificate's Authority Information Access (AIA) extension +- HTTP POST OCSP request with certificate serial number +- Parse OCSP response (good/revoked/unknown) +- Cache response per serial (TTL from nextUpdate field) +- Soft-fail: if responder unreachable, configurable policy (allow with warning vs reject) + +### 5.3 MiFID II Compliance + +**Files**: `trading_engine/src/compliance/` (commented out modules) + +**Implementation**: +- **Transaction reporting** (RTS 25): Record all trade decisions with timestamps, rationale +- **Best execution** (RTS 27/28): Track execution venues, prices achieved, compare to reference +- **Client order handling**: Priority queuing, time-stamping at microsecond precision + +### 5.4 MAR Market Abuse Surveillance + +**Implementation**: +- **Wash trading detection**: Cross-reference buy/sell orders from same account +- **Layering/spoofing**: Detect orders placed and cancelled quickly on same side +- **Unusual volume**: Statistical anomaly detection on trading volume per symbol + +### 5.5 Audit Trails + +**File**: `trading_engine/src/compliance/sox_compliance.rs` +**Current**: TODO for async persistence. + +**Fix**: Async write to PostgreSQL + S3 for immutable audit log. Every trade decision, risk check, and compliance event gets a tamper-evident record. + +--- + +## Phase 6: Observability & Infrastructure + +### 6.1 OpenTelemetry Re-enable (HIGH) + +**File**: `common/src/lib.rs` — observability module commented out +**Current**: All 3 major services have `// TODO: Re-enable when observability module is fixed`. + +**Fix**: Fix `common/src/observability/` type errors with `tracing_subscriber` v0.3. Re-enable OTLP exporter + Jaeger integration. Add spans to critical paths (order lifecycle, risk checks, ML inference). + +### 6.2 Prometheus Panic Fix (HIGH) + +**File**: `trading_engine/src/trading_operations.rs:42-245` +**Current**: 11 `panic!()` calls on metrics registration failure. + +**Fix**: Replace with `try_register()` or `register_or_get()` pattern. If metric already registered (e.g., after hot reload), reuse existing. Never panic on metrics infrastructure failure. + +### 6.3 Kill Switch Monitor (HIGH) + +**File**: `risk/src/safety/kill_switch.rs:363` +**Current**: `start_monitoring()` returns `Ok(())` immediately. + +**Fix**: `tokio::spawn` background loop: +- Check Redis connectivity every 5s +- If Redis unreachable for >30s, activate local kill switch (fail-safe) +- Heartbeat broadcast to other service instances +- Detect network partition scenarios + +### 6.4 InfluxDB Time-Series (MEDIUM) + +**File**: `services/backtesting_service/src/storage.rs:44` +**Current**: `_influxdb_client: Option<()>`. + +**Fix**: Wire `influxdb2` crate for performance metrics: latency percentiles, throughput, fill rates, slippage tracking. + +### 6.5 Cleanup (LOW) + +- Remove `common/src/ml_strategy_backup.rs`, `ml_strategy_fix.rs`, `ml_strategy_rsi_macd.rs` (1,157 lines orphaned) +- Remove `common/src/types.rs.rej` (rejected patch fragment) +- Remove dev-mode Vault token from `docker-compose.yml` +- Consolidate overlapping GitHub Actions workflows + +--- + +## Non-Goals + +- **Rewrite trading engine core**: SIMD, lock-free queues, RDTSC timing are production-ready +- **ML crate changes**: Already hardened (ensemble inference, validation, hyperopt) +- **Risk crate core**: 41k lines with 632 tests — already production-grade +- **New model types**: No new ML architectures (focus on infrastructure) +- **Multi-asset MIMO**: Deferred per prior design decision + +## Testing Strategy + +Each phase includes: +1. Unit tests for new components +2. Integration tests for cross-crate wiring +3. `cargo check` + `cargo test` verification before each commit +4. Existing test suites must not regress