docs: full system production readiness design (6-phase plan)
Covers data pipeline, trading service core, broker connectivity (FIX 4.4 + TWS), execution algorithms, security/compliance, and observability — based on comprehensive non-ML audit. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
349
docs/plans/2026-02-21-full-system-production-readiness-design.md
Normal file
349
docs/plans/2026-02-21-full-system-production-readiness-design.md
Normal file
@@ -0,0 +1,349 @@
|
||||
# Full System Production Readiness Design
|
||||
|
||||
**Date**: 2026-02-21
|
||||
**Status**: Approved
|
||||
**Scope**: Production-grade hardening of all non-ML crates — data pipeline, trading service, broker connectivity, execution algorithms, security/compliance, observability.
|
||||
|
||||
## Problem Statement
|
||||
|
||||
The ML crate has been validated and hardened (ensemble real inference, validation harness, hyperopt). However, the rest of the system has significant production gaps:
|
||||
- 9 critical issues (broker stubs, data stalls, hardcoded features, empty risk engine)
|
||||
- 10 high-priority issues (execution stubs, compliance gaps, auth bypasses)
|
||||
- ~18 medium/low issues (tech debt, orphaned files, placeholder storage)
|
||||
|
||||
## Architecture: 6-Phase Dependency Chain
|
||||
|
||||
```
|
||||
Phase 1: Data Pipeline ──→ real data flows
|
||||
Phase 2: Trading Service Core ──→ real features, risk, order matching
|
||||
Phase 3: Broker Connectivity ──→ real orders execute (FIX 4.4 + TWS)
|
||||
Phase 4: Execution Algorithms ──→ TWAP, VWAP, Iceberg, Sniper
|
||||
Phase 5: Security & Compliance──→ mTLS, OCSP, MiFID II, MAR
|
||||
Phase 6: Observability ──→ OpenTelemetry, metrics, kill switch
|
||||
```
|
||||
|
||||
Each phase is independently testable and deployable.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: Data Pipeline
|
||||
|
||||
**Goal**: Real market data flows from Databento through the system.
|
||||
|
||||
### 1.1 Databento Stream (CRITICAL)
|
||||
|
||||
**File**: `data/src/providers/databento/stream.rs:839`
|
||||
**Current**: `poll_next()` always returns `Poll::Pending` — stream never delivers data.
|
||||
|
||||
**Fix**: Implement real async stream using `databento` crate's `LiveClient`:
|
||||
- `DatabentoMarketDataStream` wraps `databento::LiveClient`
|
||||
- `poll_next` reads from the client's async receiver
|
||||
- Proper `Waker` registration for async task notification
|
||||
- Parse incoming DBN records into `MarketEvent` variants
|
||||
- Handle reconnection on disconnect with exponential backoff
|
||||
|
||||
### 1.2 Data Acquisition Service (CRITICAL)
|
||||
|
||||
**File**: `services/data_acquisition_service/src/downloader.rs`, `uploader.rs`
|
||||
**Current**: Both `download()` and `upload()` return hard errors.
|
||||
|
||||
**Fix**:
|
||||
- **Downloader**: Use `databento::HistoricalClient::timeseries_get_range()` for historical data
|
||||
- Accept `DownloadJob` with dataset, schema, date range, symbols
|
||||
- Stream to local temp file, validate checksum, return path
|
||||
- **Uploader**: Use `storage::ObjectStoreBackend` (already exists in storage/ crate)
|
||||
- Check existence via HEAD request (dedup)
|
||||
- Upload with proper content-type and metadata tags
|
||||
- Return `UploadResult` with object key and size
|
||||
|
||||
### 1.3 DBN Uploader MinIO Integration
|
||||
|
||||
**File**: `data/src/dbn_uploader.rs:211-264`
|
||||
**Current**: TODO comments for existence check and actual upload.
|
||||
|
||||
**Fix**: Wire the existing `ObjectStoreBackend` for dedup check + upload. The storage crate already has the MinIO client — this is plumbing.
|
||||
|
||||
### 1.4 Placeholder Trade Events
|
||||
|
||||
**File**: `data/src/providers/databento/mod.rs:450-461`
|
||||
**Current**: Unsupported event types create synthetic `trade_id: Some("placeholder")` events.
|
||||
|
||||
**Fix**: Return `None` for unsupported event types. Log at `debug!` level. Do not create synthetic data that corrupts downstream analytics.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Trading Service Core
|
||||
|
||||
**Goal**: Real features, real risk enforcement, real order matching.
|
||||
|
||||
### 2.1 Feature Extraction (CRITICAL)
|
||||
|
||||
**File**: `services/trading_service/src/ensemble_coordinator.rs:569`
|
||||
**Current**: `fetch_features_for_symbol()` returns `vec![0.5; 16]` constant.
|
||||
|
||||
**Fix**:
|
||||
- Query market data cache (Redis or in-memory ring buffer) for latest OHLCV
|
||||
- Use `data` crate's feature extractors for technical indicators
|
||||
- Build 51-dim `FeatureVector` matching the canonical layout (price 0-9, technical 10-19, order book 20-29, microstructure 30-39, position/risk 40-50)
|
||||
- Fall back to last-known features if cache miss (not zeros)
|
||||
|
||||
### 2.2 Risk Engine Wiring (CRITICAL)
|
||||
|
||||
**File**: `services/trading_service/src/state.rs:539-629`
|
||||
**Current**: `RiskEngine` and `MLEngine` are empty structs.
|
||||
|
||||
**Fix**:
|
||||
- `RiskEngine` wraps `risk::RiskEngine` (41k lines, production-ready)
|
||||
- `MLEngine` wraps `EnsembleCoordinator` with loaded model adapters
|
||||
- Both initialized during service startup with config from `config/` crate
|
||||
- Connected to the trading pipeline via `AppState`
|
||||
|
||||
### 2.3 VaR Calculation (CRITICAL)
|
||||
|
||||
**File**: `services/trading_service/src/core/risk_manager.rs:899-919`
|
||||
**Current**: Placeholder percentile calculation.
|
||||
|
||||
**Fix**: Use `risk::VarCalculator` which implements:
|
||||
- Monte Carlo VaR (10,000 simulations, configurable)
|
||||
- Historical VaR (250-day window)
|
||||
- Parametric VaR (variance-covariance)
|
||||
- All three are already implemented and tested in `risk/` crate (632 tests)
|
||||
|
||||
### 2.4 Order Book Matching (HIGH)
|
||||
|
||||
**File**: `services/trading_service/src/core/order_manager.rs:544-552`
|
||||
**Current**: `consume_entry` and `reduce_quantity` methods don't exist on `SmallBatchRing`.
|
||||
|
||||
**Fix**:
|
||||
- Add `consume_entry()` to `SmallBatchRing` — remove fully filled entry
|
||||
- Add `reduce_quantity()` to `SmallBatchRing` — partial fill, reduce remaining
|
||||
- Wire into the order matching loop for both full and partial fills
|
||||
|
||||
### 2.5 API Gateway ML Proxy (CRITICAL)
|
||||
|
||||
**File**: `services/api_gateway/src/handlers/ml.rs:217-226`
|
||||
**Current**: Returns `prediction = 0.5`, `confidence = 0.75` hardcoded.
|
||||
|
||||
**Fix**: gRPC proxy to `ml_training_service` inference endpoint:
|
||||
- Forward feature vector from request body
|
||||
- Call `ml_training_service::Predict` gRPC method
|
||||
- Return real prediction and confidence
|
||||
- Add circuit breaker: if ML service unavailable, return error (not fake data)
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Broker Connectivity
|
||||
|
||||
**Goal**: Both AMP Futures (FIX 4.4) and Interactive Brokers (TWS) execute real orders.
|
||||
|
||||
### 3.1 FIX 4.4 Protocol (AMP Futures)
|
||||
|
||||
**File**: `trading_engine/src/brokers/fix.rs` (47 lines → ~2000 lines)
|
||||
**Current**: `FixMessage` struct with HashMap fields only.
|
||||
|
||||
**Implementation**:
|
||||
|
||||
```
|
||||
FixEngine
|
||||
├── FixSession (state machine)
|
||||
│ ├── States: Disconnected → Connected → LoggedIn → Active → LoggingOut
|
||||
│ ├── Heartbeat/TestRequest (tag 35=0/1)
|
||||
│ ├── Sequence numbers (tag 34) with persistence
|
||||
│ └── Gap fill / resend request (tag 35=2/4)
|
||||
├── FixTransport (tokio TCP + optional TLS)
|
||||
│ ├── Framed reader/writer on SOH delimiter
|
||||
│ ├── Async read loop with Waker
|
||||
│ └── Reconnect with exponential backoff
|
||||
├── FixCodec
|
||||
│ ├── Serialize: BeginString(8) + BodyLength(9) + ... + Checksum(10)
|
||||
│ ├── Parse: Split on SOH, validate checksum, extract tag=value pairs
|
||||
│ └── Checksum: sum of all bytes mod 256, formatted as 3-digit string
|
||||
└── FixSequenceStore (file-backed)
|
||||
├── Persist outgoing sequence number
|
||||
└── Recovery after crash: resend from last persisted
|
||||
```
|
||||
|
||||
**Application messages**:
|
||||
- `NewOrderSingle` (35=D): ClOrdID, Symbol, Side, OrderQty, OrdType, Price, TimeInForce
|
||||
- `ExecutionReport` (35=8): Parse fills, partial fills, rejects, cancels
|
||||
- `OrderCancelRequest` (35=F): Cancel by ClOrdID
|
||||
- `OrderCancelReplaceRequest` (35=G): Modify price/quantity
|
||||
|
||||
### 3.2 Interactive Brokers (TWS API)
|
||||
|
||||
**File**: `trading_engine/src/brokers/interactive_brokers.rs` (~150 lines → ~800 lines)
|
||||
**Current**: Stub returning hardcoded "IB123456".
|
||||
|
||||
**Implementation**:
|
||||
- Real `ibapi::Client` connection to TWS/IB Gateway (port 7496/7497)
|
||||
- `submit_order()`: `client.place_order()` with proper `Contract` + `Order` structs
|
||||
- `cancel_order()`: `client.cancel_order()`
|
||||
- `get_positions()`: `client.req_positions()` with position callback
|
||||
- `subscribe_executions()`: `client.req_executions()` → channel of `Execution` events
|
||||
- Account monitoring: `client.req_account_updates()` for real-time P&L
|
||||
|
||||
### 3.3 ICMarkets Connector
|
||||
|
||||
**File**: `trading_engine/src/brokers/icmarkets.rs`
|
||||
**Current**: Copy-paste of IB stub.
|
||||
|
||||
**Implementation**: Uses FIX 4.4 (same engine as AMP Futures, different session config):
|
||||
- ICMarkets FIX endpoint configuration
|
||||
- Symbol mapping (ICM uses different symbology)
|
||||
- Session credentials from config/Vault
|
||||
|
||||
### 3.4 Order Router
|
||||
|
||||
**File**: `trading_engine/src/routing.rs` (48 lines → ~200 lines)
|
||||
**Current**: String matching on `config.default_broker`.
|
||||
|
||||
**Fix**:
|
||||
- Type-safe `BrokerType` enum (AMP, IB, ICMarkets)
|
||||
- Latency-aware routing: measure round-trip per broker, route to fastest for asset class
|
||||
- Failover: if primary broker disconnected, route to secondary
|
||||
- Smart routing: futures → AMP, equities → IB, forex → ICMarkets
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Execution Algorithms
|
||||
|
||||
**Goal**: Production-grade algorithmic execution beyond market/limit orders.
|
||||
|
||||
### 4.1 TWAP (Time-Weighted Average Price)
|
||||
|
||||
- Slice parent order into N child orders over time window
|
||||
- Random jitter ±10% of slice interval to avoid detection
|
||||
- Cancel remaining slices on complete fill
|
||||
- Track execution quality: actual avg price vs TWAP benchmark
|
||||
|
||||
### 4.2 VWAP (Volume-Weighted Average Price)
|
||||
|
||||
- Fetch historical volume profile for symbol+time window
|
||||
- Allocate child order sizes proportional to expected volume per bucket
|
||||
- Track participation rate (our volume / total market volume)
|
||||
- Dynamic adjustment: if behind schedule, increase next slice; if ahead, reduce
|
||||
|
||||
### 4.3 Iceberg
|
||||
|
||||
- Show `visible_qty` on book (e.g., 10% of total)
|
||||
- On fill notification, replenish from hidden `total_qty`
|
||||
- Randomize visible quantity ±20% per replenish to avoid detection
|
||||
- Track total filled vs remaining hidden
|
||||
|
||||
### 4.4 Sniper (Liquidity Detection)
|
||||
|
||||
- Monitor L2 order book for large resting orders at target price levels
|
||||
- When detected: send aggressive IOC (Immediate-or-Cancel) to hit the resting liquidity
|
||||
- Requires sub-millisecond reaction time → integrate with SIMD order book processing
|
||||
- Configurable minimum quantity threshold for "large" detection
|
||||
|
||||
---
|
||||
|
||||
## Phase 5: Security & Compliance
|
||||
|
||||
### 5.1 mTLS Certificate Verification (CRITICAL)
|
||||
|
||||
**File**: `services/api_gateway/src/auth/mtls/validator.rs:391`
|
||||
**Current**: `Ok(())` — accepts any certificate.
|
||||
|
||||
**Fix**: Implement X.509 signature chain verification:
|
||||
- Verify client cert signed by trusted CA (from CA bundle)
|
||||
- Check validity period (not expired, not yet valid)
|
||||
- Check key usage extensions (clientAuth)
|
||||
- Use `rustls` (already a dependency) for chain validation
|
||||
|
||||
### 5.2 OCSP Revocation (HIGH)
|
||||
|
||||
**File**: `services/api_gateway/src/auth/mtls/revocation.rs:303`
|
||||
**Current**: Logs warning, returns "not revoked".
|
||||
|
||||
**Fix**:
|
||||
- Extract OCSP responder URL from certificate's Authority Information Access (AIA) extension
|
||||
- HTTP POST OCSP request with certificate serial number
|
||||
- Parse OCSP response (good/revoked/unknown)
|
||||
- Cache response per serial (TTL from nextUpdate field)
|
||||
- Soft-fail: if responder unreachable, configurable policy (allow with warning vs reject)
|
||||
|
||||
### 5.3 MiFID II Compliance
|
||||
|
||||
**Files**: `trading_engine/src/compliance/` (commented out modules)
|
||||
|
||||
**Implementation**:
|
||||
- **Transaction reporting** (RTS 25): Record all trade decisions with timestamps, rationale
|
||||
- **Best execution** (RTS 27/28): Track execution venues, prices achieved, compare to reference
|
||||
- **Client order handling**: Priority queuing, time-stamping at microsecond precision
|
||||
|
||||
### 5.4 MAR Market Abuse Surveillance
|
||||
|
||||
**Implementation**:
|
||||
- **Wash trading detection**: Cross-reference buy/sell orders from same account
|
||||
- **Layering/spoofing**: Detect orders placed and cancelled quickly on same side
|
||||
- **Unusual volume**: Statistical anomaly detection on trading volume per symbol
|
||||
|
||||
### 5.5 Audit Trails
|
||||
|
||||
**File**: `trading_engine/src/compliance/sox_compliance.rs`
|
||||
**Current**: TODO for async persistence.
|
||||
|
||||
**Fix**: Async write to PostgreSQL + S3 for immutable audit log. Every trade decision, risk check, and compliance event gets a tamper-evident record.
|
||||
|
||||
---
|
||||
|
||||
## Phase 6: Observability & Infrastructure
|
||||
|
||||
### 6.1 OpenTelemetry Re-enable (HIGH)
|
||||
|
||||
**File**: `common/src/lib.rs` — observability module commented out
|
||||
**Current**: All 3 major services have `// TODO: Re-enable when observability module is fixed`.
|
||||
|
||||
**Fix**: Fix `common/src/observability/` type errors with `tracing_subscriber` v0.3. Re-enable OTLP exporter + Jaeger integration. Add spans to critical paths (order lifecycle, risk checks, ML inference).
|
||||
|
||||
### 6.2 Prometheus Panic Fix (HIGH)
|
||||
|
||||
**File**: `trading_engine/src/trading_operations.rs:42-245`
|
||||
**Current**: 11 `panic!()` calls on metrics registration failure.
|
||||
|
||||
**Fix**: Replace with `try_register()` or `register_or_get()` pattern. If metric already registered (e.g., after hot reload), reuse existing. Never panic on metrics infrastructure failure.
|
||||
|
||||
### 6.3 Kill Switch Monitor (HIGH)
|
||||
|
||||
**File**: `risk/src/safety/kill_switch.rs:363`
|
||||
**Current**: `start_monitoring()` returns `Ok(())` immediately.
|
||||
|
||||
**Fix**: `tokio::spawn` background loop:
|
||||
- Check Redis connectivity every 5s
|
||||
- If Redis unreachable for >30s, activate local kill switch (fail-safe)
|
||||
- Heartbeat broadcast to other service instances
|
||||
- Detect network partition scenarios
|
||||
|
||||
### 6.4 InfluxDB Time-Series (MEDIUM)
|
||||
|
||||
**File**: `services/backtesting_service/src/storage.rs:44`
|
||||
**Current**: `_influxdb_client: Option<()>`.
|
||||
|
||||
**Fix**: Wire `influxdb2` crate for performance metrics: latency percentiles, throughput, fill rates, slippage tracking.
|
||||
|
||||
### 6.5 Cleanup (LOW)
|
||||
|
||||
- Remove `common/src/ml_strategy_backup.rs`, `ml_strategy_fix.rs`, `ml_strategy_rsi_macd.rs` (1,157 lines orphaned)
|
||||
- Remove `common/src/types.rs.rej` (rejected patch fragment)
|
||||
- Remove dev-mode Vault token from `docker-compose.yml`
|
||||
- Consolidate overlapping GitHub Actions workflows
|
||||
|
||||
---
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- **Rewrite trading engine core**: SIMD, lock-free queues, RDTSC timing are production-ready
|
||||
- **ML crate changes**: Already hardened (ensemble inference, validation, hyperopt)
|
||||
- **Risk crate core**: 41k lines with 632 tests — already production-grade
|
||||
- **New model types**: No new ML architectures (focus on infrastructure)
|
||||
- **Multi-asset MIMO**: Deferred per prior design decision
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
Each phase includes:
|
||||
1. Unit tests for new components
|
||||
2. Integration tests for cross-crate wiring
|
||||
3. `cargo check` + `cargo test` verification before each commit
|
||||
4. Existing test suites must not regress
|
||||
Reference in New Issue
Block a user