docs: add production pipeline wiring design (6 tasks)
Wire existing components into end-to-end pipeline: 5 missing inference adapters, 10-model registration, market data feeding, integration test, Docker build validation, and CI pipeline triage. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
155
docs/plans/2026-02-23-production-pipeline-wiring-design.md
Normal file
155
docs/plans/2026-02-23-production-pipeline-wiring-design.md
Normal file
@@ -0,0 +1,155 @@
|
||||
# Production Pipeline Wiring — Design
|
||||
|
||||
**Date:** 2026-02-23
|
||||
**Status:** Approved
|
||||
**Goal:** Connect existing components into a working end-to-end pipeline, prove it with an integration test, and validate Docker/CI infrastructure.
|
||||
**Non-overlap:** Trading-universe handles data acquisition/organization. This work handles service-layer plumbing, integration testing, and infrastructure validation.
|
||||
|
||||
## Problem
|
||||
|
||||
The ML ensemble (10 models), conviction gates (7 gates), feature extraction (51-dim), and paper trading executor all exist and pass unit tests independently. But they are not wired together in the production service:
|
||||
|
||||
1. **5 of 10 models lack inference adapters** — TGGN, TLOB, KAN, xLSTM, Diffusion have trainable adapters but no `ModelInferenceAdapter` impl for the ensemble coordinator
|
||||
2. **Market data doesn't feed the ensemble** — `EnsembleCoordinator::update_market_data()` exists but nobody calls it during service startup
|
||||
3. **No end-to-end test** proving the critical path works: OHLCV → features → ensemble → gates → paper trade
|
||||
4. **Docker builds unvalidated** against the current 37-crate workspace
|
||||
5. **CI pipelines likely stale** — 30+ workflows, many reference old structure
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
CURRENT (disconnected)
|
||||
┌──────────┐ ┌──────────┐ ┌──────┐ ┌───────┐
|
||||
│ MarketGen │ │ Features │ │ 5/10 │ │ Paper │
|
||||
│ (exists) │ │ (exists) │ │Models │ │Trading│
|
||||
└──────────┘ └──────────┘ └──────┘ └───────┘
|
||||
✗ not wired ✗ not fed ✗ 5 missing ✗ not connected
|
||||
|
||||
TARGET (wired)
|
||||
MarketGen ──→ update_market_data() ──→ 51-dim features
|
||||
│
|
||||
EnsembleCoordinator (10 models)
|
||||
│
|
||||
7 Conviction Gates
|
||||
│
|
||||
PaperTradingExecutor
|
||||
```
|
||||
|
||||
## Task 1: Create 5 Missing Inference Adapters
|
||||
|
||||
**Files:** `ml/src/ensemble/adapters/{tggn,tlob,kan,xlstm,diffusion}.rs`
|
||||
|
||||
Each adapter follows the exact pattern of the existing 5 (DQN, PPO, TFT, Mamba2, Liquid):
|
||||
|
||||
| Model | Adapter Name | Input | Notes |
|
||||
|-------|-------------|-------|-------|
|
||||
| TGGN | `TggnInferenceAdapter` | 51-dim | Graph neural network, uses adjacency matrix |
|
||||
| TLOB | `TlobInferenceAdapter` | 51-dim | Transformer for limit order book |
|
||||
| KAN | `KanInferenceAdapter` | 51-dim | B-spline activations |
|
||||
| xLSTM | `XlstmInferenceAdapter` | 51-dim | sLSTM + mLSTM blocks, sequence buffer |
|
||||
| Diffusion | `DiffusionInferenceAdapter` | 51-dim | DDPM/DDIM denoising, multi-step |
|
||||
|
||||
Each implements `ModelInferenceAdapter` trait:
|
||||
- `fn predict(&self, features: &FeatureVector) -> MLResult<EnsemblePrediction>`
|
||||
- `fn model_id(&self) -> &str`
|
||||
- `fn model_type(&self) -> ModelType`
|
||||
|
||||
Register all 5 in `ml/src/ensemble/adapters/mod.rs`.
|
||||
|
||||
~100-150 lines per adapter, ~500-750 lines total.
|
||||
|
||||
**Tests:** 3 tests per adapter (creation, prediction range, determinism) — same pattern as existing adapters.
|
||||
|
||||
## Task 2: Register All 10 Models in main.rs
|
||||
|
||||
**File:** `services/trading_service/src/main.rs`
|
||||
|
||||
Add registration blocks for TGGN, TLOB, KAN, xLSTM, Diffusion following the existing pattern (match on adapter creation, wrap in `InferenceAdapterBridge`, call `register_loaded_model`).
|
||||
|
||||
Initial weights (equal across 10):
|
||||
| Model | Weight |
|
||||
|-------|--------|
|
||||
| DQN | 0.10 |
|
||||
| PPO | 0.10 |
|
||||
| TFT | 0.10 |
|
||||
| Mamba2 | 0.10 |
|
||||
| Liquid | 0.10 |
|
||||
| TGGN | 0.10 |
|
||||
| TLOB | 0.10 |
|
||||
| KAN | 0.10 |
|
||||
| xLSTM | 0.10 |
|
||||
| Diffusion | 0.10 |
|
||||
|
||||
~100 lines of registration code.
|
||||
|
||||
## Task 3: Wire Market Data → EnsembleCoordinator
|
||||
|
||||
**File:** `services/trading_service/src/main.rs`
|
||||
|
||||
The test/DBN market data generators already exist as modules (`test_market_data_generator`, `dbn_market_data_generator`). Wire one as a background task that:
|
||||
|
||||
1. Spawns a tokio task after ensemble coordinator initialization
|
||||
2. Reads from the configured market data source (test generator for dev, DBN for staging)
|
||||
3. Calls `ensemble_coordinator.update_market_data(symbol, price, volume, timestamp)` on each bar
|
||||
4. Logs warmup progress (bars received vs ~51 required for feature extraction)
|
||||
|
||||
~50 lines of wiring code. The generators already handle data production — this is just the connection.
|
||||
|
||||
## Task 4: End-to-End Integration Test
|
||||
|
||||
**File:** `services/trading_service/tests/e2e_pipeline_test.rs`
|
||||
|
||||
Single `#[tokio::test(flavor = "multi_thread")]` that proves the critical path:
|
||||
|
||||
1. Create `EnsembleCoordinator` with all 10 real candle adapters (untrained, random weights)
|
||||
2. Feed 60 synthetic OHLCV bars via `update_market_data()` (warmup period)
|
||||
3. Call `fetch_features_for_symbol()` — assert 51 real features, no NaN/Inf, no all-zeros
|
||||
4. Call `predict()` — assert `EnsembleDecision` returns with:
|
||||
- `action` is Buy/Sell/Hold
|
||||
- `confidence` in [0.0, 1.0]
|
||||
- `model_votes` has entries from all 10 models
|
||||
- Conviction gate results are populated
|
||||
5. Feed decision to `PaperTradingExecutor` — assert paper order generated (or HOLD from gate rejection)
|
||||
|
||||
No external dependencies (no PostgreSQL, no QuestDB, no Docker). Pure in-process test.
|
||||
|
||||
~150 lines.
|
||||
|
||||
## Task 5: Docker Build Validation
|
||||
|
||||
**Files:** All 6 Dockerfiles + `Dockerfile.foxhunt-build`
|
||||
|
||||
For each service:
|
||||
1. `docker build -t foxhunt-<service>:test -f services/<service>/Dockerfile .`
|
||||
2. Fix any compilation failures (missing system deps, stale paths)
|
||||
3. Verify the binary starts and health check passes: `docker run --rm foxhunt-<service>:test --help` or equivalent
|
||||
|
||||
Services:
|
||||
- `trading_service` (primary)
|
||||
- `api_gateway`
|
||||
- `ml_training_service`
|
||||
- `backtesting_service`
|
||||
- `trading_agent_service`
|
||||
- `broker_gateway_service`
|
||||
|
||||
Fix-only — no new Dockerfiles.
|
||||
|
||||
## Task 6: CI Pipeline Triage
|
||||
|
||||
**Directory:** `.github/workflows/`
|
||||
|
||||
1. Identify the 3 essential workflows: build (`ci.yml`), test (`test.yml`), lint/clippy (`compilation-guard.yml` or `build.yml`)
|
||||
2. Validate each uses `SQLX_OFFLINE=true`, correct Rust toolchain, correct workspace paths
|
||||
3. Fix broken steps (missing env vars, wrong cache keys, outdated actions)
|
||||
4. Delete obvious duplicates (e.g., `comprehensive-testing.yml` vs `comprehensive_testing.yml`)
|
||||
5. Ensure at least build + test + clippy pass on `main` branch
|
||||
|
||||
Fix-only — no new workflows.
|
||||
|
||||
## Constraints
|
||||
|
||||
- Build: `SQLX_OFFLINE=true cargo check --workspace`
|
||||
- Test: `SQLX_OFFLINE=true cargo test -p <crate> --lib`
|
||||
- Clippy: `#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::indexing_slicing)]`
|
||||
- No overlap with `feat/trading-universe-data-org` branch (data acquisition, universe definition, cache structure)
|
||||
- RTX 3050 Ti 4GB — adapter configs must use small hidden dims (32-64)
|
||||
Reference in New Issue
Block a user