docs: add production hardening implementation plan (53 tasks, 5 phases)

Phase 1: ML pipeline verification (checkpoint roundtrip, feature extraction)
Phase 2: Service production logic (hardcoded values, stub responses)
Phases 3-5: Backtesting, ML crate TODOs, infrastructure/security/cleanup

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

docs/plans/2026-02-22-production-hardening-implementation.md

@@ -0,0 +1,478 @@
+# Production Hardening & TODO Resolution — Implementation Plan
+
+> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
+
+**Goal:** Resolve all 80+ TODO/FIXME items across the 37-crate workspace, add checkpoint roundtrip and feature extraction pipeline tests, and clean up stale infrastructure — zero known gaps before GPU training.
+
+**Architecture:** 5 phases ordered by dependency. Phase 1 (ML pipeline verification) proves models survive deployment. Phase 2 (service logic) replaces hardcoded fake values. Phase 3 (backtesting) completes the backtesting service. Phase 4 (ML crate) resolves internal ML TODOs. Phase 5 (infrastructure) handles security, compliance, and cleanup.
+
+**Tech Stack:** Rust, Cargo workspace, SQLX_OFFLINE=true, Candle 0.9.1, safetensors, DBN market data, statrs (statistics)
+
+**Build/test commands:**
+- Compile: `SQLX_OFFLINE=true cargo check --workspace`
+- Test specific crate: `SQLX_OFFLINE=true cargo test -p <crate> --lib`
+- Test all: `SQLX_OFFLINE=true cargo test --workspace --lib`
+
+**Design doc:** `docs/plans/2026-02-22-production-hardening-design.md`
+
+**Plan files:**
+- Phase 1 (Tasks 1-5): `docs/plans/2026-02-22-production-hardening-phase1.md`
+- Phase 2 (Tasks 6-17): This file (below)
+- Phases 3-5 (Tasks 18-53): `docs/plans/2026-02-22-production-hardening-implementation-phases-3-5.md`
+
+**No overlap with:** dedup-cleanup (codebase deduplication) or liquid-cfc (CfC v2 modernization) plans.
+
+---
+
+## Phase 2: Service Production Logic (Tasks 6-17)
+
+Replaces stub values, hardcoded constants, and `None`-returning event converters across `trading_agent_service`, `trading_service`, and `api_gateway`. Most edits are 5-30 lines in existing functions.
+
+---
+
+### Task 6 — Wire real position/price data into `allocate_portfolio()`
+
+**Files:**
+- Modify: `services/trading_agent_service/src/service.rs` (lines 733-766, 1867, 1884)
+
+**Problem:** `AssetAllocation` fields `target_quantity`, `current_weight`, `current_quantity`, `rebalance_delta` are all `0.0`. `AllocationMetrics` fields `portfolio_sharpe`, `var_95`, `max_drawdown_estimate` are all `0.0`. `portfolio_turnover` and per-strategy `total_pnl` are `0.0`.
+
+**Steps:**
+
+1. Before the `proto_allocations` map closure (line 733), pre-fetch prices and positions:
+
+```rust
+// Pre-fetch latest prices and current positions for all allocation symbols
+let allocation_symbols: Vec<String> = allocations.keys().cloned().collect();
+
+let price_map: HashMap<String, f64> = {
+    let mut map = HashMap::new();
+    for symbol in &allocation_symbols {
+        let row: Option<(f64,)> = sqlx::query_as(
+            "SELECT price FROM market_data WHERE symbol = $1 ORDER BY timestamp DESC LIMIT 1"
+        )
+        .bind(symbol)
+        .fetch_optional(&self.db_pool)
+        .await
+        .unwrap_or(None);
+        if let Some((price,)) = row {
+            map.insert(symbol.clone(), price);
+        }
+    }
+    map
+};
+
+let position_map: HashMap<String, (f64, f64)> = {
+    let mut map = HashMap::new();
+    for symbol in &allocation_symbols {
+        let row: Option<(f64, f64)> = sqlx::query_as(
+            "SELECT quantity, average_price FROM positions WHERE symbol = $1 \
+             ORDER BY updated_at DESC LIMIT 1"
+        )
+        .bind(symbol)
+        .fetch_optional(&self.db_pool)
+        .await
+        .unwrap_or(None);
+        if let Some(pos) = row {
+            map.insert(symbol.clone(), pos);
+        }
+    }
+    map
+};
+```
+
+2. Replace the `AssetAllocation` construction with real calculations:
+
+```rust
+let proto_allocations: Vec<AssetAllocation> = allocations
+    .iter()
+    .map(|(symbol, capital)| {
+        let capital_f64 = capital.to_f64().unwrap_or(0.0);
+        let weight = if req.total_capital > 0.0 { capital_f64 / req.total_capital } else { 0.0 };
+        let price = price_map.get(symbol).copied().unwrap_or(0.0);
+        let target_quantity = if price > 0.0 { (capital_f64 / price).floor() } else { 0.0 };
+        let (current_qty, _avg_price) = position_map.get(symbol).copied().unwrap_or((0.0, 0.0));
+        let current_weight = if req.total_capital > 0.0 && price > 0.0 {
+            (current_qty * price) / req.total_capital
+        } else { 0.0 };
+        AssetAllocation {
+            symbol: symbol.clone(),
+            target_weight: weight,
+            target_capital: capital_f64,
+            target_quantity,
+            current_weight,
+            current_quantity: current_qty,
+            rebalance_delta: target_quantity - current_qty,
+        }
+    })
+    .collect();
+```
+
+3. Replace `AllocationMetrics` with parametric estimates:
+
+```rust
+let metrics = AllocationMetrics {
+    total_weight,
+    portfolio_volatility,
+    portfolio_sharpe: if portfolio_volatility > 1e-12 {
+        let weighted_return: f64 = proto_allocations.iter()
+            .map(|a| {
+                let asset = req.assets.iter().find(|x| x.symbol == a.symbol);
+                let exp_ret = asset.map(|x| x.composite_score).unwrap_or(0.0);
+                a.target_weight * exp_ret
+            })
+            .sum();
+        (weighted_return - 0.02) / portfolio_volatility
+    } else { 0.0 },
+    var_95: 1.645 * portfolio_volatility * total_allocated / 252_f64.sqrt(),
+    max_drawdown_estimate: 2.0 * portfolio_volatility,
+};
+```
+
+4. For `portfolio_turnover` (line 1867) and per-strategy P&L (line 1884), these require data not yet available (total_capital on request proto, strategy_id on trades table). Document the blocker:
+
+```rust
+// BLOCKER: portfolio_turnover requires total_capital on GetAgentPerformanceRequest proto.
+// Per-strategy P&L requires strategy_id column on trades table. Using 0.0 until then.
+portfolio_turnover: 0.0,
+```
+
+**Test:** `SQLX_OFFLINE=true cargo check -p trading_agent_service`
+
+**Commit:** `fix(trading_agent_service): wire position/price data into allocate_portfolio and metrics`
+
+---
+
+### Task 7 — Replace stub position_size and contribution_pct in `get_va_r()`
+
+**Files:**
+- Modify: `services/trading_service/src/services/risk.rs` (lines 363-368)
+
+**Problem:** `position_size: 1000.0` stub, `contribution_pct: equal_contribution_pct`.
+
+**Steps:**
+
+1. Before the per-symbol loop, read position manager:
+
+```rust
+let position_manager = self.state.position_manager.read().await;
+```
+
+2. Replace `SymbolVaR` construction:
+
+```rust
+let position_size = position_manager
+    .get_position(&symbol)
+    .map(|p| p.quantity.to_f64().unwrap_or(0.0))
+    .unwrap_or(0.0);
+
+symbol_vars.push(SymbolVaR {
+    symbol,
+    var_value: symbol_var,
+    position_size,
+    contribution_pct: if portfolio_var > 1e-12 {
+        (symbol_var / portfolio_var) * 100.0
+    } else {
+        equal_contribution_pct
+    },
+});
+```
+
+**Test:** `SQLX_OFFLINE=true cargo check -p trading_service`
+
+**Commit:** `fix(trading_service): use real position sizes and marginal VaR in get_va_r`
+
+---
+
+### Task 8 — Document feature pipeline integration blockers
+
+**Files:**
+- Modify: `services/trading_service/src/state.rs` (line 1270)
+- Modify: `services/trading_service/src/services/trading.rs` (line 693)
+
+**Problem:** Two TODOs about feature extraction pipeline. Both require infrastructure not yet built (bar aggregation, EnsembleCoordinator API change).
+
+**Steps:**
+
+1. In `state.rs:1270`, replace the TODO with a descriptive roadmap comment:
+
+```rust
+// Feature extraction from raw tick events requires bar aggregation:
+//   1. Add BarAggregator that collects ticks into 1-min bars
+//   2. On bar close, call _extractor.extract_ohlcv_features(bar)
+//   3. Publish extracted features to a broadcast channel
+// For now, features are extracted on-demand in extract_features_for_symbol().
+tracing::debug!(symbol = %event.symbol, "Market event received (feature extraction deferred to on-demand path)");
+```
+
+2. In `trading.rs:693`, replace the TODO:
+
+```rust
+// req.features contains 26 client-provided features (5 OHLCV + 21 indicators).
+// EnsembleCoordinator.generate_and_save_prediction() generates 51-dim features
+// internally. To use req.features, add generate_prediction_with_features() to
+// EnsembleCoordinator. Until then, internal feature extraction is authoritative.
+```
+
+**Test:** `SQLX_OFFLINE=true cargo check -p trading_service`
+
+**Commit:** `docs(trading_service): document feature pipeline integration blockers`
+
+---
+
+### Task 9 — Populate Order/Position/Execution protos in event converters
+
+**Files:**
+- Modify: `services/trading_service/src/services/trading.rs` (lines 1186-1234)
+
+**Problem:** `convert_to_order_event()` returns `order: None`, etc. The JSON payload is already parsed.
+
+**Steps:**
+
+1. In `convert_to_order_event()`, parse JSON payload into Order proto:
+
+```rust
+let order_data: serde_json::Value = serde_json::from_str(&event.payload).unwrap_or_default();
+let order = Some(crate::proto::trading::Order {
+    order_id: order_id.clone(),
+    symbol: order_data.get("symbol").and_then(|v| v.as_str()).unwrap_or("").to_string(),
+    side: order_data.get("side").and_then(|v| v.as_i64()).unwrap_or(0) as i32,
+    quantity: order_data.get("quantity").and_then(|v| v.as_f64()).unwrap_or(0.0),
+    // ... remaining fields from payload
+});
+```
+
+2. Apply same pattern to `convert_to_position_event()` and `convert_to_execution_event()`.
+
+**Test:** `SQLX_OFFLINE=true cargo check -p trading_service`
+
+**Commit:** `fix(trading_service): populate Order/Position/Execution protos in event converters`
+
+---
+
+### Task 10 — Wire realized_pnl in get_positions()
+
+**Files:**
+- Modify: `services/trading_service/src/services/trading.rs` (line 362)
+
+**Steps:**
+
+1. Pre-fetch realized PnL before the map closure:
+
+```rust
+let mut realized_pnl_map: HashMap<String, f64> = HashMap::new();
+for pos in &repo_positions {
+    if !realized_pnl_map.contains_key(&pos.symbol) {
+        if let Ok(pnl) = self.state.trading_repository
+            .get_realized_pnl(&pos.account_id, Some(&pos.symbol)).await {
+            realized_pnl_map.insert(pos.symbol.clone(), pnl);
+        }
+    }
+}
+```
+
+2. Use `realized_pnl_map.get(&pos.symbol).copied().unwrap_or(0.0)` in the closure.
+
+**Test:** `SQLX_OFFLINE=true cargo check -p trading_service`
+
+**Commit:** `fix(trading_service): pre-fetch realized PnL for position responses`
+
+---
+
+### Task 11 — Compute max_drawdown from PnL samples in A/B testing
+
+**Files:**
+- Modify: `services/trading_service/src/ab_testing_pipeline.rs` (line 423)
+
+**Steps:**
+
+1. Replace `max_drawdown: 0.0` with cumulative PnL drawdown calculation:
+
+```rust
+let max_drawdown = {
+    let mut peak = 0.0_f64;
+    let mut max_dd = 0.0_f64;
+    let mut cumulative = 0.0_f64;
+    for pnl in &metrics.pnl_samples {
+        cumulative += pnl;
+        if cumulative > peak { peak = cumulative; }
+        let dd = peak - cumulative;
+        if dd > max_dd { max_dd = dd; }
+    }
+    max_dd
+};
+```
+
+**Caveat:** Verify `GroupMetrics` has `pnl_samples: Vec<f64>`. If not, document the gap.
+
+**Test:** `SQLX_OFFLINE=true cargo check -p trading_service`
+
+**Commit:** `fix(trading_service): compute max_drawdown from PnL samples in A/B metrics`
+
+---
+
+### Task 12 — Document missing quantity field in ML order proxy
+
+**Files:**
+- Modify: `services/api_gateway/src/grpc/trading_proxy.rs` (line 2152)
+
+**Problem:** Backend `MlOrderResponse` proto has no `quantity` field. Gateway returns 1/0.
+
+**Steps:**
+
+1. Replace the TODO comment with a descriptive blocker:
+
+```rust
+// Backend MlOrderResponse proto does not include quantity field.
+// To return real quantity, add `int32 quantity = 7` to MlOrderResponse
+// in trading.proto and populate it from the order submission response.
+quantity: if backend_resp.executed { 1 } else { 0 },
+```
+
+**Test:** `SQLX_OFFLINE=true cargo check -p api_gateway`
+
+**Commit:** `docs(api_gateway): document missing quantity field in MlOrderResponse proto`
+
+---
+
+### Task 13 — Document per-symbol weight tracking requirements
+
+**Files:**
+- Modify: `services/trading_service/src/ensemble_coordinator.rs` (line 459)
+
+**Problem:** `symbol: "ALL".to_string()` — weights are global, not per-symbol.
+
+**Steps:**
+
+1. Replace the TODO with a roadmap comment:
+
+```rust
+// Per-symbol weight tracking requires refactoring model_weights from
+// HashMap<String, ModelWeight> (model_id) to HashMap<(String, String), ModelWeight>
+// (model_id + symbol). Also requires per-symbol PerformanceMetrics collection.
+// Emitting "ALL" until per-symbol performance data is collected.
+symbol: "ALL".to_string(),
+```
+
+**Test:** `SQLX_OFFLINE=true cargo check -p trading_service`
+
+**Commit:** `docs(trading_service): document per-symbol weight tracking requirements`
+
+---
+
+### Task 14 — Document OHLCV bar pipeline upgrade path
+
+**Files:**
+- Modify: `services/trading_service/src/state.rs` (line 466)
+
+**Steps:**
+
+1. Replace the TODO with a detailed roadmap:
+
+```rust
+// Current: tick-based feature approximation (51-dim vector from raw ticks).
+// Upgrade path:
+//   1. Add get_ohlcv_bars(symbol, timeframe, count) to MarketDataRepository
+//   2. Add BarAggregator service (streaming ticks -> OHLCV bars)
+//   3. Compute technical indicators (RSI, MACD, Bollinger, ATR) from bars
+//   4. Feed bars to UnifiedFeatureExtractor for 225-dim vector
+//   5. Update model input_dim from 51 to match new feature dimension
+// The tick approximation below is adequate for initial ensemble predictions.
+```
+
+**Test:** `SQLX_OFFLINE=true cargo check -p trading_service`
+
+**Commit:** `docs(trading_service): detail OHLCV bar pipeline upgrade path`
+
+---
+
+### Task 15 — Support safetensors checkpoint loading for DQN
+
+**Files:**
+- Modify: `services/trading_service/src/services/enhanced_ml.rs` (line 1251)
+- Possibly modify: `ml/src/dqn/agent.rs` (add delegating method)
+
+**Steps:**
+
+1. Check if `DQN::load_from_safetensors()` exists:
+```bash
+grep -n "load_from_safetensors" ml/src/dqn/dqn.rs ml/src/dqn/agent.rs
+```
+
+2. If `DQNAgent` doesn't expose it, add a delegating method to `agent.rs`:
+```rust
+pub fn load_from_safetensors(&mut self, path: &str) -> Result<(), MLError> {
+    self.dqn.load_from_safetensors(path)
+}
+```
+
+3. In `enhanced_ml.rs`, update `from_checkpoint` to try safetensors first:
+```rust
+let safetensors_path = checkpoint_path.with_extension("safetensors");
+if safetensors_path.exists() {
+    agent.load_from_safetensors(&safetensors_path.to_string_lossy())?;
+} else if checkpoint_path.exists() {
+    agent.load_checkpoint(checkpoint_path)?;
+} else {
+    return Err(MLError::ModelError("No checkpoint found".into()));
+}
+```
+
+**Test:** `SQLX_OFFLINE=true cargo check -p trading_service`
+
+**Commit:** `feat(trading_service): support safetensors checkpoint loading for DQN models`
+
+---
+
+### Task 16 — Store prediction loop shutdown sender for graceful cleanup
+
+**Files:**
+- Modify: `services/trading_service/src/main.rs` (line 492)
+
+**Steps:**
+
+1. Replace `std::mem::forget(prediction_shutdown_tx)` with an `Arc` wrapper:
+
+```rust
+let prediction_shutdown_handle = Arc::new(prediction_shutdown_tx);
+
+// In shutdown handler:
+let shutdown_handle = Arc::clone(&prediction_shutdown_handle);
+tokio::spawn(async move {
+    tokio::signal::ctrl_c().await.ok();
+    info!("Shutdown signal received, stopping prediction loop...");
+    let _ = shutdown_handle.send(());
+});
+```
+
+**Test:** `SQLX_OFFLINE=true cargo check -p trading_service`
+
+**Commit:** `fix(trading_service): store prediction loop shutdown sender for graceful cleanup`
+
+---
+
+### Task 17 — Query open orders for emergency_stop() response
+
+**Files:**
+- Modify: `services/trading_service/src/services/risk.rs` (line 642)
+
+**Steps:**
+
+1. After emergency shutdown, query order manager:
+
+```rust
+let affected_orders = {
+    let order_manager = self.state.order_manager.read().await;
+    order_manager.get_open_orders().await
+        .into_iter()
+        .map(|o| o.order_id.to_string())
+        .collect::<Vec<_>>()
+};
+info!("Emergency stop affected {} open orders", affected_orders.len());
+```
+
+**Caveat:** Verify `TradingServiceState` has `order_manager` field and `OrderManager` has `get_open_orders()`. If not, document the dependency.
+
+**Test:** `SQLX_OFFLINE=true cargo check -p trading_service`
+
+**Commit:** `fix(trading_service): query open orders for emergency_stop response`