From 44b38e80145d156842dd14261e2ab16586b3c1c7 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Tue, 3 Mar 2026 02:23:38 +0100 Subject: [PATCH] =?UTF-8?q?docs:=20RL=20agent=20gaps=20design=20=E2=80=94?= =?UTF-8?q?=207=20features=20for=20training=20quality=20and=20production?= =?UTF-8?q?=20readiness?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Position-aware state encoding, regime detection (feature-based + HMM), slippage modeling, curriculum learning, multi-timeframe fusion (default on), online learning with EWC, and A/B testing with Thompson Sampling. Co-Authored-By: Claude Opus 4.6 --- docs/plans/2026-03-03-rl-agent-gaps-design.md | 212 ++++++++++++++++++ 1 file changed, 212 insertions(+) create mode 100644 docs/plans/2026-03-03-rl-agent-gaps-design.md diff --git a/docs/plans/2026-03-03-rl-agent-gaps-design.md b/docs/plans/2026-03-03-rl-agent-gaps-design.md new file mode 100644 index 000000000..3480ed8d7 --- /dev/null +++ b/docs/plans/2026-03-03-rl-agent-gaps-design.md @@ -0,0 +1,212 @@ +# RL Agent Gaps — Design Document + +**Date:** 2026-03-03 +**Status:** Approved +**Approach:** Vertical slices (sequential, dependency-ordered) + +## Overview + +Audit of DQN and PPO agents identified 7 gaps across training quality and production readiness. This design addresses all 7 in dependency order. + +**Implementation order:** +1. Position-aware state encoding (foundation) +2. Regime detection (feature-based + HMM) +3. Slippage modeling +4. Curriculum learning (uses regime detection) +5. Multi-timeframe fusion (default on) +6. Online learning (needs stable architecture) +7. A/B testing framework (deployment layer) + +--- + +## 1. Position-Aware State Encoding + +**Problem:** Agent doesn't see its own position — learns constraints by trial-and-error. + +**Design:** Extend state vector with 4 features (51 → 55 dims): + +| Feature | Normalization | Range | +|---------|--------------|-------| +| `current_exposure` | Direct | [-1.0, 1.0] (Short100 to Long100) | +| `unrealized_pnl` | Divide by EMA σ (100-bar) | Unbounded, typically [-5, 5] | +| `bars_in_position` | log(1 + x) / log(1001) | [0, 1] | +| `position_cost_basis` | Relative to current price, in bps | [-500, 500] typical | + +**Location:** `crates/ml/src/features/position_features.rs` +**Impact:** All hidden dim configs, hyperopt bounds, walk-forward feature counts update from 51 → 55. + +--- + +## 2. Regime Detection + +**Problem:** `detect_regime()` returns hardcoded "normal". RegimeConditionalDQN has no real input. + +### Layer 1: Feature-Based Classifier (real-time, <1μs) + +Input: rolling 100-bar window of OHLCV. + +| Regime | Conditions | +|--------|-----------| +| Trending | ADX > 25 AND Hurst > 0.55 | +| MeanReverting | Hurst < 0.45 AND vol < median | +| Volatile | vol > 2σ above mean OR ADX < 15 | +| Normal | Default | + +Returns existing `RegimeType` enum. + +### Layer 2: HMM (offline, periodic refit) + +- 3-state HMM (Low-Vol / Trending / Crisis) on daily returns +- Fitted during walk-forward setup, refit weekly +- Provides transition matrix for persistence estimation +- Validates feature-based classifier accuracy + +**Location:** `crates/ml/src/regime_detection/` (replace stub). `feature_classifier.rs` + `hmm.rs`. +**Integration:** Wire into `RegimeConditionalDQN` head dispatch. + +--- + +## 3. Slippage Modeling + +**Problem:** Fixed 15/5/10 bps costs don't reflect volume-dependent market impact. + +### `SlippageModel` trait + +Two implementations: + +**`LinearImpactModel`** (new default): +``` +slippage = spread/2 + sqrt_impact * sqrt(volume / ADV) +``` +- `spread`: estimated from OHLCV as `0.5 * (high - low) / close * normalization` +- `sqrt_impact`: per-instrument (ES ≈ 0.1, 6E ≈ 0.3 bps/√lot) +- `ADV`: 20-day rolling average daily volume +- Temporary impact decays: `temp_impact * exp(-t / decay_half_life)` + +**`FixedCostModel`** (backward compat): +- Wraps existing 15/5/10 bps constants + +**Location:** `crates/ml/src/risk/slippage.rs` +**Integration:** Replace fixed `cost_weight` in `dqn/reward.rs` with `slippage_model.estimate_cost(action, state)`. Same for PPO and evaluation engine. + +--- + +## 4. Curriculum Learning + +**Problem:** 45-action space + all regimes from step 0 slows convergence. + +### 3-Phase Schedule + +| Phase | Exposure Actions | Regimes | Gate | +|-------|-----------------|---------|------| +| 1: Basic | Flat, Long50, Short50 (27 actions) | Normal only | Sharpe > 0.5 for 3 folds | +| 2: Full Position | All 5 exposures (45 actions) | Normal + Trending | Sharpe > 0.3 for 2 folds | +| 3: All Regimes | All 45 | All | Final phase | + +**Mechanism:** +- Action masking expanded per phase (reuse `get_valid_action_mask()`) +- Regime filtering via Section 2 detection +- Phase tracked in checkpoint metadata +- Manual override: `--curriculum-phase 3` + +**Location:** `crates/ml/src/trainers/curriculum.rs` + +--- + +## 5. Multi-Timeframe Feature Fusion (Default On) + +**Problem:** Single 1-minute resolution. No cross-timeframe context. + +### Architecture + +``` +1m bars ──→ [Encoder] ──→ emb_1m (64) ─┐ + ├─ resample 5m ──→ [Encoder] ──→ emb_5m (64) ─┤ + ├─ resample 15m ──→ [Encoder] ──→ emb_15m (64) ─┼─→ [Concat 256] ─→ [Linear 128] ─→ macro_state + └─ resample 1h ──→ [Encoder] ──→ emb_1h (64) ─┘ +``` + +- Resample from 1m OHLCV on-the-fly (no extra data download) +- Per-timeframe encoder: shared-weight LSTM +- Fusion: concatenate → linear projection → 128-dim macro context +- Combined state: position features (55) + macro context (128) = 183 input dims +- **Default on** — no feature gate + +**Location:** `crates/ml/src/features/multi_timeframe.rs` + +--- + +## 6. Online Learning (Per-Trade) + +**Problem:** Models freeze after walk-forward training. No adaptation to regime shifts. + +### Core Mechanism + +- Rolling experience buffer: 10,000 transitions (configurable) +- Mini-update every 100 trades: + - DQN: sample mini-batch, TD loss, weight update + - PPO: N-step trajectory, advantages, single epoch +- **EWC regularization** prevents catastrophic forgetting: + - Fisher Information Matrix (diagonal approx) computed after initial training + - `online_loss = task_loss + λ * Σ F_i * (θ_i - θ*_i)²` + - λ default: 1000.0, Fisher recomputed every 10,000 steps + +### Safety Rails + +| Rail | Config | Default | +|------|--------|---------| +| Gradient clipping | `max_grad_norm` | 1.0 | +| Learning rate | `online_lr_multiplier` | 0.1× base LR | +| Checkpoint rollback | Sharpe degrades > 20% over 500 trades | Auto-rollback | +| Kill switch | Rolling Sharpe < -1.0 | Freeze + alert | + +**Location:** `crates/ml/src/trainers/online_learning.rs` +**Integration:** Called from `trading_service` after trade execution. + +--- + +## 7. A/B Testing Framework + +**Problem:** No safe mechanism to deploy and compare new models. + +### Architecture + +Two slots: **Champion** (production) and **Challenger** (candidate). + +**Traffic allocation via Thompson Sampling:** +- Each model: Beta(α, β) distribution on win rate +- Sample from both; higher sample gets the trade +- Initial: Champion Beta(10, 1), Challenger Beta(1, 1) + +**Shadow mode:** Challenger runs inference but never executes (100% Champion). + +### Metrics per model + +- Rolling Sharpe (500 trades) +- Win rate, mean PnL/trade +- Max drawdown +- Action diversity (entropy) + +### Promotion criteria (all required) + +1. Challenger Sharpe > Champion Sharpe (Bayesian P > 0.95) +2. Challenger max DD ≤ 1.2 × Champion max DD +3. Minimum 500 evaluated trades +4. Manual approval via existing `ApproveModel` gRPC + +**Location:** `crates/ml/src/deployment/ab_testing.rs` +**Integration:** `trading_service` ensemble infrastructure + `ApproveModel`/`RejectModel` handlers. + +--- + +## Summary + +| # | Feature | New Files | Impact | +|---|---------|-----------|--------| +| 1 | Position-aware state | `features/position_features.rs` | State 51→55 dims | +| 2 | Regime detection | `regime_detection/{feature_classifier,hmm}.rs` | Replace stub | +| 3 | Slippage model | `risk/slippage.rs` | Reward computation | +| 4 | Curriculum learning | `trainers/curriculum.rs` | Trainer loops | +| 5 | Multi-timeframe fusion | `features/multi_timeframe.rs` | State 55→183 dims (default on) | +| 6 | Online learning | `trainers/online_learning.rs` | Inference hot path | +| 7 | A/B testing | `deployment/ab_testing.rs` | Trading service |