docs: RL agent gaps design — 7 features for training quality and production readiness
Position-aware state encoding, regime detection (feature-based + HMM), slippage modeling, curriculum learning, multi-timeframe fusion (default on), online learning with EWC, and A/B testing with Thompson Sampling. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
212
docs/plans/2026-03-03-rl-agent-gaps-design.md
Normal file
212
docs/plans/2026-03-03-rl-agent-gaps-design.md
Normal file
@@ -0,0 +1,212 @@
|
||||
# RL Agent Gaps — Design Document
|
||||
|
||||
**Date:** 2026-03-03
|
||||
**Status:** Approved
|
||||
**Approach:** Vertical slices (sequential, dependency-ordered)
|
||||
|
||||
## Overview
|
||||
|
||||
Audit of DQN and PPO agents identified 7 gaps across training quality and production readiness. This design addresses all 7 in dependency order.
|
||||
|
||||
**Implementation order:**
|
||||
1. Position-aware state encoding (foundation)
|
||||
2. Regime detection (feature-based + HMM)
|
||||
3. Slippage modeling
|
||||
4. Curriculum learning (uses regime detection)
|
||||
5. Multi-timeframe fusion (default on)
|
||||
6. Online learning (needs stable architecture)
|
||||
7. A/B testing framework (deployment layer)
|
||||
|
||||
---
|
||||
|
||||
## 1. Position-Aware State Encoding
|
||||
|
||||
**Problem:** Agent doesn't see its own position — learns constraints by trial-and-error.
|
||||
|
||||
**Design:** Extend state vector with 4 features (51 → 55 dims):
|
||||
|
||||
| Feature | Normalization | Range |
|
||||
|---------|--------------|-------|
|
||||
| `current_exposure` | Direct | [-1.0, 1.0] (Short100 to Long100) |
|
||||
| `unrealized_pnl` | Divide by EMA σ (100-bar) | Unbounded, typically [-5, 5] |
|
||||
| `bars_in_position` | log(1 + x) / log(1001) | [0, 1] |
|
||||
| `position_cost_basis` | Relative to current price, in bps | [-500, 500] typical |
|
||||
|
||||
**Location:** `crates/ml/src/features/position_features.rs`
|
||||
**Impact:** All hidden dim configs, hyperopt bounds, walk-forward feature counts update from 51 → 55.
|
||||
|
||||
---
|
||||
|
||||
## 2. Regime Detection
|
||||
|
||||
**Problem:** `detect_regime()` returns hardcoded "normal". RegimeConditionalDQN has no real input.
|
||||
|
||||
### Layer 1: Feature-Based Classifier (real-time, <1μs)
|
||||
|
||||
Input: rolling 100-bar window of OHLCV.
|
||||
|
||||
| Regime | Conditions |
|
||||
|--------|-----------|
|
||||
| Trending | ADX > 25 AND Hurst > 0.55 |
|
||||
| MeanReverting | Hurst < 0.45 AND vol < median |
|
||||
| Volatile | vol > 2σ above mean OR ADX < 15 |
|
||||
| Normal | Default |
|
||||
|
||||
Returns existing `RegimeType` enum.
|
||||
|
||||
### Layer 2: HMM (offline, periodic refit)
|
||||
|
||||
- 3-state HMM (Low-Vol / Trending / Crisis) on daily returns
|
||||
- Fitted during walk-forward setup, refit weekly
|
||||
- Provides transition matrix for persistence estimation
|
||||
- Validates feature-based classifier accuracy
|
||||
|
||||
**Location:** `crates/ml/src/regime_detection/` (replace stub). `feature_classifier.rs` + `hmm.rs`.
|
||||
**Integration:** Wire into `RegimeConditionalDQN` head dispatch.
|
||||
|
||||
---
|
||||
|
||||
## 3. Slippage Modeling
|
||||
|
||||
**Problem:** Fixed 15/5/10 bps costs don't reflect volume-dependent market impact.
|
||||
|
||||
### `SlippageModel` trait
|
||||
|
||||
Two implementations:
|
||||
|
||||
**`LinearImpactModel`** (new default):
|
||||
```
|
||||
slippage = spread/2 + sqrt_impact * sqrt(volume / ADV)
|
||||
```
|
||||
- `spread`: estimated from OHLCV as `0.5 * (high - low) / close * normalization`
|
||||
- `sqrt_impact`: per-instrument (ES ≈ 0.1, 6E ≈ 0.3 bps/√lot)
|
||||
- `ADV`: 20-day rolling average daily volume
|
||||
- Temporary impact decays: `temp_impact * exp(-t / decay_half_life)`
|
||||
|
||||
**`FixedCostModel`** (backward compat):
|
||||
- Wraps existing 15/5/10 bps constants
|
||||
|
||||
**Location:** `crates/ml/src/risk/slippage.rs`
|
||||
**Integration:** Replace fixed `cost_weight` in `dqn/reward.rs` with `slippage_model.estimate_cost(action, state)`. Same for PPO and evaluation engine.
|
||||
|
||||
---
|
||||
|
||||
## 4. Curriculum Learning
|
||||
|
||||
**Problem:** 45-action space + all regimes from step 0 slows convergence.
|
||||
|
||||
### 3-Phase Schedule
|
||||
|
||||
| Phase | Exposure Actions | Regimes | Gate |
|
||||
|-------|-----------------|---------|------|
|
||||
| 1: Basic | Flat, Long50, Short50 (27 actions) | Normal only | Sharpe > 0.5 for 3 folds |
|
||||
| 2: Full Position | All 5 exposures (45 actions) | Normal + Trending | Sharpe > 0.3 for 2 folds |
|
||||
| 3: All Regimes | All 45 | All | Final phase |
|
||||
|
||||
**Mechanism:**
|
||||
- Action masking expanded per phase (reuse `get_valid_action_mask()`)
|
||||
- Regime filtering via Section 2 detection
|
||||
- Phase tracked in checkpoint metadata
|
||||
- Manual override: `--curriculum-phase 3`
|
||||
|
||||
**Location:** `crates/ml/src/trainers/curriculum.rs`
|
||||
|
||||
---
|
||||
|
||||
## 5. Multi-Timeframe Feature Fusion (Default On)
|
||||
|
||||
**Problem:** Single 1-minute resolution. No cross-timeframe context.
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
1m bars ──→ [Encoder] ──→ emb_1m (64) ─┐
|
||||
├─ resample 5m ──→ [Encoder] ──→ emb_5m (64) ─┤
|
||||
├─ resample 15m ──→ [Encoder] ──→ emb_15m (64) ─┼─→ [Concat 256] ─→ [Linear 128] ─→ macro_state
|
||||
└─ resample 1h ──→ [Encoder] ──→ emb_1h (64) ─┘
|
||||
```
|
||||
|
||||
- Resample from 1m OHLCV on-the-fly (no extra data download)
|
||||
- Per-timeframe encoder: shared-weight LSTM
|
||||
- Fusion: concatenate → linear projection → 128-dim macro context
|
||||
- Combined state: position features (55) + macro context (128) = 183 input dims
|
||||
- **Default on** — no feature gate
|
||||
|
||||
**Location:** `crates/ml/src/features/multi_timeframe.rs`
|
||||
|
||||
---
|
||||
|
||||
## 6. Online Learning (Per-Trade)
|
||||
|
||||
**Problem:** Models freeze after walk-forward training. No adaptation to regime shifts.
|
||||
|
||||
### Core Mechanism
|
||||
|
||||
- Rolling experience buffer: 10,000 transitions (configurable)
|
||||
- Mini-update every 100 trades:
|
||||
- DQN: sample mini-batch, TD loss, weight update
|
||||
- PPO: N-step trajectory, advantages, single epoch
|
||||
- **EWC regularization** prevents catastrophic forgetting:
|
||||
- Fisher Information Matrix (diagonal approx) computed after initial training
|
||||
- `online_loss = task_loss + λ * Σ F_i * (θ_i - θ*_i)²`
|
||||
- λ default: 1000.0, Fisher recomputed every 10,000 steps
|
||||
|
||||
### Safety Rails
|
||||
|
||||
| Rail | Config | Default |
|
||||
|------|--------|---------|
|
||||
| Gradient clipping | `max_grad_norm` | 1.0 |
|
||||
| Learning rate | `online_lr_multiplier` | 0.1× base LR |
|
||||
| Checkpoint rollback | Sharpe degrades > 20% over 500 trades | Auto-rollback |
|
||||
| Kill switch | Rolling Sharpe < -1.0 | Freeze + alert |
|
||||
|
||||
**Location:** `crates/ml/src/trainers/online_learning.rs`
|
||||
**Integration:** Called from `trading_service` after trade execution.
|
||||
|
||||
---
|
||||
|
||||
## 7. A/B Testing Framework
|
||||
|
||||
**Problem:** No safe mechanism to deploy and compare new models.
|
||||
|
||||
### Architecture
|
||||
|
||||
Two slots: **Champion** (production) and **Challenger** (candidate).
|
||||
|
||||
**Traffic allocation via Thompson Sampling:**
|
||||
- Each model: Beta(α, β) distribution on win rate
|
||||
- Sample from both; higher sample gets the trade
|
||||
- Initial: Champion Beta(10, 1), Challenger Beta(1, 1)
|
||||
|
||||
**Shadow mode:** Challenger runs inference but never executes (100% Champion).
|
||||
|
||||
### Metrics per model
|
||||
|
||||
- Rolling Sharpe (500 trades)
|
||||
- Win rate, mean PnL/trade
|
||||
- Max drawdown
|
||||
- Action diversity (entropy)
|
||||
|
||||
### Promotion criteria (all required)
|
||||
|
||||
1. Challenger Sharpe > Champion Sharpe (Bayesian P > 0.95)
|
||||
2. Challenger max DD ≤ 1.2 × Champion max DD
|
||||
3. Minimum 500 evaluated trades
|
||||
4. Manual approval via existing `ApproveModel` gRPC
|
||||
|
||||
**Location:** `crates/ml/src/deployment/ab_testing.rs`
|
||||
**Integration:** `trading_service` ensemble infrastructure + `ApproveModel`/`RejectModel` handlers.
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
| # | Feature | New Files | Impact |
|
||||
|---|---------|-----------|--------|
|
||||
| 1 | Position-aware state | `features/position_features.rs` | State 51→55 dims |
|
||||
| 2 | Regime detection | `regime_detection/{feature_classifier,hmm}.rs` | Replace stub |
|
||||
| 3 | Slippage model | `risk/slippage.rs` | Reward computation |
|
||||
| 4 | Curriculum learning | `trainers/curriculum.rs` | Trainer loops |
|
||||
| 5 | Multi-timeframe fusion | `features/multi_timeframe.rs` | State 55→183 dims (default on) |
|
||||
| 6 | Online learning | `trainers/online_learning.rs` | Inference hot path |
|
||||
| 7 | A/B testing | `deployment/ab_testing.rs` | Trading service |
|
||||
Reference in New Issue
Block a user