# RL Agent Gaps — Design Document

**Date:** 2026-03-03
**Status:** Approved
**Approach:** Vertical slices (sequential, dependency-ordered)

## Overview

Audit of DQN and PPO agents identified 7 gaps across training quality and production readiness. This design addresses all 7 in dependency order.

**Implementation order:**
1. Position-aware state encoding (foundation)
2. Regime detection (feature-based + HMM)
3. Slippage modeling
4. Curriculum learning (uses regime detection)
5. Multi-timeframe fusion (default on)
6. Online learning (needs stable architecture)
7. A/B testing framework (deployment layer)

---

## 1. Position-Aware State Encoding

**Problem:** Agent doesn't see its own position — learns constraints by trial-and-error.

**Design:** Extend state vector with 4 features (51 → 55 dims):

| Feature | Normalization | Range |
|---------|--------------|-------|
| `current_exposure` | Direct | [-1.0, 1.0] (Short100 to Long100) |
| `unrealized_pnl` | Divide by EMA σ (100-bar) | Unbounded, typically [-5, 5] |
| `bars_in_position` | log(1 + x) / log(1001) | [0, 1] |
| `position_cost_basis` | Relative to current price, in bps | [-500, 500] typical |

**Location:** `crates/ml/src/features/position_features.rs`
**Impact:** All hidden dim configs, hyperopt bounds, walk-forward feature counts update from 51 → 55.

---

## 2. Regime Detection

**Problem:** `detect_regime()` returns hardcoded "normal". RegimeConditionalDQN has no real input.

### Layer 1: Feature-Based Classifier (real-time, <1μs)

Input: rolling 100-bar window of OHLCV.

| Regime | Conditions |
|--------|-----------|
| Trending | ADX > 25 AND Hurst > 0.55 |
| MeanReverting | Hurst < 0.45 AND vol < median |
| Volatile | vol > 2σ above mean OR ADX < 15 |
| Normal | Default |

Returns existing `RegimeType` enum.

### Layer 2: HMM (offline, periodic refit)

- 3-state HMM (Low-Vol / Trending / Crisis) on daily returns
- Fitted during walk-forward setup, refit weekly
- Provides transition matrix for persistence estimation
- Validates feature-based classifier accuracy

**Location:** `crates/ml/src/regime_detection/` (replace stub). `feature_classifier.rs` + `hmm.rs`.
**Integration:** Wire into `RegimeConditionalDQN` head dispatch.

---

## 3. Slippage Modeling

**Problem:** Fixed 15/5/10 bps costs don't reflect volume-dependent market impact.

### `SlippageModel` trait

Two implementations:

**`LinearImpactModel`** (new default):
```
slippage = spread/2 + sqrt_impact * sqrt(volume / ADV)
```
- `spread`: estimated from OHLCV as `0.5 * (high - low) / close * normalization`
- `sqrt_impact`: per-instrument (ES ≈ 0.1, 6E ≈ 0.3 bps/√lot)
- `ADV`: 20-day rolling average daily volume
- Temporary impact decays: `temp_impact * exp(-t / decay_half_life)`

**`FixedCostModel`** (backward compat):
- Wraps existing 15/5/10 bps constants

**Location:** `crates/ml/src/risk/slippage.rs`
**Integration:** Replace fixed `cost_weight` in `dqn/reward.rs` with `slippage_model.estimate_cost(action, state)`. Same for PPO and evaluation engine.

---

## 4. Curriculum Learning

**Problem:** 45-action space + all regimes from step 0 slows convergence.

### 3-Phase Schedule

| Phase | Exposure Actions | Regimes | Gate |
|-------|-----------------|---------|------|
| 1: Basic | Flat, Long50, Short50 (27 actions) | Normal only | Sharpe > 0.5 for 3 folds |
| 2: Full Position | All 5 exposures (45 actions) | Normal + Trending | Sharpe > 0.3 for 2 folds |
| 3: All Regimes | All 45 | All | Final phase |

**Mechanism:**
- Action masking expanded per phase (reuse `get_valid_action_mask()`)
- Regime filtering via Section 2 detection
- Phase tracked in checkpoint metadata
- Manual override: `--curriculum-phase 3`

**Location:** `crates/ml/src/trainers/curriculum.rs`

---

## 5. Multi-Timeframe Feature Fusion (Default On)

**Problem:** Single 1-minute resolution. No cross-timeframe context.

### Architecture

```
1m bars ──→ [Encoder] ──→ emb_1m (64)  ─┐
  ├─ resample 5m  ──→ [Encoder] ──→ emb_5m (64)  ─┤
  ├─ resample 15m ──→ [Encoder] ──→ emb_15m (64) ─┼─→ [Concat 256] ─→ [Linear 128] ─→ macro_state
  └─ resample 1h  ──→ [Encoder] ──→ emb_1h (64)  ─┘
```

- Resample from 1m OHLCV on-the-fly (no extra data download)
- Per-timeframe encoder: shared-weight LSTM
- Fusion: concatenate → linear projection → 128-dim macro context
- Combined state: position features (55) + macro context (128) = 183 input dims
- **Default on** — no feature gate

**Location:** `crates/ml/src/features/multi_timeframe.rs`

---

## 6. Online Learning (Per-Trade)

**Problem:** Models freeze after walk-forward training. No adaptation to regime shifts.

### Core Mechanism

- Rolling experience buffer: 10,000 transitions (configurable)
- Mini-update every 100 trades:
  - DQN: sample mini-batch, TD loss, weight update
  - PPO: N-step trajectory, advantages, single epoch
- **EWC regularization** prevents catastrophic forgetting:
  - Fisher Information Matrix (diagonal approx) computed after initial training
  - `online_loss = task_loss + λ * Σ F_i * (θ_i - θ*_i)²`
  - λ default: 1000.0, Fisher recomputed every 10,000 steps

### Safety Rails

| Rail | Config | Default |
|------|--------|---------|
| Gradient clipping | `max_grad_norm` | 1.0 |
| Learning rate | `online_lr_multiplier` | 0.1× base LR |
| Checkpoint rollback | Sharpe degrades > 20% over 500 trades | Auto-rollback |
| Kill switch | Rolling Sharpe < -1.0 | Freeze + alert |

**Location:** `crates/ml/src/trainers/online_learning.rs`
**Integration:** Called from `trading_service` after trade execution.

---

## 7. A/B Testing Framework

**Problem:** No safe mechanism to deploy and compare new models.

### Architecture

Two slots: **Champion** (production) and **Challenger** (candidate).

**Traffic allocation via Thompson Sampling:**
- Each model: Beta(α, β) distribution on win rate
- Sample from both; higher sample gets the trade
- Initial: Champion Beta(10, 1), Challenger Beta(1, 1)

**Shadow mode:** Challenger runs inference but never executes (100% Champion).

### Metrics per model

- Rolling Sharpe (500 trades)
- Win rate, mean PnL/trade
- Max drawdown
- Action diversity (entropy)

### Promotion criteria (all required)

1. Challenger Sharpe > Champion Sharpe (Bayesian P > 0.95)
2. Challenger max DD ≤ 1.2 × Champion max DD
3. Minimum 500 evaluated trades
4. Manual approval via existing `ApproveModel` gRPC

**Location:** `crates/ml/src/deployment/ab_testing.rs`
**Integration:** `trading_service` ensemble infrastructure + `ApproveModel`/`RejectModel` handlers.

---

## Summary

| # | Feature | New Files | Impact |
|---|---------|-----------|--------|
| 1 | Position-aware state | `features/position_features.rs` | State 51→55 dims |
| 2 | Regime detection | `regime_detection/{feature_classifier,hmm}.rs` | Replace stub |
| 3 | Slippage model | `risk/slippage.rs` | Reward computation |
| 4 | Curriculum learning | `trainers/curriculum.rs` | Trainer loops |
| 5 | Multi-timeframe fusion | `features/multi_timeframe.rs` | State 55→183 dims (default on) |
| 6 | Online learning | `trainers/online_learning.rs` | Inference hot path |
| 7 | A/B testing | `deployment/ab_testing.rs` | Trading service |