foxhunt/docs/plans/2026-03-03-rl-agent-gaps-design.md

RL Agent Gaps — Design Document

Date: 2026-03-03 Status: Approved Approach: Vertical slices (sequential, dependency-ordered)

Overview

Audit of DQN and PPO agents identified 7 gaps across training quality and production readiness. This design addresses all 7 in dependency order.

Implementation order:

1. Position-aware state encoding (foundation)
2. Regime detection (feature-based + HMM)
3. Slippage modeling
4. Curriculum learning (uses regime detection)
5. Multi-timeframe fusion (default on)
6. Online learning (needs stable architecture)
7. A/B testing framework (deployment layer)

---

1. Position-Aware State Encoding

Problem: Agent doesn't see its own position — learns constraints by trial-and-error.

Design: Extend state vector with 4 features (51 → 55 dims):

Feature	Normalization	Range
current_exposure	Direct	[-1.0, 1.0] (Short100 to Long100)
unrealized_pnl	Divide by EMA σ (100-bar)	Unbounded, typically [-5, 5]
bars_in_position	log(1 + x) / log(1001)	[0, 1]
position_cost_basis	Relative to current price, in bps	[-500, 500] typical

Location: crates/ml/src/features/position_features.rs Impact: All hidden dim configs, hyperopt bounds, walk-forward feature counts update from 51 → 55.

---

2. Regime Detection

Problem: detect_regime() returns hardcoded "normal". RegimeConditionalDQN has no real input.

Layer 1: Feature-Based Classifier (real-time, <1μs)

Input: rolling 100-bar window of OHLCV.

Regime	Conditions
Trending	ADX > 25 AND Hurst > 0.55
MeanReverting	Hurst < 0.45 AND vol < median
Volatile	vol > 2σ above mean OR ADX < 15
Normal	Default

Returns existing RegimeType enum.

Layer 2: HMM (offline, periodic refit)

• 3-state HMM (Low-Vol / Trending / Crisis) on daily returns
• Fitted during walk-forward setup, refit weekly
• Provides transition matrix for persistence estimation
• Validates feature-based classifier accuracy

Location: crates/ml/src/regime_detection/ (replace stub). feature_classifier.rs + hmm.rs. Integration: Wire into RegimeConditionalDQN head dispatch.

---

3. Slippage Modeling

Problem: Fixed 15/5/10 bps costs don't reflect volume-dependent market impact.

SlippageModel trait

Two implementations:

LinearImpactModel (new default):

slippage = spread/2 + sqrt_impact * sqrt(volume / ADV)

• spread: estimated from OHLCV as 0.5 * (high - low) / close * normalization
• sqrt_impact: per-instrument (ES ≈ 0.1, 6E ≈ 0.3 bps/√lot)
• ADV: 20-day rolling average daily volume
• Temporary impact decays: temp_impact * exp(-t / decay_half_life)

FixedCostModel (backward compat):

• Wraps existing 15/5/10 bps constants

Location: crates/ml/src/risk/slippage.rs Integration: Replace fixed cost_weight in dqn/reward.rs with slippage_model.estimate_cost(action, state). Same for PPO and evaluation engine.

---

4. Curriculum Learning

Problem: 45-action space + all regimes from step 0 slows convergence.

3-Phase Schedule

Phase	Exposure Actions	Regimes	Gate
1: Basic	Flat, Long50, Short50 (27 actions)	Normal only	Sharpe > 0.5 for 3 folds
2: Full Position	All 5 exposures (45 actions)	Normal + Trending	Sharpe > 0.3 for 2 folds
3: All Regimes	All 45	All	Final phase

Mechanism:

• Action masking expanded per phase (reuse get_valid_action_mask())
• Regime filtering via Section 2 detection
• Phase tracked in checkpoint metadata
• Manual override: --curriculum-phase 3

Location: crates/ml/src/trainers/curriculum.rs

---

5. Multi-Timeframe Feature Fusion (Default On)

Problem: Single 1-minute resolution. No cross-timeframe context.

Architecture

1m bars ──→ [Encoder] ──→ emb_1m (64)  ─┐
  ├─ resample 5m  ──→ [Encoder] ──→ emb_5m (64)  ─┤
  ├─ resample 15m ──→ [Encoder] ──→ emb_15m (64) ─┼─→ [Concat 256] ─→ [Linear 128] ─→ macro_state
  └─ resample 1h  ──→ [Encoder] ──→ emb_1h (64)  ─┘

• Resample from 1m OHLCV on-the-fly (no extra data download)
• Per-timeframe encoder: shared-weight LSTM
• Fusion: concatenate → linear projection → 128-dim macro context
• Combined state: position features (55) + macro context (128) = 183 input dims
• Default on — no feature gate

Location: crates/ml/src/features/multi_timeframe.rs

---

6. Online Learning (Per-Trade)

Problem: Models freeze after walk-forward training. No adaptation to regime shifts.

Core Mechanism

• Rolling experience buffer: 10,000 transitions (configurable)
• Mini-update every 100 trades:
  • DQN: sample mini-batch, TD loss, weight update
  • PPO: N-step trajectory, advantages, single epoch
• EWC regularization prevents catastrophic forgetting:
  • Fisher Information Matrix (diagonal approx) computed after initial training
  • online_loss = task_loss + λ * Σ F_i * (θ_i - θ*_i)²
  • λ default: 1000.0, Fisher recomputed every 10,000 steps

Safety Rails

Rail	Config	Default
Gradient clipping	max_grad_norm	1.0
Learning rate	online_lr_multiplier	0.1× base LR
Checkpoint rollback	Sharpe degrades > 20% over 500 trades	Auto-rollback
Kill switch	Rolling Sharpe < -1.0	Freeze + alert

Location: crates/ml/src/trainers/online_learning.rs Integration: Called from trading_service after trade execution.

---

7. A/B Testing Framework

Problem: No safe mechanism to deploy and compare new models.

Architecture

Two slots: Champion (production) and Challenger (candidate).

Traffic allocation via Thompson Sampling:

• Each model: Beta(α, β) distribution on win rate
• Sample from both; higher sample gets the trade
• Initial: Champion Beta(10, 1), Challenger Beta(1, 1)

Shadow mode: Challenger runs inference but never executes (100% Champion).

Metrics per model

• Rolling Sharpe (500 trades)
• Win rate, mean PnL/trade
• Max drawdown
• Action diversity (entropy)

Promotion criteria (all required)

1. Challenger Sharpe > Champion Sharpe (Bayesian P > 0.95)
2. Challenger max DD ≤ 1.2 × Champion max DD
3. Minimum 500 evaluated trades
4. Manual approval via existing ApproveModel gRPC

Location: crates/ml/src/deployment/ab_testing.rs Integration: trading_service ensemble infrastructure + ApproveModel/RejectModel handlers.

---

Summary

#	Feature	New Files	Impact
1	Position-aware state	features/position_features.rs	State 51→55 dims
2	Regime detection	regime_detection/{feature_classifier,hmm}.rs	Replace stub
3	Slippage model	risk/slippage.rs	Reward computation
4	Curriculum learning	trainers/curriculum.rs	Trainer loops
5	Multi-timeframe fusion	features/multi_timeframe.rs	State 55→183 dims (default on)
6	Online learning	trainers/online_learning.rs	Inference hot path
7	A/B testing	deployment/ab_testing.rs	Trading service