spec: OOS Performance Enhancement — 8 components for closing val/OOS gap
1. CVaR objective (risk-sensitive Bellman target, adaptive α) 2. Adaptive DSR (training Sharpe EMA drives w_dsr weight) 3. Ensemble heads (3 heads, epistemic uncertainty for exploration) 4. Mamba2 temporal scan (K=8 bar context, O(K) selective scan) 5. Multi-horizon prediction (1/5/20 bar value heads, regime-blended) 6. Adaptive atom positions (learned C51 spacing via softmax — NOVEL) 7. Regime-conditioned branch gating (ADX/CUSUM → branch importance — NOVEL) Designed from live H100 analysis: val_Sharpe=45 but training_Sharpe=-1..+1. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
@@ -0,0 +1,245 @@
|
||||
# OOS Performance Enhancement — Design Spec
|
||||
|
||||
**Goal:** Close the val_Sharpe/OOS gap (val=45 but training Sharpe oscillates -1 to +1) by adding risk-sensitive objectives, temporal context, ensemble diversity, multi-horizon prediction, adaptive reward shaping, and two novel techniques (adaptive atom positions + regime-conditioned branch gating).
|
||||
|
||||
**Problem:** The model achieves val_Sharpe=45+ on a fixed validation window but training Sharpe (OOS proxy) oscillates -1.2 to +0.9. The model finds good strategies but can't consistently profit on fresh data. Root causes: (1) optimizes E[Q] not worst-case, (2) no temporal context — sees one bar at a time, (3) single Q-estimate — can't distinguish epistemic from aleatoric uncertainty, (4) fixed reward shaping — can't adapt to market regime, (5) fixed atom spacing — wastes resolution on unlikely returns.
|
||||
|
||||
**Architecture:** 8 independent components composing through the existing pipeline.
|
||||
|
||||
---
|
||||
|
||||
## Component 1: CVaR Objective for Bellman Target
|
||||
|
||||
### Problem
|
||||
Expected SARSA target optimizes E[Q] — tolerates high-variance strategies if the mean is good. Training Sharpe oscillates because the model takes risky bets.
|
||||
|
||||
### Solution
|
||||
Replace E[Q] with CVaR_α[Q] in the Bellman target action sampling:
|
||||
|
||||
```
|
||||
CVaR_α[Q_a] = mean of atoms in the bottom α-percentile of action a's distribution
|
||||
target_action ~ softmax(CVaR_α[Q_target] / tau)
|
||||
```
|
||||
|
||||
α is adaptive: starts at 0.5 (balanced), decreases toward 0.1 (risk-averse) as iqn_readiness increases.
|
||||
|
||||
### Implementation
|
||||
- Modify `c51_loss_kernel.cu`: the Expected SARSA target sampling computes softmax weights from Q-values. Change to compute CVaR from the CDF (same infrastructure as quantile_q_select) and use CVaR for the softmax weights.
|
||||
- `α = 0.5 - 0.4 * iqn_readiness` — adaptive schedule.
|
||||
- Zero extra parameters.
|
||||
|
||||
---
|
||||
|
||||
## Component 2: Adaptive DSR Reward Shaping
|
||||
|
||||
### Problem
|
||||
`w_dsr = 1.0` (fixed). Can't adapt to market regime — trending markets need low DSR, ranging needs high.
|
||||
|
||||
### Solution
|
||||
```
|
||||
sharpe_ema = EMA(training_sharpe, adaptive_alpha)
|
||||
w_dsr_adaptive = w_dsr_base * clamp(1.0 - sharpe_ema, 0.5, 3.0)
|
||||
```
|
||||
|
||||
Negative Sharpe → high DSR (force risk reduction). Positive Sharpe → low DSR (exploit).
|
||||
|
||||
### Implementation
|
||||
- `training_sharpe_ema: f32` field, updated at epoch end from experience collection Sharpe.
|
||||
- `w_dsr_adaptive` written to pinned device-mapped scalar, read by env_step kernel.
|
||||
- Zero kernel changes — replaces the fixed `w_dsr` config value.
|
||||
|
||||
---
|
||||
|
||||
## Component 3: Ensemble Heads (3 heads)
|
||||
|
||||
### Problem
|
||||
`ensemble_heads=1`. Single Q-estimate. Uncertainty from C51 is aleatoric (market randomness), not epistemic (model ignorance). Can't distinguish "I don't know" from "the market is random."
|
||||
|
||||
### Solution
|
||||
Increase to `ensemble_heads=3`. Each shares the trunk but has independent branch weights.
|
||||
|
||||
```
|
||||
Q_mean = mean(Q_1, Q_2, Q_3)
|
||||
epistemic_var = var(Q_1, Q_2, Q_3) // model disagreement
|
||||
exploration_bonus = sqrt(epistemic_var)
|
||||
Q_select = Q_mean + exploration_bonus // OFU on epistemic uncertainty
|
||||
```
|
||||
|
||||
### Implementation
|
||||
- Config change: `ensemble_heads: 3`. Infrastructure already exists (`ensemble_kernels.cu`).
|
||||
- ~2× branch parameter increase (~174K extra params).
|
||||
- Ensemble variance replaces quantile Q-select for exploration (more principled).
|
||||
|
||||
---
|
||||
|
||||
## Component 4: Mamba2 Selective Scan for Temporal Context
|
||||
|
||||
### Problem
|
||||
Model processes one bar at a time. No memory of recent market behavior within an episode.
|
||||
|
||||
### Solution
|
||||
Rolling buffer of last K=8 trunk activations. Mamba2 selective scan compresses into temporal context:
|
||||
|
||||
```
|
||||
A_t = sigmoid(W_A @ h_t) // forget gate (input-dependent)
|
||||
B_t = W_B @ h_t // input projection
|
||||
x_t = A_t * x_{t-1} + B_t * h_t // recurrent scan (O(K) time)
|
||||
temporal_context = (W_C @ h_t) * x_t
|
||||
|
||||
h_s2_enriched = h_s2 + temporal_context // residual
|
||||
```
|
||||
|
||||
### Implementation
|
||||
- `h_history[B, K, SH2]` rolling buffer (~128MB for K=8, acceptable on H100).
|
||||
- State_dim=16. Parameters: 3 × 256 × 16 = 12,288.
|
||||
- New CUDA kernel `mamba2_temporal_scan`. Runs outside CUDA graph.
|
||||
- From `crates/ml-supervised/src/mamba/scan_algorithms.rs`.
|
||||
|
||||
---
|
||||
|
||||
## Component 5: Multi-Horizon Prediction
|
||||
|
||||
### Problem
|
||||
Single-step return prediction. Urgency branch has no temporal signal — can't distinguish scalping from swing opportunities.
|
||||
|
||||
### Solution
|
||||
3 value heads (shared trunk, shared branches): 1-bar, 5-bar, 20-bar horizons.
|
||||
|
||||
```
|
||||
Q_1bar = compute_expected_q(v_logits_1, adv_logits)
|
||||
Q_5bar = compute_expected_q(v_logits_5, adv_logits)
|
||||
Q_20bar = compute_expected_q(v_logits_20, adv_logits)
|
||||
|
||||
// Regime-weighted blend:
|
||||
trend_weight = sigmoid((ADX - 25) / 5)
|
||||
Q_blended = (1-trend_weight) * Q_1bar + trend_weight * Q_20bar
|
||||
```
|
||||
|
||||
### Implementation
|
||||
- 2 extra value heads: W_v1_5[VH, SH2], b_v1_5, W_v2_5[NA, VH], b_v2_5 (× 2 horizons).
|
||||
- ~144K extra params.
|
||||
- 3 passes of n-step kernel with step counts {1, 5, 20}.
|
||||
- Urgency branch learns: Q_1bar >> Q_20bar → act now. Q_20bar >> Q_1bar → wait.
|
||||
|
||||
---
|
||||
|
||||
## Component 6: Adaptive Atom Positions (Novel)
|
||||
|
||||
### Problem
|
||||
C51 atoms at fixed uniform positions z[j] = v_min + j * delta_z. 90% of atoms may cover returns that never happen, while the critical region near zero has too few atoms.
|
||||
|
||||
### Solution
|
||||
Learnable atom positions via softmax-normalized spacing:
|
||||
|
||||
```
|
||||
spacing_raw[51] // learnable parameter vector
|
||||
spacing = softmax(spacing_raw) // sum-to-1 normalized
|
||||
positions[j] = v_min + (v_max - v_min) * cumsum(spacing)[j]
|
||||
```
|
||||
|
||||
The model concentrates atoms where returns have mass. Near-zero returns get higher resolution.
|
||||
|
||||
### Implementation
|
||||
- 51 floats per branch × 4 branches = 204 extra params (in params_buf).
|
||||
- New CUDA kernel `adaptive_atom_positions`: computes positions from spacing_raw.
|
||||
- Replaces fixed `v_min + j * delta_z` in compute_expected_q, c51_loss, c51_grad.
|
||||
- Gradient flows through atom positions via the C51 loss.
|
||||
|
||||
---
|
||||
|
||||
## Component 7: Regime-Conditioned Branch Gating (Novel)
|
||||
|
||||
### Problem
|
||||
All 4 branches contribute equally to action selection. But branch importance is regime-dependent: direction matters in trends, magnitude matters in ranges.
|
||||
|
||||
### Solution
|
||||
Learned regime gate produces per-branch importance weights:
|
||||
|
||||
```
|
||||
regime_input = [ADX, CUSUM, Q_gap, atom_utilization] // 4 features
|
||||
branch_importance[4] = softmax(W_regime @ regime_input + b_regime)
|
||||
|
||||
// Weight Q-values by branch importance:
|
||||
Q_select[a] = Σ_d branch_importance[d] * Q_d[a_d]
|
||||
```
|
||||
|
||||
### Implementation
|
||||
- `W_regime[4, 4] + b_regime[4]` = 20 params.
|
||||
- New CUDA kernel `regime_branch_gate`: computes softmax importance, scales Q-values.
|
||||
- Runs after Q-mean centering, before Q-attention.
|
||||
- Features already available in the state vector (ADX=40, CUSUM=41) and Q-stats (Q-gap, utilization).
|
||||
|
||||
---
|
||||
|
||||
## Full Pipeline (after all 8 components)
|
||||
|
||||
```
|
||||
── Trunk ──
|
||||
h_s2 = trunk_forward(states)
|
||||
h_s2 = attention(h_s2)
|
||||
h_s2_enriched = h_s2 + mamba2_temporal_scan(h_history) [Component 4]
|
||||
|
||||
── Branches (3 ensemble heads × 4 branches) ──
|
||||
For each head e, branch d:
|
||||
h_bd_e = KAN_gate(VSN(h_s2_enriched)) @ W_bdf_e_d
|
||||
|
||||
── Value Heads (3 horizons) ──
|
||||
v_logits_1/5/20 = value_head_1/5/20(h_s2_enriched) [Component 5]
|
||||
|
||||
── Q-Values ──
|
||||
Q_raw = compute_expected_q(v_logits_blended, adv_logits) [Component 5]
|
||||
Q_raw = adaptive_atom_positions(Q_raw) [Component 6]
|
||||
Q_centered = Q_raw - mean(Q_raw) [existing]
|
||||
Q_regime = regime_branch_gate(Q_centered, ADX, CUSUM) [Component 7]
|
||||
Q_coord = cross_branch_attention(Q_regime) [existing]
|
||||
Q_graph = graph_message_pass(Q_coord) [existing]
|
||||
|
||||
── Action Selection ──
|
||||
Q_ensemble_mean = mean(Q across 3 heads) [Component 3]
|
||||
exploration = sqrt(var(Q across 3 heads)) [Component 3]
|
||||
Q_select = Q_ensemble_mean + exploration
|
||||
|
||||
── Training ──
|
||||
CVaR target: softmax(CVaR_α[Q_target] / tau) [Component 1]
|
||||
Adaptive DSR: w_dsr *= clamp(1 - sharpe_ema, 0.5, 3.0) [Component 2]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Implementation Order
|
||||
|
||||
1. Component 2 (Adaptive DSR) — zero kernel changes, CPU-side, immediate OOS impact
|
||||
2. Component 1 (CVaR objective) — modify existing c51_loss kernel, zero params
|
||||
3. Component 7 (Regime branch gating) — 20 params, one kernel
|
||||
4. Component 6 (Adaptive atom positions) — 204 params, modifies existing kernels
|
||||
5. Component 3 (Ensemble heads) — config change, existing infrastructure
|
||||
6. Component 4 (Mamba2 temporal scan) — new trunk layer, 12K params
|
||||
7. Component 5 (Multi-horizon prediction) — 2 new value heads, 144K params
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria
|
||||
|
||||
| Metric | Current (train-w2z55 ep 21) | Target | Source |
|
||||
|--------|---------------------------|--------|--------|
|
||||
| Training Sharpe | -1.2 to +0.9 oscillating | >0.5 sustained | Components 1, 2 |
|
||||
| val_Sharpe | 45.81 (climbing) | >50 sustained | Components 4, 5, 6 |
|
||||
| val/OOS gap | val=45, training=0 | val≈training | Components 1, 2, 3 |
|
||||
| Exploration quality | Aleatoric-based (C51 variance) | Epistemic-based (ensemble disagreement) | Component 3 |
|
||||
| Temporal awareness | None (single bar) | 8-bar Mamba2 context | Component 4 |
|
||||
| Branch importance | Uniform (25% each) | Regime-adaptive | Component 7 |
|
||||
| Atom resolution | Uniform (wasted on unlikely returns) | Concentrated where returns have mass | Component 6 |
|
||||
|
||||
---
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|-----------|
|
||||
| CVaR too conservative → stops trading | α starts at 0.5 (balanced), not 0.1 (extreme) |
|
||||
| Adaptive DSR oscillates | EMA smoothing, clamped to [0.5, 3.0] |
|
||||
| 3 ensemble heads → 2× params → overfitting | Shared trunk, only branches triplicated |
|
||||
| Mamba2 history buffer 128MB | K=4 for 64MB if needed; H100 has 80GB |
|
||||
| Multi-horizon 3× C51 loss compute | Value heads are small (VH=256); branch GEMMs shared |
|
||||
| Adaptive atoms destabilize C51 loss | Softmax spacing → always valid positions; small learning rate for spacing params |
|
||||
| Regime gate collapses to uniform | Softmax with temperature; initialized to uniform |
|
||||
Reference in New Issue
Block a user