75885bccaf
- DSR uses pre-update A/B values (Moody & Saffell correct) - peak_equity/prev_equity init to initial_capital (not zero) - Division-by-zero guards on drawdown and return calculation - PORTFOLIO_STRIDE=12 constant replaces hardcoded 3 (7 locations listed) - Remove use_dsr flag (w_dsr>0 implies enabled) - RewardSection struct needed in training_profile.rs - All TOML profiles use same [reward] section Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
341 lines
17 KiB
Markdown
341 lines
17 KiB
Markdown
# GPU Composite Reward Function — Design Spec
|
||
|
||
## Problem
|
||
|
||
The current GPU reward is raw per-step PnL minus transaction costs:
|
||
```
|
||
reward = position * (next_close - close) - |delta| * close * tx_cost_multiplier + hold_bonus
|
||
```
|
||
|
||
This produces:
|
||
- 3% win rate, 100% max drawdown, Sharpe ~0.06
|
||
- No risk-adjusted signal (model optimizes raw PnL, not Sharpe)
|
||
- No drawdown awareness (model never learns to cut losses)
|
||
- No participation incentive (model can achieve "good" Sharpe by trading once)
|
||
- DSR implemented in CPU `reward.rs` but NOT wired to GPU kernel — dead code
|
||
- Backtest evaluator only runs post-training (hyperopt objective), never during training
|
||
|
||
## Solution
|
||
|
||
Replace the scalar PnL reward with a **GPU-native composite reward** computed entirely in the `experience_env_step` CUDA kernel. Zero CPU involvement. All state maintained in the existing per-episode `portfolio_states` GPU buffer.
|
||
|
||
### Reward Formula
|
||
|
||
```
|
||
// Regime-adaptive weights (from market features already in GPU memory)
|
||
adx = features[bar_idx * MARKET_DIM + 40]
|
||
cusum = features[bar_idx * MARKET_DIM + 41]
|
||
trend_scale = clamp(adx / 25.0, 0.5, 2.0) // ADX>25 = trending
|
||
vol_scale = clamp(cusum / 0.5, 0.5, 2.0) // high CUSUM = volatile
|
||
|
||
eff_w_pnl = w_pnl * trend_scale // boost PnL weight in trends
|
||
eff_w_dsr = w_dsr * (2.0 - trend_scale) // boost DSR weight in mean-reversion
|
||
eff_w_dd = w_dd * vol_scale // boost DD protection in volatile regimes
|
||
|
||
// Asymmetric profit/loss scaling (prospect theory)
|
||
asym_pnl = normalized_pnl >= 0
|
||
? normalized_pnl
|
||
: normalized_pnl * loss_aversion // losses hurt more (default 1.5x)
|
||
|
||
// Position-time decay (stale position rent)
|
||
hold_time_penalty = position != 0
|
||
? hold_time * time_decay_rate // linear rent per step held
|
||
: 0.0
|
||
|
||
// Composite reward
|
||
reward = eff_w_dsr * dsr_t // Sharpe contribution
|
||
+ eff_w_pnl * asym_pnl // Asymmetric normalized PnL
|
||
- eff_w_dd * drawdown_penalty_t // Drawdown penalty
|
||
- w_idle * idle_penalty_t // Idle penalty
|
||
- hold_time_penalty // Position staleness rent
|
||
- tx_cost // Transaction friction
|
||
```
|
||
|
||
### Component 1: Differential Sharpe Ratio (DSR)
|
||
|
||
Per-step incremental Sharpe contribution (Moody & Saffell 2001):
|
||
|
||
```
|
||
// Return (guard division by zero on prev_equity)
|
||
equity_denom = max(prev_equity, 1.0)
|
||
R_t = pnl_t / equity_denom
|
||
|
||
// Compute deltas BEFORE updating EMA (pre-update values for numerator+denominator)
|
||
delta_A = R_t - A_{t-1}
|
||
delta_B = R_t^2 - B_{t-1}
|
||
variance = B_{t-1} - A_{t-1}^2
|
||
denominator = max(|variance|, epsilon)^(3/2)
|
||
|
||
// DSR from pre-update state (Moody & Saffell correct formulation)
|
||
dsr_t = (B_{t-1} * delta_A - 0.5 * A_{t-1} * delta_B) / denominator
|
||
|
||
// NOW update EMA
|
||
A_t = A_{t-1} + eta * delta_A
|
||
B_t = B_{t-1} + eta * delta_B
|
||
```
|
||
|
||
**State**: 2 floats per episode (A_t, B_t)
|
||
**Cost**: ~10 FLOPs per step
|
||
**Why**: Dense reward that directly optimizes annualized Sharpe ratio. Each step gets immediate feedback on whether this action improved or degraded the portfolio's risk-adjusted return.
|
||
**Edge cases**: `prev_equity <= 0` → clamp denominator to 1.0. `variance ≈ 0` (no return history) → `dsr_t ≈ 0` (epsilon guard). Degenerate case returns 0.0 (not the CPU's `(r*100).clamp(-5,5)` — GPU is the canonical implementation).
|
||
|
||
### Component 2: Normalized PnL
|
||
|
||
```
|
||
raw_pnl = position * (next_close - close)
|
||
pnl_ema = pnl_ema + eta * (raw_pnl - pnl_ema) // running mean
|
||
pnl_var = pnl_var + eta * ((raw_pnl - pnl_ema)^2 - pnl_var) // running variance
|
||
pnl_std = sqrt(pnl_var + 1e-8)
|
||
normalized_pnl = (raw_pnl - pnl_ema) / pnl_std // z-scored PnL
|
||
```
|
||
|
||
**State**: 2 floats per episode (pnl_ema, pnl_var)
|
||
**Why**: Raw PnL varies wildly across instruments and timeframes. Z-scoring makes the reward scale-invariant — same gradient magnitude whether trading ES (point value $50) or NQ (point value $20). Works equally well on tick data and 1-minute bars.
|
||
|
||
### Component 3: Drawdown Penalty
|
||
|
||
```
|
||
equity = cash + position * close
|
||
peak_equity = max(peak_equity, equity)
|
||
drawdown = peak_equity > 0.0
|
||
? (peak_equity - equity) / peak_equity // fractional drawdown [0, 1]
|
||
: 0.0 // guard: no equity = no drawdown
|
||
drawdown_penalty_t = max(0, drawdown - dd_threshold) // only penalize beyond threshold
|
||
```
|
||
|
||
**State**: 1 float per episode (peak_equity)
|
||
**Init**: `peak_equity = initial_capital` (NOT zero — zero causes div-by-zero on first step)
|
||
**Parameter**: `dd_threshold` (default 0.02 = 2% drawdown tolerance before penalty)
|
||
**Why**: The model must learn to cut losing positions. Without drawdown awareness, it holds losers hoping for recovery — exactly the behavior that produces 100% max drawdown.
|
||
|
||
### Component 4: Idle Penalty
|
||
|
||
```
|
||
is_flat = |position| < 0.001
|
||
if is_flat:
|
||
flat_counter += 1
|
||
else:
|
||
flat_counter = 0
|
||
|
||
idle_penalty_t = min(flat_counter * idle_scale, max_idle_penalty)
|
||
```
|
||
|
||
**State**: 1 float per episode (flat_counter, stored as float for GPU simplicity)
|
||
**Parameters**: `idle_scale` (default 0.001), `max_idle_penalty` (default 0.05)
|
||
**Why**: Replaces the old `hold_bonus` (which REWARDED flatness). Instead, penalizes prolonged inactivity. The model must demonstrate edge by participating in the market. Short flat periods (rebalancing) are fine — only sustained idleness is penalized.
|
||
|
||
### Component 5: Regime-Adaptive Weight Scaling
|
||
|
||
```
|
||
adx = features[bar_idx * MARKET_DIM + 40] // already computed per-step
|
||
cusum = features[bar_idx * MARKET_DIM + 41] // already in feature vector
|
||
trend_scale = clamp(adx / 25.0, 0.5, 2.0)
|
||
vol_scale = clamp(cusum / 0.5, 0.5, 2.0)
|
||
|
||
eff_w_pnl = w_pnl * trend_scale // trending: ride momentum
|
||
eff_w_dsr = w_dsr * (2.0 - trend_scale) // mean-reverting: optimize Sharpe
|
||
eff_w_dd = w_dd * vol_scale // volatile: protect capital
|
||
```
|
||
|
||
**State**: 0 floats (reads from existing feature buffer)
|
||
**Cost**: ~5 FLOPs per step
|
||
**Why**: Markets alternate between trending, mean-reverting, and volatile regimes. A fixed reward blend optimizes for the average regime — which is no regime. Regime-adaptive scaling teaches the model different strategies through the reward signal itself. In a strong trend (ADX>50), PnL weight doubles and DSR halves — the model learns to hold winners. In volatile chop (high CUSUM), drawdown protection doubles — the model learns to cut quickly. No extra hyperopt dimensions needed; the ADX/CUSUM features already encode regime state.
|
||
|
||
### Component 6: Asymmetric Profit/Loss Scaling
|
||
|
||
```
|
||
asym_pnl = normalized_pnl >= 0
|
||
? normalized_pnl
|
||
: normalized_pnl * loss_aversion // default 1.5
|
||
```
|
||
|
||
**State**: 0 floats
|
||
**Parameter**: `loss_aversion` (default 1.5, hyperopt range [1.0, 3.0])
|
||
**Cost**: 1 comparison + 1 multiply per step
|
||
**Why**: Prospect theory for RL. Symmetric rewards produce symmetric behavior — the model treats a $100 loss and a $100 gain as equally important. In production trading, they're not. A loss reduces capital (compounding drag), damages confidence metrics (Sharpe, drawdown), and may trigger risk limits. `loss_aversion=1.5` means a $100 loss produces 1.5x the negative reward of a $100 gain's positive reward. This teaches the model to: (1) cut losers early, (2) size positions conservatively when uncertain, (3) be aggressive only with high-conviction entries. The asymmetry naturally produces higher win rates and lower drawdowns without explicitly penalizing either.
|
||
|
||
### Component 7: Position-Time Decay
|
||
|
||
```
|
||
if |position| > 0.001:
|
||
hold_time += 1
|
||
hold_time_penalty = hold_time * time_decay_rate // linear rent
|
||
else:
|
||
hold_time = 0
|
||
hold_time_penalty = 0
|
||
```
|
||
|
||
**State**: 1 float per episode (hold_time)
|
||
**Parameters**: `time_decay_rate` (default 0.0005, hyperopt range [0.0001, 0.005])
|
||
**Cost**: 1 comparison + 1 multiply per step
|
||
**Why**: In HFT, a position that was entered 500 bars ago is stale — the alpha that justified entry has decayed. The model should either be actively managing (adjusting size, taking profit, stopping out) or flat. Position-time decay applies a linearly increasing "rent" for holding. After 100 steps at `time_decay_rate=0.0005`, the penalty is 0.05 — comparable to one transaction cost. This forces the model to: (1) enter positions with clear exit criteria, (2) take partial profits as the trade ages, (3) cut positions that aren't working within a bounded time horizon. Unlike a fixed hold penalty, the decay scales with duration — short holds (1-10 steps) pay almost nothing, but overstaying (100+ steps) becomes expensive. The time horizon is tunable via hyperopt to match the instrument's characteristic holding period.
|
||
|
||
### Component 8: Transaction Cost (unchanged)
|
||
|
||
```
|
||
tx_cost = |delta_position| * close * tx_cost_multiplier
|
||
```
|
||
|
||
No change from current implementation. Already correct.
|
||
|
||
## GPU State Layout
|
||
|
||
Introduce `PORTFOLIO_STRIDE = 12` constant (replaces all hardcoded `3`).
|
||
|
||
Extend the existing `portfolio_states` buffer from 3 floats to 12 floats per episode:
|
||
|
||
```
|
||
Index Field Init Value Description
|
||
[0] position 0.0 current position size
|
||
[1] cash initial_capital available cash
|
||
[2] portfolio_val initial_capital mark-to-market equity
|
||
[3] dsr_A 0.0 EMA of returns
|
||
[4] dsr_B 0.0 EMA of squared returns
|
||
[5] pnl_ema 0.0 running mean PnL
|
||
[6] pnl_var 0.0 running variance PnL
|
||
[7] peak_equity initial_capital high-water mark (NOT zero — causes div/0)
|
||
[8] flat_counter 0.0 consecutive flat steps (float for GPU)
|
||
[9] prev_equity initial_capital equity at previous step (NOT zero — causes div/0)
|
||
[10] hold_time 0.0 consecutive steps with position
|
||
[11] realized_pnl 0.0 cumulative realized PnL
|
||
```
|
||
|
||
**Memory impact**: 9 extra floats × 2048 episodes × 4 bytes = 73 KB. Negligible.
|
||
|
||
### Code locations requiring stride update (3 → PORTFOLIO_STRIDE)
|
||
|
||
All 7 hardcoded stride-3 references that MUST change:
|
||
|
||
| File | Line | Current | Change to |
|
||
|------|------|---------|-----------|
|
||
| `experience_kernels.cu` | 169 | `portfolio_states + i * 3` | `i * PORTFOLIO_STRIDE` |
|
||
| `experience_kernels.cu` | 420 | `portfolio_states + i * 3` | `i * PORTFOLIO_STRIDE` |
|
||
| `gpu_experience_collector.rs` | 575 | `alloc_episodes * 3` | `alloc_episodes * PORTFOLIO_STRIDE` |
|
||
| `gpu_experience_collector.rs` | 579 | `vec![0.0; alloc_episodes * 3]` | `alloc_episodes * PORTFOLIO_STRIDE` |
|
||
| `gpu_experience_collector.rs` | 581 | `i * 3` | `i * PORTFOLIO_STRIDE` |
|
||
| `gpu_experience_collector.rs` | 1166 | `alloc_episodes * 3` | `alloc_episodes * PORTFOLIO_STRIDE` |
|
||
| `gpu_experience_collector.rs` | 1168 | `i * 3` | `i * PORTFOLIO_STRIDE` |
|
||
|
||
The existing `PORTFOLIO_STATE_SIZE = 8` at `gpu_experience_collector.rs:51` is for the PPO/legacy path (NOT used by DQN experience collection). Do not change it — introduce a new `const PORTFOLIO_STRIDE: usize = 12` for the DQN path.
|
||
|
||
Note: `experience_kernels.cu` contains TWO different portfolio concepts in the same file — `experience_env_step` uses stride-3 (DQN, changing to 12), `portfolio_sim_kernel` uses stride-8 (legacy, unchanged).
|
||
|
||
## Hyperopt Search Space
|
||
|
||
Add 7 new dimensions to the PSO/TPE search space (bringing total from 31D to 38D):
|
||
|
||
| Parameter | Range | Scale | Phase Fast Fixed | Description |
|
||
|-----------|-------|-------|-----------------|-------------|
|
||
| `w_dsr` | [0.1, 2.0] | linear | 1.0 | DSR (Sharpe) weight |
|
||
| `w_pnl` | [0.0, 1.0] | linear | 0.3 | Normalized PnL weight |
|
||
| `w_dd` | [0.0, 5.0] | linear | 1.0 | Drawdown penalty weight |
|
||
| `w_idle` | [0.0, 0.1] | linear | 0.01 | Idle penalty weight |
|
||
| `dd_threshold` | [0.01, 0.10] | linear | 0.02 | Drawdown tolerance before penalty |
|
||
| `loss_aversion` | [1.0, 3.0] | linear | 1.5 | Asymmetric loss scaling factor |
|
||
| `time_decay_rate` | [0.0001, 0.005] | log | 0.0005 | Position staleness rent per step |
|
||
|
||
Phase Fast fixes all reward weights to sensible defaults — optimizer searches only learning dynamics. Phase Full can search reward weights to find the optimal blend for the instrument.
|
||
|
||
`eta` (DSR/normalization decay) already exists as `dsr_eta` in the search space.
|
||
|
||
Regime scaling (ADX/CUSUM) adds zero search dimensions — the scaling factors are derived from market features, not tuned.
|
||
|
||
### Phase 3: Reward Tuning (optional future)
|
||
|
||
A potential third hyperopt phase dedicated to reward weights only. Fix architecture (Phase 2 best) AND dynamics (Phase 1 best), search only the 7 reward dimensions. This would be very fast (~10s/trial) since only the experience collection changes, not the network or training dynamics.
|
||
|
||
## Kernel Changes
|
||
|
||
### Modified kernel: `experience_env_step`
|
||
|
||
File: `crates/ml/src/cuda_pipeline/experience_kernels.cu`
|
||
|
||
New scalar parameters (7 reward weights + 2 feature buffer args):
|
||
```cuda
|
||
float w_dsr, float w_pnl, float w_dd, float w_idle,
|
||
float dd_threshold, float loss_aversion, float time_decay_rate,
|
||
const float* __restrict__ features, // market features for regime detection
|
||
int market_dim // MARKET_DIM (42)
|
||
```
|
||
|
||
Kernel body: replace the 3-line reward computation (lines 454-465) with the full composite formula including regime scaling, asymmetric PnL, and position-time decay. All state reads/writes through the existing `portfolio_states` pointer with stride 12 instead of 3.
|
||
|
||
The `features` pointer is already passed to `experience_state_gather` — extend to `experience_env_step` for ADX/CUSUM access.
|
||
|
||
### Modified struct: `ExperienceCollectorConfig`
|
||
|
||
File: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs`
|
||
|
||
Add 7 fields matching the kernel parameters. Default values from TOML config.
|
||
|
||
### Modified allocation: `portfolio_states`
|
||
|
||
Current: `alloc_episodes * 3` floats
|
||
New: `alloc_episodes * 12` floats
|
||
|
||
All existing code that reads `portfolio_states[0..3]` must update to stride 12. New state at indices 3-11 initialized to zero at episode start (memset in `reset_episode` path).
|
||
|
||
## TOML Configuration
|
||
|
||
Add to ALL training profiles (`dqn-production.toml`, `dqn-smoketest.toml`, `dqn-hyperopt.toml`):
|
||
```toml
|
||
[reward]
|
||
w_dsr = 1.0
|
||
w_pnl = 0.3
|
||
w_dd = 1.0
|
||
w_idle = 0.01
|
||
dd_threshold = 0.02
|
||
loss_aversion = 1.5
|
||
time_decay_rate = 0.0005
|
||
```
|
||
|
||
Requires new `RewardSection` struct in `training_profile.rs` with `Option<f64>` fields, and `apply_to()` mapping to `DQNHyperparameters` reward fields.
|
||
|
||
Add to `config/training/dqn-hyperopt.toml` `[search_space]`:
|
||
```toml
|
||
w_dsr = [0.1, 2.0]
|
||
w_pnl = [0.0, 1.0]
|
||
w_dd = [0.0, 5.0]
|
||
w_idle = [0.0, 0.1]
|
||
dd_threshold = [0.01, 0.10]
|
||
```
|
||
|
||
## What Gets Removed
|
||
|
||
1. **`hold_reward` / `hold_penalty` / `hold_penalty_weight`** — replaced by `idle_penalty` (inverted semantics). The kernel parameter `hold_rw` at `gpu_experience_collector.rs:1002-1020` is removed from the launch args.
|
||
2. **CPU `RewardNormalizer`** in `reward.rs` — normalization now in GPU kernel (pnl_ema/pnl_var)
|
||
3. **CPU DSR code** in `reward.rs` — DSR now in GPU kernel (dsr_A/dsr_B)
|
||
4. **`use_dsr` bool flag** in `ExperienceCollectorConfig` — redundant when `w_dsr > 0` implies DSR enabled. Remove dual-control confusion.
|
||
5. **`hold_penalty_weight`** from hyperopt search space — replaced by `w_idle`
|
||
|
||
## What Stays
|
||
|
||
1. **Transaction cost model** — already correct, no change
|
||
2. **Backtest evaluator** — still used for hyperopt objective and walk-forward validation
|
||
3. **`compute_validation_loss`** — still computes greedy Sharpe on validation data per epoch
|
||
4. **`compute_epoch_financials`** — still tracks training-set Sharpe for monitoring/early stopping
|
||
5. **C51 distributional loss** — still the RL loss function (unchanged). The reward function shapes WHAT the Q-values learn, not HOW they learn it.
|
||
|
||
## Testing Strategy
|
||
|
||
1. **Unit test**: synthetic price series (linear trend, mean-reverting, random walk) → verify each reward component produces expected sign/magnitude
|
||
2. **Smoke test**: 3 epochs with composite reward on ES.FUT → loss decreases, Q-values non-zero, Sharpe > 0
|
||
3. **A/B test**: same hyperopt config with old reward vs new reward, compare trial Sharpe distributions
|
||
4. **Regression**: all 17 existing GPU tests must pass (reward weights default to backward-compatible values)
|
||
|
||
## No Backward Compatibility Mode
|
||
|
||
The composite reward IS the reward function. There is no "legacy PnL-only" mode. All profiles — production, smoketest, hyperopt — use the full 8-component composite with the same default weights.
|
||
|
||
**All TOML profiles** set the same `[reward]` section. The smoketest uses smaller networks and fewer epochs but the same reward formula. Tests validate the real system, not a simplified stub.
|
||
|
||
The old `hold_bonus` path is deleted entirely. No feature flag, no fallback.
|
||
|
||
## Performance Impact
|
||
|
||
- **Per-step**: +25 FLOPs (3 divisions, 2 sqrts, 15 multiplies, 2 comparisons, 2 clamps) on top of ~500 FLOPs for the existing kernel. <5% overhead.
|
||
- **Memory**: +73 KB for 2048 episodes. 0.009% of RTX 3050's 4GB.
|
||
- **Feature read**: 2 extra global memory reads per step (ADX, CUSUM from feature buffer — already in L2 cache from state_gather kernel).
|
||
- **Kernel launch**: zero additional launches. All computation fused into existing `experience_env_step`.
|
||
- **Register pressure**: +12 registers per thread for reward state. Well within sm_86/sm_90 limits (255 registers/thread).
|