foxhunt/docs/superpowers/specs/2026-03-22-gpu-composite-reward-design.md

GPU Composite Reward Function — Design Spec

Problem

The current GPU reward is raw per-step PnL minus transaction costs:

reward = position * (next_close - close) - |delta| * close * tx_cost_multiplier + hold_bonus

This produces:

• 3% win rate, 100% max drawdown, Sharpe ~0.06
• No risk-adjusted signal (model optimizes raw PnL, not Sharpe)
• No drawdown awareness (model never learns to cut losses)
• No participation incentive (model can achieve "good" Sharpe by trading once)
• DSR implemented in CPU reward.rs but NOT wired to GPU kernel — dead code
• Backtest evaluator only runs post-training (hyperopt objective), never during training

Solution

Replace the scalar PnL reward with a GPU-native composite reward computed entirely in the experience_env_step CUDA kernel. Zero CPU involvement. All state maintained in the existing per-episode portfolio_states GPU buffer.

Reward Formula

// Regime-adaptive weights (from market features already in GPU memory)
adx   = features[bar_idx * MARKET_DIM + 40]
cusum = features[bar_idx * MARKET_DIM + 41]
trend_scale = clamp(adx / 25.0, 0.5, 2.0)           // ADX>25 = trending
vol_scale   = clamp(cusum / 0.5, 0.5, 2.0)           // high CUSUM = volatile

eff_w_pnl  = w_pnl * trend_scale                     // boost PnL weight in trends
eff_w_dsr  = w_dsr * (2.0 - trend_scale)              // boost DSR weight in mean-reversion
eff_w_dd   = w_dd  * vol_scale                        // boost DD protection in volatile regimes

// Asymmetric profit/loss scaling (prospect theory)
asym_pnl = normalized_pnl >= 0
         ? normalized_pnl
         : normalized_pnl * loss_aversion              // losses hurt more (default 1.5x)

// Position-time decay (stale position rent)
hold_time_penalty = position != 0
                  ? hold_time * time_decay_rate        // linear rent per step held
                  : 0.0

// Composite reward
reward = eff_w_dsr * dsr_t                             // Sharpe contribution
       + eff_w_pnl * asym_pnl                          // Asymmetric normalized PnL
       - eff_w_dd  * drawdown_penalty_t                // Drawdown penalty
       - w_idle    * idle_penalty_t                     // Idle penalty
       - hold_time_penalty                              // Position staleness rent
       - tx_cost                                        // Transaction friction

Component 1: Differential Sharpe Ratio (DSR)

Per-step incremental Sharpe contribution (Moody & Saffell 2001):

// Return (guard division by zero on prev_equity)
equity_denom = max(prev_equity, 1.0)
R_t = pnl_t / equity_denom

// Compute deltas BEFORE updating EMA (pre-update values for numerator+denominator)
delta_A = R_t - A_{t-1}
delta_B = R_t^2 - B_{t-1}
variance = B_{t-1} - A_{t-1}^2
denominator = max(|variance|, epsilon)^(3/2)

// DSR from pre-update state (Moody & Saffell correct formulation)
dsr_t = (B_{t-1} * delta_A - 0.5 * A_{t-1} * delta_B) / denominator

// NOW update EMA
A_t = A_{t-1} + eta * delta_A
B_t = B_{t-1} + eta * delta_B

State: 2 floats per episode (A_t, B_t) Cost: ~10 FLOPs per step Why: Dense reward that directly optimizes annualized Sharpe ratio. Each step gets immediate feedback on whether this action improved or degraded the portfolio's risk-adjusted return. Edge cases: prev_equity <= 0 → clamp denominator to 1.0. variance ≈ 0 (no return history) → dsr_t ≈ 0 (epsilon guard). Degenerate case returns 0.0 (not the CPU's (r*100).clamp(-5,5) — GPU is the canonical implementation).

Component 2: Normalized PnL

raw_pnl = position * (next_close - close)
pnl_ema = pnl_ema + eta * (raw_pnl - pnl_ema)     // running mean
pnl_var = pnl_var + eta * ((raw_pnl - pnl_ema)^2 - pnl_var)  // running variance
pnl_std = sqrt(pnl_var + 1e-8)
normalized_pnl = (raw_pnl - pnl_ema) / pnl_std     // z-scored PnL

State: 2 floats per episode (pnl_ema, pnl_var) Why: Raw PnL varies wildly across instruments and timeframes. Z-scoring makes the reward scale-invariant — same gradient magnitude whether trading ES (point value $50) or NQ (point value $20). Works equally well on tick data and 1-minute bars.

Component 3: Drawdown Penalty

equity = cash + position * close
peak_equity = max(peak_equity, equity)
drawdown = peak_equity > 0.0
         ? (peak_equity - equity) / peak_equity      // fractional drawdown [0, 1]
         : 0.0                                        // guard: no equity = no drawdown
drawdown_penalty_t = max(0, drawdown - dd_threshold)  // only penalize beyond threshold

State: 1 float per episode (peak_equity) Init: peak_equity = initial_capital (NOT zero — zero causes div-by-zero on first step) Parameter: dd_threshold (default 0.02 = 2% drawdown tolerance before penalty) Why: The model must learn to cut losing positions. Without drawdown awareness, it holds losers hoping for recovery — exactly the behavior that produces 100% max drawdown.

Component 4: Idle Penalty

is_flat = |position| < 0.001
if is_flat:
    flat_counter += 1
else:
    flat_counter = 0

idle_penalty_t = min(flat_counter * idle_scale, max_idle_penalty)

State: 1 float per episode (flat_counter, stored as float for GPU simplicity) Parameters: idle_scale (default 0.001), max_idle_penalty (default 0.05) Why: Replaces the old hold_bonus (which REWARDED flatness). Instead, penalizes prolonged inactivity. The model must demonstrate edge by participating in the market. Short flat periods (rebalancing) are fine — only sustained idleness is penalized.

Component 5: Regime-Adaptive Weight Scaling

adx   = features[bar_idx * MARKET_DIM + 40]       // already computed per-step
cusum = features[bar_idx * MARKET_DIM + 41]       // already in feature vector
trend_scale = clamp(adx / 25.0, 0.5, 2.0)
vol_scale   = clamp(cusum / 0.5, 0.5, 2.0)

eff_w_pnl = w_pnl * trend_scale                   // trending: ride momentum
eff_w_dsr = w_dsr * (2.0 - trend_scale)            // mean-reverting: optimize Sharpe
eff_w_dd  = w_dd  * vol_scale                      // volatile: protect capital

State: 0 floats (reads from existing feature buffer) Cost: ~5 FLOPs per step Why: Markets alternate between trending, mean-reverting, and volatile regimes. A fixed reward blend optimizes for the average regime — which is no regime. Regime-adaptive scaling teaches the model different strategies through the reward signal itself. In a strong trend (ADX>50), PnL weight doubles and DSR halves — the model learns to hold winners. In volatile chop (high CUSUM), drawdown protection doubles — the model learns to cut quickly. No extra hyperopt dimensions needed; the ADX/CUSUM features already encode regime state.

Component 6: Asymmetric Profit/Loss Scaling

asym_pnl = normalized_pnl >= 0
         ? normalized_pnl
         : normalized_pnl * loss_aversion           // default 1.5

State: 0 floats Parameter: loss_aversion (default 1.5, hyperopt range [1.0, 3.0]) Cost: 1 comparison + 1 multiply per step Why: Prospect theory for RL. Symmetric rewards produce symmetric behavior — the model treats a $100 loss and a $100 gain as equally important. In production trading, they're not. A loss reduces capital (compounding drag), damages confidence metrics (Sharpe, drawdown), and may trigger risk limits. loss_aversion=1.5 means a $100 loss produces 1.5x the negative reward of a $100 gain's positive reward. This teaches the model to: (1) cut losers early, (2) size positions conservatively when uncertain, (3) be aggressive only with high-conviction entries. The asymmetry naturally produces higher win rates and lower drawdowns without explicitly penalizing either.

Component 7: Position-Time Decay

if |position| > 0.001:
    hold_time += 1
    hold_time_penalty = hold_time * time_decay_rate  // linear rent
else:
    hold_time = 0
    hold_time_penalty = 0

State: 1 float per episode (hold_time) Parameters: time_decay_rate (default 0.0005, hyperopt range [0.0001, 0.005]) Cost: 1 comparison + 1 multiply per step Why: In HFT, a position that was entered 500 bars ago is stale — the alpha that justified entry has decayed. The model should either be actively managing (adjusting size, taking profit, stopping out) or flat. Position-time decay applies a linearly increasing "rent" for holding. After 100 steps at time_decay_rate=0.0005, the penalty is 0.05 — comparable to one transaction cost. This forces the model to: (1) enter positions with clear exit criteria, (2) take partial profits as the trade ages, (3) cut positions that aren't working within a bounded time horizon. Unlike a fixed hold penalty, the decay scales with duration — short holds (1-10 steps) pay almost nothing, but overstaying (100+ steps) becomes expensive. The time horizon is tunable via hyperopt to match the instrument's characteristic holding period.

Component 8: Transaction Cost (unchanged)

tx_cost = |delta_position| * close * tx_cost_multiplier

No change from current implementation. Already correct.

GPU State Layout

Introduce PORTFOLIO_STRIDE = 12 constant (replaces all hardcoded 3).

Extend the existing portfolio_states buffer from 3 floats to 12 floats per episode:

Index  Field          Init Value        Description
[0]    position       0.0               current position size
[1]    cash           initial_capital   available cash
[2]    portfolio_val  initial_capital   mark-to-market equity
[3]    dsr_A          0.0               EMA of returns
[4]    dsr_B          0.0               EMA of squared returns
[5]    pnl_ema        0.0               running mean PnL
[6]    pnl_var        0.0               running variance PnL
[7]    peak_equity    initial_capital   high-water mark (NOT zero — causes div/0)
[8]    flat_counter   0.0               consecutive flat steps (float for GPU)
[9]    prev_equity    initial_capital   equity at previous step (NOT zero — causes div/0)
[10]   hold_time      0.0               consecutive steps with position
[11]   realized_pnl   0.0               cumulative realized PnL

Memory impact: 9 extra floats × 2048 episodes × 4 bytes = 73 KB. Negligible.

Code locations requiring stride update (3 → PORTFOLIO_STRIDE)

All 7 hardcoded stride-3 references that MUST change:

File	Line	Current	Change to
experience_kernels.cu	169	portfolio_states + i * 3	i * PORTFOLIO_STRIDE
experience_kernels.cu	420	portfolio_states + i * 3	i * PORTFOLIO_STRIDE
gpu_experience_collector.rs	575	alloc_episodes * 3	alloc_episodes * PORTFOLIO_STRIDE
gpu_experience_collector.rs	579	vec![0.0; alloc_episodes * 3]	alloc_episodes * PORTFOLIO_STRIDE
gpu_experience_collector.rs	581	i * 3	i * PORTFOLIO_STRIDE
gpu_experience_collector.rs	1166	alloc_episodes * 3	alloc_episodes * PORTFOLIO_STRIDE
gpu_experience_collector.rs	1168	i * 3	i * PORTFOLIO_STRIDE

The existing PORTFOLIO_STATE_SIZE = 8 at gpu_experience_collector.rs:51 is for the PPO/legacy path (NOT used by DQN experience collection). Do not change it — introduce a new const PORTFOLIO_STRIDE: usize = 12 for the DQN path.

Note: experience_kernels.cu contains TWO different portfolio concepts in the same file — experience_env_step uses stride-3 (DQN, changing to 12), portfolio_sim_kernel uses stride-8 (legacy, unchanged).

Hyperopt Search Space

Add 7 new dimensions to the PSO/TPE search space (bringing total from 31D to 38D):

Parameter	Range	Scale	Phase Fast Fixed	Description
w_dsr	[0.1, 2.0]	linear	1.0	DSR (Sharpe) weight
w_pnl	[0.0, 1.0]	linear	0.3	Normalized PnL weight
w_dd	[0.0, 5.0]	linear	1.0	Drawdown penalty weight
w_idle	[0.0, 0.1]	linear	0.01	Idle penalty weight
dd_threshold	[0.01, 0.10]	linear	0.02	Drawdown tolerance before penalty
loss_aversion	[1.0, 3.0]	linear	1.5	Asymmetric loss scaling factor
time_decay_rate	[0.0001, 0.005]	log	0.0005	Position staleness rent per step

Phase Fast fixes all reward weights to sensible defaults — optimizer searches only learning dynamics. Phase Full can search reward weights to find the optimal blend for the instrument.

eta (DSR/normalization decay) already exists as dsr_eta in the search space.

Regime scaling (ADX/CUSUM) adds zero search dimensions — the scaling factors are derived from market features, not tuned.

Phase 3: Reward Tuning (optional future)

A potential third hyperopt phase dedicated to reward weights only. Fix architecture (Phase 2 best) AND dynamics (Phase 1 best), search only the 7 reward dimensions. This would be very fast (~10s/trial) since only the experience collection changes, not the network or training dynamics.

Kernel Changes

Modified kernel: experience_env_step

File: crates/ml/src/cuda_pipeline/experience_kernels.cu

New scalar parameters (7 reward weights + 2 feature buffer args):

float w_dsr, float w_pnl, float w_dd, float w_idle,
float dd_threshold, float loss_aversion, float time_decay_rate,
const float* __restrict__ features,   // market features for regime detection
int market_dim                        // MARKET_DIM (42)

Kernel body: replace the 3-line reward computation (lines 454-465) with the full composite formula including regime scaling, asymmetric PnL, and position-time decay. All state reads/writes through the existing portfolio_states pointer with stride 12 instead of 3.

The features pointer is already passed to experience_state_gather — extend to experience_env_step for ADX/CUSUM access.

Modified struct: ExperienceCollectorConfig

File: crates/ml/src/cuda_pipeline/gpu_experience_collector.rs

Add 7 fields matching the kernel parameters. Default values from TOML config.

Modified allocation: portfolio_states

Current: alloc_episodes * 3 floats New: alloc_episodes * 12 floats

All existing code that reads portfolio_states[0..3] must update to stride 12. New state at indices 3-11 initialized to zero at episode start (memset in reset_episode path).

TOML Configuration

Add to ALL training profiles (dqn-production.toml, dqn-smoketest.toml, dqn-hyperopt.toml):

[reward]
w_dsr = 1.0
w_pnl = 0.3
w_dd = 1.0
w_idle = 0.01
dd_threshold = 0.02
loss_aversion = 1.5
time_decay_rate = 0.0005

Requires new RewardSection struct in training_profile.rs with Option<f64> fields, and apply_to() mapping to DQNHyperparameters reward fields.

Add to config/training/dqn-hyperopt.toml [search_space]:

w_dsr = [0.1, 2.0]
w_pnl = [0.0, 1.0]
w_dd = [0.0, 5.0]
w_idle = [0.0, 0.1]
dd_threshold = [0.01, 0.10]

What Gets Removed

1. hold_reward / hold_penalty / hold_penalty_weight — replaced by idle_penalty (inverted semantics). The kernel parameter hold_rw at gpu_experience_collector.rs:1002-1020 is removed from the launch args.
2. CPU RewardNormalizer in reward.rs — normalization now in GPU kernel (pnl_ema/pnl_var)
3. CPU DSR code in reward.rs — DSR now in GPU kernel (dsr_A/dsr_B)
4. use_dsr bool flag in ExperienceCollectorConfig — redundant when w_dsr > 0 implies DSR enabled. Remove dual-control confusion.
5. hold_penalty_weight from hyperopt search space — replaced by w_idle

What Stays

1. Transaction cost model — already correct, no change
2. Backtest evaluator — still used for hyperopt objective and walk-forward validation
3. compute_validation_loss — still computes greedy Sharpe on validation data per epoch
4. compute_epoch_financials — still tracks training-set Sharpe for monitoring/early stopping
5. C51 distributional loss — still the RL loss function (unchanged). The reward function shapes WHAT the Q-values learn, not HOW they learn it.

Testing Strategy

1. Unit test: synthetic price series (linear trend, mean-reverting, random walk) → verify each reward component produces expected sign/magnitude
2. Smoke test: 3 epochs with composite reward on ES.FUT → loss decreases, Q-values non-zero, Sharpe > 0
3. A/B test: same hyperopt config with old reward vs new reward, compare trial Sharpe distributions
4. Regression: all 17 existing GPU tests must pass (reward weights default to backward-compatible values)

No Backward Compatibility Mode

The composite reward IS the reward function. There is no "legacy PnL-only" mode. All profiles — production, smoketest, hyperopt — use the full 8-component composite with the same default weights.

All TOML profiles set the same [reward] section. The smoketest uses smaller networks and fewer epochs but the same reward formula. Tests validate the real system, not a simplified stub.

The old hold_bonus path is deleted entirely. No feature flag, no fallback.

Performance Impact

• Per-step: +25 FLOPs (3 divisions, 2 sqrts, 15 multiplies, 2 comparisons, 2 clamps) on top of ~500 FLOPs for the existing kernel. <5% overhead.
• Memory: +73 KB for 2048 episodes. 0.009% of RTX 3050's 4GB.
• Feature read: 2 extra global memory reads per step (ADX, CUSUM from feature buffer — already in L2 cache from state_gather kernel).
• Kernel launch: zero additional launches. All computation fused into existing experience_env_step.
• Register pressure: +12 registers per thread for reward state. Well within sm_86/sm_90 limits (255 registers/thread).