# GPU Composite Reward Function — Design Spec ## Problem The current GPU reward is raw per-step PnL minus transaction costs: ``` reward = position * (next_close - close) - |delta| * close * tx_cost_multiplier + hold_bonus ``` This produces: - 3% win rate, 100% max drawdown, Sharpe ~0.06 - No risk-adjusted signal (model optimizes raw PnL, not Sharpe) - No drawdown awareness (model never learns to cut losses) - No participation incentive (model can achieve "good" Sharpe by trading once) - DSR implemented in CPU `reward.rs` but NOT wired to GPU kernel — dead code - Backtest evaluator only runs post-training (hyperopt objective), never during training ## Solution Replace the scalar PnL reward with a **GPU-native composite reward** computed entirely in the `experience_env_step` CUDA kernel. Zero CPU involvement. All state maintained in the existing per-episode `portfolio_states` GPU buffer. ### Reward Formula ``` // Regime-adaptive weights (from market features already in GPU memory) adx = features[bar_idx * MARKET_DIM + 40] cusum = features[bar_idx * MARKET_DIM + 41] trend_scale = clamp(adx / 25.0, 0.5, 2.0) // ADX>25 = trending vol_scale = clamp(cusum / 0.5, 0.5, 2.0) // high CUSUM = volatile eff_w_pnl = w_pnl * trend_scale // boost PnL weight in trends eff_w_dsr = w_dsr * (2.0 - trend_scale) // boost DSR weight in mean-reversion eff_w_dd = w_dd * vol_scale // boost DD protection in volatile regimes // Asymmetric profit/loss scaling (prospect theory) asym_pnl = normalized_pnl >= 0 ? normalized_pnl : normalized_pnl * loss_aversion // losses hurt more (default 1.5x) // Position-time decay (stale position rent) hold_time_penalty = position != 0 ? hold_time * time_decay_rate // linear rent per step held : 0.0 // Composite reward reward = eff_w_dsr * dsr_t // Sharpe contribution + eff_w_pnl * asym_pnl // Asymmetric normalized PnL - eff_w_dd * drawdown_penalty_t // Drawdown penalty - w_idle * idle_penalty_t // Idle penalty - hold_time_penalty // Position staleness rent - tx_cost // Transaction friction ``` ### Component 1: Differential Sharpe Ratio (DSR) Per-step incremental Sharpe contribution (Moody & Saffell 2001): ``` // Return (guard division by zero on prev_equity) equity_denom = max(prev_equity, 1.0) R_t = pnl_t / equity_denom // Compute deltas BEFORE updating EMA (pre-update values for numerator+denominator) delta_A = R_t - A_{t-1} delta_B = R_t^2 - B_{t-1} variance = B_{t-1} - A_{t-1}^2 denominator = max(|variance|, epsilon)^(3/2) // DSR from pre-update state (Moody & Saffell correct formulation) dsr_t = (B_{t-1} * delta_A - 0.5 * A_{t-1} * delta_B) / denominator // NOW update EMA A_t = A_{t-1} + eta * delta_A B_t = B_{t-1} + eta * delta_B ``` **State**: 2 floats per episode (A_t, B_t) **Cost**: ~10 FLOPs per step **Why**: Dense reward that directly optimizes annualized Sharpe ratio. Each step gets immediate feedback on whether this action improved or degraded the portfolio's risk-adjusted return. **Edge cases**: `prev_equity <= 0` → clamp denominator to 1.0. `variance ≈ 0` (no return history) → `dsr_t ≈ 0` (epsilon guard). Degenerate case returns 0.0 (not the CPU's `(r*100).clamp(-5,5)` — GPU is the canonical implementation). ### Component 2: Normalized PnL ``` raw_pnl = position * (next_close - close) pnl_ema = pnl_ema + eta * (raw_pnl - pnl_ema) // running mean pnl_var = pnl_var + eta * ((raw_pnl - pnl_ema)^2 - pnl_var) // running variance pnl_std = sqrt(pnl_var + 1e-8) normalized_pnl = (raw_pnl - pnl_ema) / pnl_std // z-scored PnL ``` **State**: 2 floats per episode (pnl_ema, pnl_var) **Why**: Raw PnL varies wildly across instruments and timeframes. Z-scoring makes the reward scale-invariant — same gradient magnitude whether trading ES (point value $50) or NQ (point value $20). Works equally well on tick data and 1-minute bars. ### Component 3: Drawdown Penalty ``` equity = cash + position * close peak_equity = max(peak_equity, equity) drawdown = peak_equity > 0.0 ? (peak_equity - equity) / peak_equity // fractional drawdown [0, 1] : 0.0 // guard: no equity = no drawdown drawdown_penalty_t = max(0, drawdown - dd_threshold) // only penalize beyond threshold ``` **State**: 1 float per episode (peak_equity) **Init**: `peak_equity = initial_capital` (NOT zero — zero causes div-by-zero on first step) **Parameter**: `dd_threshold` (default 0.02 = 2% drawdown tolerance before penalty) **Why**: The model must learn to cut losing positions. Without drawdown awareness, it holds losers hoping for recovery — exactly the behavior that produces 100% max drawdown. ### Component 4: Idle Penalty ``` is_flat = |position| < 0.001 if is_flat: flat_counter += 1 else: flat_counter = 0 idle_penalty_t = min(flat_counter * idle_scale, max_idle_penalty) ``` **State**: 1 float per episode (flat_counter, stored as float for GPU simplicity) **Parameters**: `idle_scale` (default 0.001), `max_idle_penalty` (default 0.05) **Why**: Replaces the old `hold_bonus` (which REWARDED flatness). Instead, penalizes prolonged inactivity. The model must demonstrate edge by participating in the market. Short flat periods (rebalancing) are fine — only sustained idleness is penalized. ### Component 5: Regime-Adaptive Weight Scaling ``` adx = features[bar_idx * MARKET_DIM + 40] // already computed per-step cusum = features[bar_idx * MARKET_DIM + 41] // already in feature vector trend_scale = clamp(adx / 25.0, 0.5, 2.0) vol_scale = clamp(cusum / 0.5, 0.5, 2.0) eff_w_pnl = w_pnl * trend_scale // trending: ride momentum eff_w_dsr = w_dsr * (2.0 - trend_scale) // mean-reverting: optimize Sharpe eff_w_dd = w_dd * vol_scale // volatile: protect capital ``` **State**: 0 floats (reads from existing feature buffer) **Cost**: ~5 FLOPs per step **Why**: Markets alternate between trending, mean-reverting, and volatile regimes. A fixed reward blend optimizes for the average regime — which is no regime. Regime-adaptive scaling teaches the model different strategies through the reward signal itself. In a strong trend (ADX>50), PnL weight doubles and DSR halves — the model learns to hold winners. In volatile chop (high CUSUM), drawdown protection doubles — the model learns to cut quickly. No extra hyperopt dimensions needed; the ADX/CUSUM features already encode regime state. ### Component 6: Asymmetric Profit/Loss Scaling ``` asym_pnl = normalized_pnl >= 0 ? normalized_pnl : normalized_pnl * loss_aversion // default 1.5 ``` **State**: 0 floats **Parameter**: `loss_aversion` (default 1.5, hyperopt range [1.0, 3.0]) **Cost**: 1 comparison + 1 multiply per step **Why**: Prospect theory for RL. Symmetric rewards produce symmetric behavior — the model treats a $100 loss and a $100 gain as equally important. In production trading, they're not. A loss reduces capital (compounding drag), damages confidence metrics (Sharpe, drawdown), and may trigger risk limits. `loss_aversion=1.5` means a $100 loss produces 1.5x the negative reward of a $100 gain's positive reward. This teaches the model to: (1) cut losers early, (2) size positions conservatively when uncertain, (3) be aggressive only with high-conviction entries. The asymmetry naturally produces higher win rates and lower drawdowns without explicitly penalizing either. ### Component 7: Position-Time Decay ``` if |position| > 0.001: hold_time += 1 hold_time_penalty = hold_time * time_decay_rate // linear rent else: hold_time = 0 hold_time_penalty = 0 ``` **State**: 1 float per episode (hold_time) **Parameters**: `time_decay_rate` (default 0.0005, hyperopt range [0.0001, 0.005]) **Cost**: 1 comparison + 1 multiply per step **Why**: In HFT, a position that was entered 500 bars ago is stale — the alpha that justified entry has decayed. The model should either be actively managing (adjusting size, taking profit, stopping out) or flat. Position-time decay applies a linearly increasing "rent" for holding. After 100 steps at `time_decay_rate=0.0005`, the penalty is 0.05 — comparable to one transaction cost. This forces the model to: (1) enter positions with clear exit criteria, (2) take partial profits as the trade ages, (3) cut positions that aren't working within a bounded time horizon. Unlike a fixed hold penalty, the decay scales with duration — short holds (1-10 steps) pay almost nothing, but overstaying (100+ steps) becomes expensive. The time horizon is tunable via hyperopt to match the instrument's characteristic holding period. ### Component 8: Transaction Cost (unchanged) ``` tx_cost = |delta_position| * close * tx_cost_multiplier ``` No change from current implementation. Already correct. ## GPU State Layout Introduce `PORTFOLIO_STRIDE = 12` constant (replaces all hardcoded `3`). Extend the existing `portfolio_states` buffer from 3 floats to 12 floats per episode: ``` Index Field Init Value Description [0] position 0.0 current position size [1] cash initial_capital available cash [2] portfolio_val initial_capital mark-to-market equity [3] dsr_A 0.0 EMA of returns [4] dsr_B 0.0 EMA of squared returns [5] pnl_ema 0.0 running mean PnL [6] pnl_var 0.0 running variance PnL [7] peak_equity initial_capital high-water mark (NOT zero — causes div/0) [8] flat_counter 0.0 consecutive flat steps (float for GPU) [9] prev_equity initial_capital equity at previous step (NOT zero — causes div/0) [10] hold_time 0.0 consecutive steps with position [11] realized_pnl 0.0 cumulative realized PnL ``` **Memory impact**: 9 extra floats × 2048 episodes × 4 bytes = 73 KB. Negligible. ### Code locations requiring stride update (3 → PORTFOLIO_STRIDE) All 7 hardcoded stride-3 references that MUST change: | File | Line | Current | Change to | |------|------|---------|-----------| | `experience_kernels.cu` | 169 | `portfolio_states + i * 3` | `i * PORTFOLIO_STRIDE` | | `experience_kernels.cu` | 420 | `portfolio_states + i * 3` | `i * PORTFOLIO_STRIDE` | | `gpu_experience_collector.rs` | 575 | `alloc_episodes * 3` | `alloc_episodes * PORTFOLIO_STRIDE` | | `gpu_experience_collector.rs` | 579 | `vec![0.0; alloc_episodes * 3]` | `alloc_episodes * PORTFOLIO_STRIDE` | | `gpu_experience_collector.rs` | 581 | `i * 3` | `i * PORTFOLIO_STRIDE` | | `gpu_experience_collector.rs` | 1166 | `alloc_episodes * 3` | `alloc_episodes * PORTFOLIO_STRIDE` | | `gpu_experience_collector.rs` | 1168 | `i * 3` | `i * PORTFOLIO_STRIDE` | The existing `PORTFOLIO_STATE_SIZE = 8` at `gpu_experience_collector.rs:51` is for the PPO/legacy path (NOT used by DQN experience collection). Do not change it — introduce a new `const PORTFOLIO_STRIDE: usize = 12` for the DQN path. Note: `experience_kernels.cu` contains TWO different portfolio concepts in the same file — `experience_env_step` uses stride-3 (DQN, changing to 12), `portfolio_sim_kernel` uses stride-8 (legacy, unchanged). ## Hyperopt Search Space Add 7 new dimensions to the PSO/TPE search space (bringing total from 31D to 38D): | Parameter | Range | Scale | Phase Fast Fixed | Description | |-----------|-------|-------|-----------------|-------------| | `w_dsr` | [0.1, 2.0] | linear | 1.0 | DSR (Sharpe) weight | | `w_pnl` | [0.0, 1.0] | linear | 0.3 | Normalized PnL weight | | `w_dd` | [0.0, 5.0] | linear | 1.0 | Drawdown penalty weight | | `w_idle` | [0.0, 0.1] | linear | 0.01 | Idle penalty weight | | `dd_threshold` | [0.01, 0.10] | linear | 0.02 | Drawdown tolerance before penalty | | `loss_aversion` | [1.0, 3.0] | linear | 1.5 | Asymmetric loss scaling factor | | `time_decay_rate` | [0.0001, 0.005] | log | 0.0005 | Position staleness rent per step | Phase Fast fixes all reward weights to sensible defaults — optimizer searches only learning dynamics. Phase Full can search reward weights to find the optimal blend for the instrument. `eta` (DSR/normalization decay) already exists as `dsr_eta` in the search space. Regime scaling (ADX/CUSUM) adds zero search dimensions — the scaling factors are derived from market features, not tuned. ### Phase 3: Reward Tuning (optional future) A potential third hyperopt phase dedicated to reward weights only. Fix architecture (Phase 2 best) AND dynamics (Phase 1 best), search only the 7 reward dimensions. This would be very fast (~10s/trial) since only the experience collection changes, not the network or training dynamics. ## Kernel Changes ### Modified kernel: `experience_env_step` File: `crates/ml/src/cuda_pipeline/experience_kernels.cu` New scalar parameters (7 reward weights + 2 feature buffer args): ```cuda float w_dsr, float w_pnl, float w_dd, float w_idle, float dd_threshold, float loss_aversion, float time_decay_rate, const float* __restrict__ features, // market features for regime detection int market_dim // MARKET_DIM (42) ``` Kernel body: replace the 3-line reward computation (lines 454-465) with the full composite formula including regime scaling, asymmetric PnL, and position-time decay. All state reads/writes through the existing `portfolio_states` pointer with stride 12 instead of 3. The `features` pointer is already passed to `experience_state_gather` — extend to `experience_env_step` for ADX/CUSUM access. ### Modified struct: `ExperienceCollectorConfig` File: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs` Add 7 fields matching the kernel parameters. Default values from TOML config. ### Modified allocation: `portfolio_states` Current: `alloc_episodes * 3` floats New: `alloc_episodes * 12` floats All existing code that reads `portfolio_states[0..3]` must update to stride 12. New state at indices 3-11 initialized to zero at episode start (memset in `reset_episode` path). ## TOML Configuration Add to ALL training profiles (`dqn-production.toml`, `dqn-smoketest.toml`, `dqn-hyperopt.toml`): ```toml [reward] w_dsr = 1.0 w_pnl = 0.3 w_dd = 1.0 w_idle = 0.01 dd_threshold = 0.02 loss_aversion = 1.5 time_decay_rate = 0.0005 ``` Requires new `RewardSection` struct in `training_profile.rs` with `Option` fields, and `apply_to()` mapping to `DQNHyperparameters` reward fields. Add to `config/training/dqn-hyperopt.toml` `[search_space]`: ```toml w_dsr = [0.1, 2.0] w_pnl = [0.0, 1.0] w_dd = [0.0, 5.0] w_idle = [0.0, 0.1] dd_threshold = [0.01, 0.10] ``` ## What Gets Removed 1. **`hold_reward` / `hold_penalty` / `hold_penalty_weight`** — replaced by `idle_penalty` (inverted semantics). The kernel parameter `hold_rw` at `gpu_experience_collector.rs:1002-1020` is removed from the launch args. 2. **CPU `RewardNormalizer`** in `reward.rs` — normalization now in GPU kernel (pnl_ema/pnl_var) 3. **CPU DSR code** in `reward.rs` — DSR now in GPU kernel (dsr_A/dsr_B) 4. **`use_dsr` bool flag** in `ExperienceCollectorConfig` — redundant when `w_dsr > 0` implies DSR enabled. Remove dual-control confusion. 5. **`hold_penalty_weight`** from hyperopt search space — replaced by `w_idle` ## What Stays 1. **Transaction cost model** — already correct, no change 2. **Backtest evaluator** — still used for hyperopt objective and walk-forward validation 3. **`compute_validation_loss`** — still computes greedy Sharpe on validation data per epoch 4. **`compute_epoch_financials`** — still tracks training-set Sharpe for monitoring/early stopping 5. **C51 distributional loss** — still the RL loss function (unchanged). The reward function shapes WHAT the Q-values learn, not HOW they learn it. ## Testing Strategy 1. **Unit test**: synthetic price series (linear trend, mean-reverting, random walk) → verify each reward component produces expected sign/magnitude 2. **Smoke test**: 3 epochs with composite reward on ES.FUT → loss decreases, Q-values non-zero, Sharpe > 0 3. **A/B test**: same hyperopt config with old reward vs new reward, compare trial Sharpe distributions 4. **Regression**: all 17 existing GPU tests must pass (reward weights default to backward-compatible values) ## No Backward Compatibility Mode The composite reward IS the reward function. There is no "legacy PnL-only" mode. All profiles — production, smoketest, hyperopt — use the full 8-component composite with the same default weights. **All TOML profiles** set the same `[reward]` section. The smoketest uses smaller networks and fewer epochs but the same reward formula. Tests validate the real system, not a simplified stub. The old `hold_bonus` path is deleted entirely. No feature flag, no fallback. ## Performance Impact - **Per-step**: +25 FLOPs (3 divisions, 2 sqrts, 15 multiplies, 2 comparisons, 2 clamps) on top of ~500 FLOPs for the existing kernel. <5% overhead. - **Memory**: +73 KB for 2048 episodes. 0.009% of RTX 3050's 4GB. - **Feature read**: 2 extra global memory reads per step (ADX, CUSUM from feature buffer — already in L2 cache from state_gather kernel). - **Kernel launch**: zero additional launches. All computation fused into existing `experience_env_step`. - **Register pressure**: +12 registers per thread for reward state. Well within sm_86/sm_90 limits (255 registers/thread).