# GPU Composite Reward Function — Design Spec

## Problem

The current GPU reward is raw per-step PnL minus transaction costs:
```
reward = position * (next_close - close) - |delta| * close * tx_cost_multiplier + hold_bonus
```

This produces:
- 3% win rate, 100% max drawdown, Sharpe ~0.06
- No risk-adjusted signal (model optimizes raw PnL, not Sharpe)
- No drawdown awareness (model never learns to cut losses)
- No participation incentive (model can achieve "good" Sharpe by trading once)
- DSR implemented in CPU `reward.rs` but NOT wired to GPU kernel — dead code
- Backtest evaluator only runs post-training (hyperopt objective), never during training

## Solution

Replace the scalar PnL reward with a **GPU-native composite reward** computed entirely in the `experience_env_step` CUDA kernel. Zero CPU involvement. All state maintained in the existing per-episode `portfolio_states` GPU buffer.

### Reward Formula

```
// Regime-adaptive weights (from market features already in GPU memory)
adx   = features[bar_idx * MARKET_DIM + 40]
cusum = features[bar_idx * MARKET_DIM + 41]
trend_scale = clamp(adx / 25.0, 0.5, 2.0)           // ADX>25 = trending
vol_scale   = clamp(cusum / 0.5, 0.5, 2.0)           // high CUSUM = volatile

eff_w_pnl  = w_pnl * trend_scale                     // boost PnL weight in trends
eff_w_dsr  = w_dsr * (2.0 - trend_scale)              // boost DSR weight in mean-reversion
eff_w_dd   = w_dd  * vol_scale                        // boost DD protection in volatile regimes

// Asymmetric profit/loss scaling (prospect theory)
asym_pnl = normalized_pnl >= 0
         ? normalized_pnl
         : normalized_pnl * loss_aversion              // losses hurt more (default 1.5x)

// Position-time decay (stale position rent)
hold_time_penalty = position != 0
                  ? hold_time * time_decay_rate        // linear rent per step held
                  : 0.0

// Composite reward
reward = eff_w_dsr * dsr_t                             // Sharpe contribution
       + eff_w_pnl * asym_pnl                          // Asymmetric normalized PnL
       - eff_w_dd  * drawdown_penalty_t                // Drawdown penalty
       - w_idle    * idle_penalty_t                     // Idle penalty
       - hold_time_penalty                              // Position staleness rent
       - tx_cost                                        // Transaction friction
```

### Component 1: Differential Sharpe Ratio (DSR)

Per-step incremental Sharpe contribution (Moody & Saffell 2001):

```
// Return (guard division by zero on prev_equity)
equity_denom = max(prev_equity, 1.0)
R_t = pnl_t / equity_denom

// Compute deltas BEFORE updating EMA (pre-update values for numerator+denominator)
delta_A = R_t - A_{t-1}
delta_B = R_t^2 - B_{t-1}
variance = B_{t-1} - A_{t-1}^2
denominator = max(|variance|, epsilon)^(3/2)

// DSR from pre-update state (Moody & Saffell correct formulation)
dsr_t = (B_{t-1} * delta_A - 0.5 * A_{t-1} * delta_B) / denominator

// NOW update EMA
A_t = A_{t-1} + eta * delta_A
B_t = B_{t-1} + eta * delta_B
```

**State**: 2 floats per episode (A_t, B_t)
**Cost**: ~10 FLOPs per step
**Why**: Dense reward that directly optimizes annualized Sharpe ratio. Each step gets immediate feedback on whether this action improved or degraded the portfolio's risk-adjusted return.
**Edge cases**: `prev_equity <= 0` → clamp denominator to 1.0. `variance ≈ 0` (no return history) → `dsr_t ≈ 0` (epsilon guard). Degenerate case returns 0.0 (not the CPU's `(r*100).clamp(-5,5)` — GPU is the canonical implementation).

### Component 2: Normalized PnL

```
raw_pnl = position * (next_close - close)
pnl_ema = pnl_ema + eta * (raw_pnl - pnl_ema)     // running mean
pnl_var = pnl_var + eta * ((raw_pnl - pnl_ema)^2 - pnl_var)  // running variance
pnl_std = sqrt(pnl_var + 1e-8)
normalized_pnl = (raw_pnl - pnl_ema) / pnl_std     // z-scored PnL
```

**State**: 2 floats per episode (pnl_ema, pnl_var)
**Why**: Raw PnL varies wildly across instruments and timeframes. Z-scoring makes the reward scale-invariant — same gradient magnitude whether trading ES (point value $50) or NQ (point value $20). Works equally well on tick data and 1-minute bars.

### Component 3: Drawdown Penalty

```
equity = cash + position * close
peak_equity = max(peak_equity, equity)
drawdown = peak_equity > 0.0
         ? (peak_equity - equity) / peak_equity      // fractional drawdown [0, 1]
         : 0.0                                        // guard: no equity = no drawdown
drawdown_penalty_t = max(0, drawdown - dd_threshold)  // only penalize beyond threshold
```

**State**: 1 float per episode (peak_equity)
**Init**: `peak_equity = initial_capital` (NOT zero — zero causes div-by-zero on first step)
**Parameter**: `dd_threshold` (default 0.02 = 2% drawdown tolerance before penalty)
**Why**: The model must learn to cut losing positions. Without drawdown awareness, it holds losers hoping for recovery — exactly the behavior that produces 100% max drawdown.

### Component 4: Idle Penalty

```
is_flat = |position| < 0.001
if is_flat:
    flat_counter += 1
else:
    flat_counter = 0

idle_penalty_t = min(flat_counter * idle_scale, max_idle_penalty)
```

**State**: 1 float per episode (flat_counter, stored as float for GPU simplicity)
**Parameters**: `idle_scale` (default 0.001), `max_idle_penalty` (default 0.05)
**Why**: Replaces the old `hold_bonus` (which REWARDED flatness). Instead, penalizes prolonged inactivity. The model must demonstrate edge by participating in the market. Short flat periods (rebalancing) are fine — only sustained idleness is penalized.

### Component 5: Regime-Adaptive Weight Scaling

```
adx   = features[bar_idx * MARKET_DIM + 40]       // already computed per-step
cusum = features[bar_idx * MARKET_DIM + 41]       // already in feature vector
trend_scale = clamp(adx / 25.0, 0.5, 2.0)
vol_scale   = clamp(cusum / 0.5, 0.5, 2.0)

eff_w_pnl = w_pnl * trend_scale                   // trending: ride momentum
eff_w_dsr = w_dsr * (2.0 - trend_scale)            // mean-reverting: optimize Sharpe
eff_w_dd  = w_dd  * vol_scale                      // volatile: protect capital
```

**State**: 0 floats (reads from existing feature buffer)
**Cost**: ~5 FLOPs per step
**Why**: Markets alternate between trending, mean-reverting, and volatile regimes. A fixed reward blend optimizes for the average regime — which is no regime. Regime-adaptive scaling teaches the model different strategies through the reward signal itself. In a strong trend (ADX>50), PnL weight doubles and DSR halves — the model learns to hold winners. In volatile chop (high CUSUM), drawdown protection doubles — the model learns to cut quickly. No extra hyperopt dimensions needed; the ADX/CUSUM features already encode regime state.

### Component 6: Asymmetric Profit/Loss Scaling

```
asym_pnl = normalized_pnl >= 0
         ? normalized_pnl
         : normalized_pnl * loss_aversion           // default 1.5
```

**State**: 0 floats
**Parameter**: `loss_aversion` (default 1.5, hyperopt range [1.0, 3.0])
**Cost**: 1 comparison + 1 multiply per step
**Why**: Prospect theory for RL. Symmetric rewards produce symmetric behavior — the model treats a $100 loss and a $100 gain as equally important. In production trading, they're not. A loss reduces capital (compounding drag), damages confidence metrics (Sharpe, drawdown), and may trigger risk limits. `loss_aversion=1.5` means a $100 loss produces 1.5x the negative reward of a $100 gain's positive reward. This teaches the model to: (1) cut losers early, (2) size positions conservatively when uncertain, (3) be aggressive only with high-conviction entries. The asymmetry naturally produces higher win rates and lower drawdowns without explicitly penalizing either.

### Component 7: Position-Time Decay

```
if |position| > 0.001:
    hold_time += 1
    hold_time_penalty = hold_time * time_decay_rate  // linear rent
else:
    hold_time = 0
    hold_time_penalty = 0
```

**State**: 1 float per episode (hold_time)
**Parameters**: `time_decay_rate` (default 0.0005, hyperopt range [0.0001, 0.005])
**Cost**: 1 comparison + 1 multiply per step
**Why**: In HFT, a position that was entered 500 bars ago is stale — the alpha that justified entry has decayed. The model should either be actively managing (adjusting size, taking profit, stopping out) or flat. Position-time decay applies a linearly increasing "rent" for holding. After 100 steps at `time_decay_rate=0.0005`, the penalty is 0.05 — comparable to one transaction cost. This forces the model to: (1) enter positions with clear exit criteria, (2) take partial profits as the trade ages, (3) cut positions that aren't working within a bounded time horizon. Unlike a fixed hold penalty, the decay scales with duration — short holds (1-10 steps) pay almost nothing, but overstaying (100+ steps) becomes expensive. The time horizon is tunable via hyperopt to match the instrument's characteristic holding period.

### Component 8: Transaction Cost (unchanged)

```
tx_cost = |delta_position| * close * tx_cost_multiplier
```

No change from current implementation. Already correct.

## GPU State Layout

Introduce `PORTFOLIO_STRIDE = 12` constant (replaces all hardcoded `3`).

Extend the existing `portfolio_states` buffer from 3 floats to 12 floats per episode:

```
Index  Field          Init Value        Description
[0]    position       0.0               current position size
[1]    cash           initial_capital   available cash
[2]    portfolio_val  initial_capital   mark-to-market equity
[3]    dsr_A          0.0               EMA of returns
[4]    dsr_B          0.0               EMA of squared returns
[5]    pnl_ema        0.0               running mean PnL
[6]    pnl_var        0.0               running variance PnL
[7]    peak_equity    initial_capital   high-water mark (NOT zero — causes div/0)
[8]    flat_counter   0.0               consecutive flat steps (float for GPU)
[9]    prev_equity    initial_capital   equity at previous step (NOT zero — causes div/0)
[10]   hold_time      0.0               consecutive steps with position
[11]   realized_pnl   0.0               cumulative realized PnL
```

**Memory impact**: 9 extra floats × 2048 episodes × 4 bytes = 73 KB. Negligible.

### Code locations requiring stride update (3 → PORTFOLIO_STRIDE)

All 7 hardcoded stride-3 references that MUST change:

| File | Line | Current | Change to |
|------|------|---------|-----------|
| `experience_kernels.cu` | 169 | `portfolio_states + i * 3` | `i * PORTFOLIO_STRIDE` |
| `experience_kernels.cu` | 420 | `portfolio_states + i * 3` | `i * PORTFOLIO_STRIDE` |
| `gpu_experience_collector.rs` | 575 | `alloc_episodes * 3` | `alloc_episodes * PORTFOLIO_STRIDE` |
| `gpu_experience_collector.rs` | 579 | `vec![0.0; alloc_episodes * 3]` | `alloc_episodes * PORTFOLIO_STRIDE` |
| `gpu_experience_collector.rs` | 581 | `i * 3` | `i * PORTFOLIO_STRIDE` |
| `gpu_experience_collector.rs` | 1166 | `alloc_episodes * 3` | `alloc_episodes * PORTFOLIO_STRIDE` |
| `gpu_experience_collector.rs` | 1168 | `i * 3` | `i * PORTFOLIO_STRIDE` |

The existing `PORTFOLIO_STATE_SIZE = 8` at `gpu_experience_collector.rs:51` is for the PPO/legacy path (NOT used by DQN experience collection). Do not change it — introduce a new `const PORTFOLIO_STRIDE: usize = 12` for the DQN path.

Note: `experience_kernels.cu` contains TWO different portfolio concepts in the same file — `experience_env_step` uses stride-3 (DQN, changing to 12), `portfolio_sim_kernel` uses stride-8 (legacy, unchanged).

## Hyperopt Search Space

Add 7 new dimensions to the PSO/TPE search space (bringing total from 31D to 38D):

| Parameter | Range | Scale | Phase Fast Fixed | Description |
|-----------|-------|-------|-----------------|-------------|
| `w_dsr` | [0.1, 2.0] | linear | 1.0 | DSR (Sharpe) weight |
| `w_pnl` | [0.0, 1.0] | linear | 0.3 | Normalized PnL weight |
| `w_dd` | [0.0, 5.0] | linear | 1.0 | Drawdown penalty weight |
| `w_idle` | [0.0, 0.1] | linear | 0.01 | Idle penalty weight |
| `dd_threshold` | [0.01, 0.10] | linear | 0.02 | Drawdown tolerance before penalty |
| `loss_aversion` | [1.0, 3.0] | linear | 1.5 | Asymmetric loss scaling factor |
| `time_decay_rate` | [0.0001, 0.005] | log | 0.0005 | Position staleness rent per step |

Phase Fast fixes all reward weights to sensible defaults — optimizer searches only learning dynamics. Phase Full can search reward weights to find the optimal blend for the instrument.

`eta` (DSR/normalization decay) already exists as `dsr_eta` in the search space.

Regime scaling (ADX/CUSUM) adds zero search dimensions — the scaling factors are derived from market features, not tuned.

### Phase 3: Reward Tuning (optional future)

A potential third hyperopt phase dedicated to reward weights only. Fix architecture (Phase 2 best) AND dynamics (Phase 1 best), search only the 7 reward dimensions. This would be very fast (~10s/trial) since only the experience collection changes, not the network or training dynamics.

## Kernel Changes

### Modified kernel: `experience_env_step`

File: `crates/ml/src/cuda_pipeline/experience_kernels.cu`

New scalar parameters (7 reward weights + 2 feature buffer args):
```cuda
float w_dsr, float w_pnl, float w_dd, float w_idle,
float dd_threshold, float loss_aversion, float time_decay_rate,
const float* __restrict__ features,   // market features for regime detection
int market_dim                        // MARKET_DIM (42)
```

Kernel body: replace the 3-line reward computation (lines 454-465) with the full composite formula including regime scaling, asymmetric PnL, and position-time decay. All state reads/writes through the existing `portfolio_states` pointer with stride 12 instead of 3.

The `features` pointer is already passed to `experience_state_gather` — extend to `experience_env_step` for ADX/CUSUM access.

### Modified struct: `ExperienceCollectorConfig`

File: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs`

Add 7 fields matching the kernel parameters. Default values from TOML config.

### Modified allocation: `portfolio_states`

Current: `alloc_episodes * 3` floats
New: `alloc_episodes * 12` floats

All existing code that reads `portfolio_states[0..3]` must update to stride 12. New state at indices 3-11 initialized to zero at episode start (memset in `reset_episode` path).

## TOML Configuration

Add to ALL training profiles (`dqn-production.toml`, `dqn-smoketest.toml`, `dqn-hyperopt.toml`):
```toml
[reward]
w_dsr = 1.0
w_pnl = 0.3
w_dd = 1.0
w_idle = 0.01
dd_threshold = 0.02
loss_aversion = 1.5
time_decay_rate = 0.0005
```

Requires new `RewardSection` struct in `training_profile.rs` with `Option<f64>` fields, and `apply_to()` mapping to `DQNHyperparameters` reward fields.

Add to `config/training/dqn-hyperopt.toml` `[search_space]`:
```toml
w_dsr = [0.1, 2.0]
w_pnl = [0.0, 1.0]
w_dd = [0.0, 5.0]
w_idle = [0.0, 0.1]
dd_threshold = [0.01, 0.10]
```

## What Gets Removed

1. **`hold_reward` / `hold_penalty` / `hold_penalty_weight`** — replaced by `idle_penalty` (inverted semantics). The kernel parameter `hold_rw` at `gpu_experience_collector.rs:1002-1020` is removed from the launch args.
2. **CPU `RewardNormalizer`** in `reward.rs` — normalization now in GPU kernel (pnl_ema/pnl_var)
3. **CPU DSR code** in `reward.rs` — DSR now in GPU kernel (dsr_A/dsr_B)
4. **`use_dsr` bool flag** in `ExperienceCollectorConfig` — redundant when `w_dsr > 0` implies DSR enabled. Remove dual-control confusion.
5. **`hold_penalty_weight`** from hyperopt search space — replaced by `w_idle`

## What Stays

1. **Transaction cost model** — already correct, no change
2. **Backtest evaluator** — still used for hyperopt objective and walk-forward validation
3. **`compute_validation_loss`** — still computes greedy Sharpe on validation data per epoch
4. **`compute_epoch_financials`** — still tracks training-set Sharpe for monitoring/early stopping
5. **C51 distributional loss** — still the RL loss function (unchanged). The reward function shapes WHAT the Q-values learn, not HOW they learn it.

## Testing Strategy

1. **Unit test**: synthetic price series (linear trend, mean-reverting, random walk) → verify each reward component produces expected sign/magnitude
2. **Smoke test**: 3 epochs with composite reward on ES.FUT → loss decreases, Q-values non-zero, Sharpe > 0
3. **A/B test**: same hyperopt config with old reward vs new reward, compare trial Sharpe distributions
4. **Regression**: all 17 existing GPU tests must pass (reward weights default to backward-compatible values)

## No Backward Compatibility Mode

The composite reward IS the reward function. There is no "legacy PnL-only" mode. All profiles — production, smoketest, hyperopt — use the full 8-component composite with the same default weights.

**All TOML profiles** set the same `[reward]` section. The smoketest uses smaller networks and fewer epochs but the same reward formula. Tests validate the real system, not a simplified stub.

The old `hold_bonus` path is deleted entirely. No feature flag, no fallback.

## Performance Impact

- **Per-step**: +25 FLOPs (3 divisions, 2 sqrts, 15 multiplies, 2 comparisons, 2 clamps) on top of ~500 FLOPs for the existing kernel. <5% overhead.
- **Memory**: +73 KB for 2048 episodes. 0.009% of RTX 3050's 4GB.
- **Feature read**: 2 extra global memory reads per step (ADX, CUSUM from feature buffer — already in L2 cache from state_gather kernel).
- **Kernel launch**: zero additional launches. All computation fused into existing `experience_env_step`.
- **Register pressure**: +12 registers per thread for reward state. Well within sm_86/sm_90 limits (255 registers/thread).
