# Phase 1 — Reward Term Inventory

Companion to `2026-04-21-unified-train-val-env-design.md`. Enumerates every
term that contributes to the training or validation reward, classifies each
as P&L-aligned (keeper) or behavioral (remover), and records the location
in source. Evidence from `experience_kernels.cu` line numbers as of merge
commit `9223795c7`.

## Core P&L-aligned terms (present in both envs — keepers)

These map directly to the equity change a real trading account would
experience. Removing any of them would make the reward diverge from real P&L.

| Term | Training location | Validation location | Semantics |
|---|---|---|---|
| Core trade return | `experience_kernels.cu:1687` (`2.0 × vol_normalized_return`) | `backtest_env_kernel.cu:251` (`step_ret = ΔV/V`) | Realized equity change |
| Transaction cost | `trade_physics.cuh:execute_trade` | Same (shared) | Broker fees + spread |
| Drawdown penalty | `experience_kernels.cu:1885` (`compute_drawdown_penalty`) | Implicit via capital floor | Pushback when equity < peak × (1 - dd_threshold) |
| Inventory/holding cost | `experience_kernels.cu:1839` (`-holding_cost_rate × \|pos\|`) | `backtest_env_kernel.cu:257` (same) | Continuous carry cost of holding position |
| Churn penalty | Via commitment_lambda (see below) | `backtest_env_kernel.cu:266` (`churn_penalty_scale`) | Pushback on rapid position flips |
| Capital floor | `experience_kernels.cu:1924` (`reward = -5 - 5×risk_taken`) | `backtest_env_kernel.cu:122-139` (liquidation) | Catastrophic penalty when equity breaks the 25% drawdown floor |

These are semantically consistent across envs (though magnitudes differ —
training uses larger absolute penalties because of vol normalization). The
unified env should preserve these with identical numerics.

## Training-only behavioral terms (removers)

Each of these rewards a *decision-making behavior* rather than a *realized
outcome*. They were added to address specific training pathologies but
break the design goal that training reward should equal realized P&L.

### Sparse (trade-completion) behavioral terms

| Term | Location | Formula | Why it's behavioral |
|---|---|---|---|
| `order_credit_weight` | `experience_kernels.cu:1700` | `+= w × (market_cost - taken_cost) × 100 / equity` | Rewards choosing limit-over-market order types regardless of whether the limit actually got a better fill (assumes theoretical savings). Real fills are already captured in actual `tx_cost` via `compute_tx_cost(position, raw_close, ..., order_type_idx, ...)`. |
| `risk_efficiency_weight` | `experience_kernels.cu:1706` | `+= w × segment_return / \|intra_trade_max_dd\|` | Rewards trades with low path-drawdown. The drawdown penalty already exists as a separate P&L-aligned term — this double-counts but asymmetrically (only on winners). |
| `urgency_credit_weight` | `experience_kernels.cu:1717` | `+= w × fill_improvement / vol_proxy` | Rewards favorable fill vs entry price. But the fill price IS the entry price — `fill_improvement = (raw_close - entry_price)/entry_price` is just an alias for the unrealized P&L at the moment of evaluation. Redundant with the core return. |
| `kelly_sizing_weight` | `experience_kernels.cu:1746` | `-= w × \|actual_fraction - kelly_f\| × 0.5` | Penalizes deviation from Kelly-optimal fraction computed with priors over win/loss statistics. Kelly is a derived quantity — rewarding for matching it is rewarding the model for matching a formula, not for making money. |

### Dense (per-bar) behavioral terms

| Term | Location | Formula | Why it's behavioral |
|---|---|---|---|
| `micro_reward_scale` | `experience_kernels.cu:1836` | `reward = w × tanh((flow_momentum + price_confirm + book_aggression + hold_quality - spread_penalty) / temp)` | Rewards alignment with OFI momentum signals regardless of realized P&L. The network can score well on this by following the signal dogmatically even when it's misleading. |
| `commitment_lambda` | `experience_kernels.cu:1890-1896` | `-= λ × (1 - exp(-\|Δpos\| / τ)) × cost_scale` | "Action commitment penalty." Discourages oscillation. But transaction cost already penalizes position changes, and churn penalty already handles rapid flips. Triple-counting. |

### Always-applied behavioral terms

| Term | Location | Formula | Why it's behavioral |
|---|---|---|---|
| `reward_noise_scale` | `experience_kernels.cu:1874-1881` | `reward += noise_magnitude × pseudo_noise` (label smoothing) | Regularization proxy. Inserts noise into the reward signal. Useful for preventing overfitting, but should be applied at the gradient/target level (e.g., Q-target smoothing), not at the reward level where it changes the objective. |
| `position_entropy_weight` | `experience_kernels.cu:1976` | `+= w × combined_entropy` at episode end | Rewards exploration across action buckets. Exploration is already handled by ε-greedy + noisy nets; rewarding it in the objective changes what "optimal" means. |

### Dormant / already-removed behavioral terms

These appear in the config / plumbing but are no longer added to the
reward. Dead weight:

| Term | Status |
|---|---|
| `w_dsr` (Differential Sharpe Ratio) | EMA is tracked (`experience_kernels.cu:1868-1870`) but no longer added to reward. Dead plumbing. |
| `exit_timing_weight` | Commented out (experience_kernels.cu:1724 says "REMOVED: post_exit_move used raw_next"). Dead plumbing. |
| `ofi_reward_weight` | Not actually applied inside the kernel — replaced by `micro_reward_scale` path. Dead. |

## Validation-only behavioral terms

| Term | Status |
|---|---|
| `opp_cost_scale` | Multiplied by `0.0f × is_flat` at `backtest_env_kernel.cu:275` — always zero. Dead. |

## Summary

- **7 P&L-aligned terms** to preserve in the unified env
- **8 active behavioral training-only terms** to delete
- **4 dormant/dead terms** to clean up (including cleanup of their config
  fields and kernel args)

## Magnitude comparison (approximate per-bar contribution)

With production-config weights (`w_dsr=5.0`, `order_credit=1.0`, etc.)
measured from the 50-epoch smoke test:

| Term | Magnitude per bar |
|---|---|
| Core vol-normalized return | ±0.05 to ±0.5 (bulk of signal) |
| `order_credit_weight × ...` | +0.01 to +0.2 when trade completes (~5% of bars) |
| `risk_efficiency_weight × ...` | 0 to +0.5 on winners only |
| `urgency_credit_weight × ...` | 0 to +1.0 on any positioned bar with entry |
| `micro_reward_scale × tanh(...)` | 0 (disabled: `micro_reward_scale=0.0` in smoke config) |
| `-holding_cost_rate × \|pos\|` | -0.0001 × position per bar |
| `compute_drawdown_penalty` | 0 to -5 (capped) when in drawdown |
| `reward_noise_scale × pseudo_noise` | ±5% of reward (label smoothing) |
| `commitment_lambda × ...` | -0.01 to -0.1 when position changes |
| `position_entropy_weight × entropy` | 0 to +0.02 at episode end |

The BIG non-P&L contributors are `urgency_credit_weight` (can exceed the
core signal on any positioned bar) and `risk_efficiency_weight` (asymmetric
amplifier of winners). These are prime candidates for the first pass of
deletion because their magnitudes can dominate the core signal.

## Honest recalibration of Phase 2 results

Before going further: the "5× improvement" and "20× MaxDD improvement"
framings in earlier commit messages were misleading. Absolute numbers:

- **Annualized Sharpe of 0** = break-even (doing nothing)
- **Sharpe 1** = decent; **Sharpe 2+** = genuinely good
- **Sharpe of -17** = still catastrophic — losing 17 standard deviations
  per year annualized. You'd blow the account.

The more honest per-bar metric:
- Before Phase 2: `Sharpe_raw` ≈ -0.39/bar (losing a LOT per bar)
- After Phase 2: `Sharpe_raw` ≈ -0.09/bar (losing a little per bar)

The improvement is real but **only relative**. Phase 2 deletions
revealed that the shaping terms were **actively creating losses**, not
masking a profitable policy. The underlying policy was only slightly
unprofitable, not wildly so. We stopped actively destroying capital.
We didn't find alpha.

### What q_gap=0.17 really tells us

q_gap is a **capacity metric** (the network CAN differentiate actions),
not a **profitability metric** (the actions it chose don't predict).
A capable-but-unprofitable network means the directional signal in the
42-dim feature vector isn't predictive enough at the training timescale
— a data/feature problem, not an algorithm problem.

### What remains to reach profitability (Sharpe > 0)

1. **TD propagation diagnostic** (Task #8) — verify the network can
   propagate sparse rewards back to trade-entry bars. If `cov(Q(s_entry,a),
   realized_trade_return) < 0.1`, the sparse-reward regime doesn't give
   enough gradient signal and no amount of reward cleanup fixes it.
2. **Unified env kernel** (Phase 3) — eliminate remaining train/val
   asymmetries (saboteur, fill model, reward composition differences).
3. **Possibly a different objective entirely** — pure P&L at HFT/MFT
   timescales may have SNR ≈ 0.01. Consider:
   - Longer horizon (trade-level rather than bar-level)
   - Ensemble of multi-horizon Q-networks
   - Different target (return quantiles, Sortino, path quality)
4. **Richer features** — 42 market + 20 OFI may be information-starved
   for the target alpha. Tick-level features, trade-flow imbalance,
   cross-market signals, calendar/event features.

### Honest headline

**Phase 2 has made validation measurable again.** The catastrophic
-150 Sharpe was uninformative noise; the -17 Sharpe is genuine signal
about the policy's real trading performance. That's infrastructure
progress — a cleaner diagnostic substrate — not profit.

Before celebrating future P&L improvements, sanity check: is the
improvement *crossing zero* (break-even) or just *less negative*? Less
negative is still losing.

## Gems, pearls, and novel moves

Reviewing the remover list for ideas worth preserving *in a different form*.
Each "behavioral" term was added to address a real phenomenon; deletion
without replacement regresses to the pathology the term was patching.

### Gems (preserve the underlying idea, relocate the mechanism)

#### 🔹 Probabilistic fill model

Training has saboteur-driven `fill_ioc_fill_prob ∈ [0.65, 0.95]` — 15%+ of
limit orders don't fill. Validation always fills deterministically. This
is **real-world execution physics**, not a shaping term. The asymmetry
explains part of the train/val Sharpe gap: training policies are robust to
missed fills; validation policies are brittle.

**Relocation**: In the unified env, probabilistic fills apply in BOTH modes
via a shared Philox RNG stream. Training uses the full adversarial range
(0.65–0.95); validation uses a single calibrated mean (e.g. 0.80). Neither
uses `1.0` — that would create a new mismatch in the opposite direction.

#### 🔹 Kelly as physics constraint, not reward

`kelly_sizing_weight` penalizes deviation from Kelly-optimal fraction.
Wrong level: rewards matching a formula instead of rewarding outcomes. But
Kelly IS a mathematically-correct cap on growth-optimal sizing.

**Relocation**: Delete the reward term. Add a hard position-size constraint
in `trade_physics.cuh`:

```c
float kelly_cap = kelly_fraction(win_stats) * max_position * safety_multiplier;
target_position = clampf(target_position, -kelly_cap, +kelly_cap);
```

The network can't over-lever because the environment physically won't let
it — not because the reward penalizes it. Same destination, different
layer of the stack.

#### 🔹 Path quality via per-step drawdown integration

`risk_efficiency_weight` rewards trades with low intra-trade drawdown —
i.e., path quality beyond final P&L. It double-counts asymmetrically
(only on winners), but the underlying signal is real.

**Relocation**: The time-integral of `compute_drawdown_penalty` IS path
quality, applied uniformly (not winners-only). Make sure the drawdown
penalty runs every bar, not just at trade completion. No separate reward
term needed — integration does the work.

#### 🔹 Saboteur noise relocated, not deleted

Saboteur (`spread_mult`, `fill_prob`, `slippage_mult` per-episode) made
the training policy robust. Validation had none of it, so training
learned strategies that rely on clean execution.

**Relocation**: Saboteur becomes a per-bar Philox-driven noise source
scaled by `exploration_scale ∈ [0, 1]` (pinned device-mapped scalar).
Training: `exploration_scale = 1.0` (full adversarial). Validation:
`exploration_scale = 0.5` (calibrated to realistic execution noise, NOT
zero). Sets a deliberate train/val gap that mirrors deployment-vs-backtest
conditions rather than hiding them.

### Pearls (subtle insights)

#### 💎 Sparse reward is fine if TD propagation works

The reason `micro_reward_scale` was added: trade-completion reward is
sparse (one reward per completed trade), so gradient signal is weak on
holding bars. But the Q-learning framework handles this via TD bootstrap
— IQN, C51, n-step exist specifically to propagate sparse signal across
time.

**Diagnostic before deleting**: verify whether Q-values for entry bars
respond to trade-completion rewards. If Q(entry) tracks the average of
subsequent trade outcomes, TD works — delete `micro_reward_scale`
unconditionally. If Q(entry) is decoupled from outcomes, the bug is in
n-step propagation (possibly `td_lambda = 0.9` being too low, or reward
clipping truncating the signal before TD sees it). Fix that, then delete.

#### 💎 Every behavioral term is one of four failure modes

| Category | Terms | Fix |
|---|---|---|
| Double-count of P&L-aligned terms | `order_credit`, `urgency_credit`, `risk_efficiency` | Just delete |
| Misplaced physics | `commitment_lambda`, `kelly_sizing`, `saboteur` (relocation) | Move to trade_physics / env constraints |
| Wrong-level regularization | `reward_noise_scale`, `position_entropy_weight` | Move to Q-target smoothing / diagnostic logging |
| Compensating for downstream bug | `micro_reward_scale` | Fix the downstream bug (TD propagation) |

No term is semantically "neutral." Each was a patch for a specific
pathology better solved elsewhere.

### Novel architectural moves

#### ⭐ Diagnostic-only entropy tracking

Don't reward entropy. But TRACK it. `H(dir_hist)` and `H(mag_hist)` per
epoch, logged alongside HEALTH_DIAG. When entropy drops below a
threshold, existing `LOW EXPOSURE DIVERSITY` warnings fire — elevate
those from warnings to principled collapse signals. Exploration bonuses
belong in ε-greedy / noisy-nets, not in the reward.

#### ⭐ Single `exploration_scale` scalar replaces all mode toggles

Unified env takes `exploration_scale: f32` via pinned device-mapped
memory (same pattern as distillation alpha — see
`2026-04-21-unified-train-val-env-design.md` §Option C):

- `1.0` → full training: saboteur on, ε-greedy at current ε,
  counterfactual flips at `cf_ratio`
- `0.5` → semi-realistic validation: saboteur half-strength, ε=0, no
  counterfactuals
- `0.0` → deterministic diagnostic: all adversarial off, pure argmax

One scalar, three behaviors. No boolean flags
(`feedback_no_feature_flags.md` compliant). Kernel reads it once per step
from the pinned pointer.

#### ⭐ Layered reward with per-term budget caps

```
reward = pnl_per_step                                  // core, always present
       + shaping_scale × clamp(0, DRAWDOWN_BUDGET,  drawdown_penalty)
       + shaping_scale × clamp(0, CHURN_BUDGET,     churn_penalty)
       + shaping_scale × clamp(0, HOLDING_BUDGET,   holding_cost)
```

Each shaping term has an absolute dollar-equivalent budget. Total shaping
contribution is bounded by `Σ budgets × shaping_scale`. The train/val
Sharpe gap upper bound becomes **computable**:
`gap_upper_bound = Σ budgets × shaping_scale × num_bars / σ(pnl_per_step)`.
Today that gap is uncomputable magic; tomorrow it's a scalar we can
budget against.

#### ⭐ Q-target smoothing replaces reward noise

`reward_noise_scale` adds uniform noise to the reward, changing the
objective. Better: add N(0, σ) noise to the Q-target distribution in the
C51 Bellman backward (`c51_loss_kernel.cu`). Same regularization effect
(prevents overfitting to exact target values) without perturbing what
the network believes "reward" means. Applied at gradient level, not
reward level.

## Updated disposition table (Phase 2 scope)

| Term | Disposition | Replacement |
|---|---|---|
| `order_credit_weight` | DELETE | Core tx_cost already captures real fill savings |
| `risk_efficiency_weight` | DELETE | Per-step drawdown integration |
| `urgency_credit_weight` | DELETE | Redundant with vol-normalized core return |
| `kelly_sizing_weight` | DELETE → physics | Kelly-fraction position cap in `trade_physics.cuh` |
| `micro_reward_scale` | DIAGNOSE → DELETE | Verify TD propagation first; fix `td_lambda`/n-step if broken |
| `commitment_lambda` | DELETE | Strengthen tx_cost slippage model instead |
| `reward_noise_scale` | MOVE | N(0,σ) on Q-target in C51 Bellman |
| `position_entropy_weight` | MOVE | Diagnostic log via HEALTH_DIAG |
| `saboteur_params` (training-only today) | RELOCATE | Apply in both modes scaled by `exploration_scale` |
| `w_dsr`, `exit_timing_weight`, `ofi_reward_weight`, `opp_cost_scale` | DELETE dead plumbing | N/A — already dormant |

## Next step (Phase 2)

Per the design doc:

> New file: `crates/ml/src/cuda_pipeline/unified_env_kernel.cu`. One
> `__global__ void unified_env_step(...)` that:
> - Takes the exploration/shaping scale scalars as `const float*` device
>   pointers (pinned device-mapped, same pattern as distillation alpha).
> - Computes trade physics once.
> - Computes reward = `pnl_per_step + shaping_scale × shaping_bundle`.
> - Writes `step_return = pnl_per_step` to a separate output buffer.

With Phase 1 done, Phase 2 has a concrete deletion list AND a concrete
relocation plan. The relocations (Kelly cap, saboteur scaling, Q-target
smoothing) can land as independent PRs before the unified kernel rewrite,
reducing Phase 2's change surface.

## Better-form taxonomy (where each deleted gem belongs)

Empirical observation from Phase 2 execution: every "v7 gem" reward term
was a good idea expressed at the wrong layer of the stack. Deletion alone
improved val Sharpe from -150 → -17 and val MaxDD from 15% → 0.27% (smoke
test). But the underlying intents are real and deserve proper homes.

The rule that emerged: **a gem becomes useful at the level of cause, not
effect.** Rewarding an outcome-adjacent behavior always underperforms
encoding the mechanism that would naturally produce that outcome.

| Deleted reward term | Intent | Correct layer | Status |
|---|---|---|---|
| `order_credit_weight` | Limit orders are cheaper | Physics: `compute_tx_cost(order_type_idx)` differentiates fills by order type | Already in physics; delete was pure redundancy |
| `risk_efficiency_weight` | Path quality matters (low intra-trade drawdown) | Reward: per-step drawdown penalty; time-integral gives path quality for free | Already in reward as per-step `compute_drawdown_penalty` |
| `urgency_credit_weight` | Favorable fills earn credit | Reward: already in `vol_normalized_return` | Pure redundancy — the "fill improvement" quantity IS unrealized P&L |
| `kelly_sizing_weight` | Size per Kelly | **Physics constraint** (`apply_kelly_cap` in trade_physics.cuh) | ✅ Done Phase 2, both envs |
| `micro_reward_scale` | Dense signal on sparse rewards | **Feature**, not reward: OFI/MBP-10 signals belong in the state vector so the Q-network can learn to use them. If density is needed, fix TD propagation (n-step, td_lambda, reward clipping), not reward shaping | ⏳ TD diagnostic pending (Task #8) |
| `commitment_lambda` | Position changes have cost | **Physics**: strengthen tx_cost slippage model (market-impact term proportional to position delta) | ✅ Already in `compute_tx_cost` (trade_physics.cuh:144–180) — Almgren-Chriss sqrt market-impact: `impact_scale = 1 + sqrt(\|delta\|/max_position)`. `commitment_lambda` was pure duplication of the existing impact term. |
| `reward_noise_scale` | Regularize Q-targets | **Q-target smoothing** at gradient level (`LABEL_SMOOTHING_BASE` in c51_loss_kernel.cu) | ✅ Done Phase 2, health-coupled |
| `position_entropy_weight` | Prevent policy narrowing | **Diagnostic signal** in HEALTH_DIAG + gate for existing `shrink_perturb` plasticity injection (D3/N3) | ✅ Done: `last_action_entropy` surfaced in HEALTH_DIAG; D3/N3 plasticity gate now fires on `entropy < 0.3` OR `health < 0.3` (training_loop.rs:1950) |
| `w_dsr` | Reward Sharpe directly | **Model-selection metric**, not objective. Track DSR EMA for HEALTH_DIAG / early-stopping / exploration gating (high DSR → exploit, low DSR → explore) | ⏳ Pending |
| `saboteur_params` | Robust to execution noise | **Shared fill model** via `exploration_scale ∈ [0, 1]` scalar in unified env. Training at 1.0, validation at 0.5 (calibrated deployment noise), neither at 0.0 | ⏳ Pending Phase 3 unified env |
| `exit_timing_weight` | Exits time the next move | **Dead** — the intent (avoid exits that leave money on the table) is structurally captured by trailing stops (already in trade_physics). The old implementation used `raw_next` (future price) which was lookahead bias. No future-proof reformulation exists. | ✅ Dead; delete with prejudice |
| `ofi_reward_weight` | OFI signals earn reward | **Dead** — already passed as feature through state[OFI_START..] (20-dim OFI block). The reward term was double-use | ✅ Dead |
| `opportunity_cost_scale` | Flat when Q-gap is wide is wasteful | **Feature**: Q-gap is already observable via state (the network computes it itself). If "don't be flat when you see edge" is a desirable policy, the Q-values will push for it naturally | ⏳ Consider deleting entirely in follow-up |

### Pattern — 4 better-form homes

1. **Physics layer** (trade_physics.cuh or env kernel): when the intent is a structural constraint on what can happen. Kelly cap, tx_cost, margin cap, capital floor, slippage model.

2. **Feature layer** (state vector): when the intent is "let the network observe X and decide." OFI, MBP-10, Q-gap components already observable via ISV.

3. **Diagnostic layer** (HEALTH_DIAG + existing health-triggered mechanisms): when the intent is monitoring/alerting. Position entropy, DSR, policy narrowing.

4. **Gradient-level regularization** (loss kernels): when the intent is preventing overfitting to exact targets. Label smoothing on Q-distribution, entropy regularization in policy gradients, etc. — NOT perturbing the reward itself.

Not "reward layer." The reward is `pnl_per_step` plus a minimal set of
P&L-aligned penalties (drawdown, churn, holding, tx_cost, capital floor).
Everything else is a category error.

### Implementation priority for the better forms

- **P1** (most impact): Relocate `w_dsr` → metric; relocate `position_entropy_weight` → diagnostic. Both prevent the information loss from deletion without reintroducing reward distortion.
- **P2** (useful once P1 done): Strengthen tx_cost with market-impact slippage to absorb what `commitment_lambda` was targeting. **(Already satisfied — Almgren-Chriss sqrt impact is in `compute_tx_cost` and active in both envs.)**
- **P3** (requires unified env — Phase 3): Relocate `saboteur_params` to both envs via `exploration_scale`.
- **Don't relocate**: `order_credit_weight`, `risk_efficiency_weight`, `urgency_credit_weight`, `exit_timing_weight`, `ofi_reward_weight`, `opportunity_cost_scale` — these are empirically redundant or structurally equivalent to existing physics/feature-layer expressions.

## Open questions for Phase 2

1. **TD propagation diagnostic**: before deleting `micro_reward_scale`, we
   need to verify Q-values at trade-entry bars respond to
   trade-completion rewards. Simplest test: train with sparse rewards
   only for 10 epochs; measure `cov(Q(s_entry, a), realized_trade_return)`.
   If > 0.3, TD is working. If < 0.1, fix TD first.

2. **Kelly priors**: current `kelly_sizing_weight` uses priors
   `win_rate ≈ 0.5, payoff_ratio ≈ 1.0` before any trades. For the
   position-cap version, need a conservative initial `kelly_cap` (e.g.
   0.25 × max_position) that loosens as real win/loss stats accumulate.
   Open: should the cap be per-episode or per-run? Per-run gives more
   data but crosses regime boundaries.

3. **Saboteur calibration for validation**: `exploration_scale = 0.5`
   picks "halfway adversarial" — but what's the RIGHT noise level for
   honest backtest? Calibrate against live broker execution stats if
   available; otherwise use a known-conservative midpoint.
