docs(design): phase-1 reward inventory for env unification

Enumerates every term that contributes to training or validation reward,
classifies each as P&L-aligned (keeper) or behavioral (remover), with
source line references into experience_kernels.cu and backtest_env_kernel.cu.

Findings:
- 7 P&L-aligned terms to preserve in the unified env (core return,
  tx_cost, drawdown, inventory, churn, capital floor, + vol normalization)
- 8 active behavioral training-only terms to delete:
  * order_credit_weight  (rewards theoretical limit-order savings,
    double-counts actual tx_cost)
  * risk_efficiency_weight  (double-counts drawdown asymmetrically)
  * urgency_credit_weight  (redundant with core return)
  * kelly_sizing_weight  (rewards matching a formula, not outcomes)
  * micro_reward_scale  (OFI-momentum signal follower)
  * commitment_lambda  (triple-counts churn)
  * reward_noise_scale  (belongs at gradient level, not reward level)
  * position_entropy_weight  (changes what "optimal" means)
- 4 dormant/dead terms to clean up (w_dsr, exit_timing, ofi_reward,
  opp_cost_scale — all plumbing with no kernel effect)

Biggest offenders by magnitude: urgency_credit and risk_efficiency can
dominate the core signal on a single positioned bar.

Phase 2 (unified_env_kernel implementation) can now proceed with a
concrete deletion list, not a judgment call.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

docs/superpowers/specs/2026-04-21-phase1-reward-inventory.md

@@ -0,0 +1,116 @@
+# Phase 1 — Reward Term Inventory
+
+Companion to `2026-04-21-unified-train-val-env-design.md`. Enumerates every
+term that contributes to the training or validation reward, classifies each
+as P&L-aligned (keeper) or behavioral (remover), and records the location
+in source. Evidence from `experience_kernels.cu` line numbers as of merge
+commit `9223795c7`.
+
+## Core P&L-aligned terms (present in both envs — keepers)
+
+These map directly to the equity change a real trading account would
+experience. Removing any of them would make the reward diverge from real P&L.
+
+| Term | Training location | Validation location | Semantics |
+|---|---|---|---|
+| Core trade return | `experience_kernels.cu:1687` (`2.0 × vol_normalized_return`) | `backtest_env_kernel.cu:251` (`step_ret = ΔV/V`) | Realized equity change |
+| Transaction cost | `trade_physics.cuh:execute_trade` | Same (shared) | Broker fees + spread |
+| Drawdown penalty | `experience_kernels.cu:1885` (`compute_drawdown_penalty`) | Implicit via capital floor | Pushback when equity < peak × (1 - dd_threshold) |
+| Inventory/holding cost | `experience_kernels.cu:1839` (`-holding_cost_rate × \|pos\|`) | `backtest_env_kernel.cu:257` (same) | Continuous carry cost of holding position |
+| Churn penalty | Via commitment_lambda (see below) | `backtest_env_kernel.cu:266` (`churn_penalty_scale`) | Pushback on rapid position flips |
+| Capital floor | `experience_kernels.cu:1924` (`reward = -5 - 5×risk_taken`) | `backtest_env_kernel.cu:122-139` (liquidation) | Catastrophic penalty when equity breaks the 25% drawdown floor |
+
+These are semantically consistent across envs (though magnitudes differ —
+training uses larger absolute penalties because of vol normalization). The
+unified env should preserve these with identical numerics.
+
+## Training-only behavioral terms (removers)
+
+Each of these rewards a *decision-making behavior* rather than a *realized
+outcome*. They were added to address specific training pathologies but
+break the design goal that training reward should equal realized P&L.
+
+### Sparse (trade-completion) behavioral terms
+
+| Term | Location | Formula | Why it's behavioral |
+|---|---|---|---|
+| `order_credit_weight` | `experience_kernels.cu:1700` | `+= w × (market_cost - taken_cost) × 100 / equity` | Rewards choosing limit-over-market order types regardless of whether the limit actually got a better fill (assumes theoretical savings). Real fills are already captured in actual `tx_cost` via `compute_tx_cost(position, raw_close, ..., order_type_idx, ...)`. |
+| `risk_efficiency_weight` | `experience_kernels.cu:1706` | `+= w × segment_return / \|intra_trade_max_dd\|` | Rewards trades with low path-drawdown. The drawdown penalty already exists as a separate P&L-aligned term — this double-counts but asymmetrically (only on winners). |
+| `urgency_credit_weight` | `experience_kernels.cu:1717` | `+= w × fill_improvement / vol_proxy` | Rewards favorable fill vs entry price. But the fill price IS the entry price — `fill_improvement = (raw_close - entry_price)/entry_price` is just an alias for the unrealized P&L at the moment of evaluation. Redundant with the core return. |
+| `kelly_sizing_weight` | `experience_kernels.cu:1746` | `-= w × \|actual_fraction - kelly_f\| × 0.5` | Penalizes deviation from Kelly-optimal fraction computed with priors over win/loss statistics. Kelly is a derived quantity — rewarding for matching it is rewarding the model for matching a formula, not for making money. |
+
+### Dense (per-bar) behavioral terms
+
+| Term | Location | Formula | Why it's behavioral |
+|---|---|---|---|
+| `micro_reward_scale` | `experience_kernels.cu:1836` | `reward = w × tanh((flow_momentum + price_confirm + book_aggression + hold_quality - spread_penalty) / temp)` | Rewards alignment with OFI momentum signals regardless of realized P&L. The network can score well on this by following the signal dogmatically even when it's misleading. |
+| `commitment_lambda` | `experience_kernels.cu:1890-1896` | `-= λ × (1 - exp(-\|Δpos\| / τ)) × cost_scale` | "Action commitment penalty." Discourages oscillation. But transaction cost already penalizes position changes, and churn penalty already handles rapid flips. Triple-counting. |
+
+### Always-applied behavioral terms
+
+| Term | Location | Formula | Why it's behavioral |
+|---|---|---|---|
+| `reward_noise_scale` | `experience_kernels.cu:1874-1881` | `reward += noise_magnitude × pseudo_noise` (label smoothing) | Regularization proxy. Inserts noise into the reward signal. Useful for preventing overfitting, but should be applied at the gradient/target level (e.g., Q-target smoothing), not at the reward level where it changes the objective. |
+| `position_entropy_weight` | `experience_kernels.cu:1976` | `+= w × combined_entropy` at episode end | Rewards exploration across action buckets. Exploration is already handled by ε-greedy + noisy nets; rewarding it in the objective changes what "optimal" means. |
+
+### Dormant / already-removed behavioral terms
+
+These appear in the config / plumbing but are no longer added to the
+reward. Dead weight:
+
+| Term | Status |
+|---|---|
+| `w_dsr` (Differential Sharpe Ratio) | EMA is tracked (`experience_kernels.cu:1868-1870`) but no longer added to reward. Dead plumbing. |
+| `exit_timing_weight` | Commented out (experience_kernels.cu:1724 says "REMOVED: post_exit_move used raw_next"). Dead plumbing. |
+| `ofi_reward_weight` | Not actually applied inside the kernel — replaced by `micro_reward_scale` path. Dead. |
+
+## Validation-only behavioral terms
+
+| Term | Status |
+|---|---|
+| `opp_cost_scale` | Multiplied by `0.0f × is_flat` at `backtest_env_kernel.cu:275` — always zero. Dead. |
+
+## Summary
+
+- **7 P&L-aligned terms** to preserve in the unified env
+- **8 active behavioral training-only terms** to delete
+- **4 dormant/dead terms** to clean up (including cleanup of their config
+  fields and kernel args)
+
+## Magnitude comparison (approximate per-bar contribution)
+
+With production-config weights (`w_dsr=5.0`, `order_credit=1.0`, etc.)
+measured from the 50-epoch smoke test:
+
+| Term | Magnitude per bar |
+|---|---|
+| Core vol-normalized return | ±0.05 to ±0.5 (bulk of signal) |
+| `order_credit_weight × ...` | +0.01 to +0.2 when trade completes (~5% of bars) |
+| `risk_efficiency_weight × ...` | 0 to +0.5 on winners only |
+| `urgency_credit_weight × ...` | 0 to +1.0 on any positioned bar with entry |
+| `micro_reward_scale × tanh(...)` | 0 (disabled: `micro_reward_scale=0.0` in smoke config) |
+| `-holding_cost_rate × \|pos\|` | -0.0001 × position per bar |
+| `compute_drawdown_penalty` | 0 to -5 (capped) when in drawdown |
+| `reward_noise_scale × pseudo_noise` | ±5% of reward (label smoothing) |
+| `commitment_lambda × ...` | -0.01 to -0.1 when position changes |
+| `position_entropy_weight × entropy` | 0 to +0.02 at episode end |
+
+The BIG non-P&L contributors are `urgency_credit_weight` (can exceed the
+core signal on any positioned bar) and `risk_efficiency_weight` (asymmetric
+amplifier of winners). These are prime candidates for the first pass of
+deletion because their magnitudes can dominate the core signal.
+
+## Next step (Phase 2)
+
+Per the design doc:
+
+> New file: `crates/ml/src/cuda_pipeline/unified_env_kernel.cu`. One
+> `__global__ void unified_env_step(...)` that:
+> - Takes the exploration/shaping scale scalars as `const float*` device
+>   pointers (pinned device-mapped, same pattern as distillation alpha).
+> - Computes trade physics once.
+> - Computes reward = `pnl_per_step + shaping_scale × shaping_bundle`.
+> - Writes `step_return = pnl_per_step` to a separate output buffer.
+
+With Phase 1 done, Phase 2 has a concrete deletion list rather than a
+judgment call. Safe to proceed.