diff --git a/docs/superpowers/specs/2026-04-21-phase1-reward-inventory.md b/docs/superpowers/specs/2026-04-21-phase1-reward-inventory.md new file mode 100644 index 000000000..52391cdf7 --- /dev/null +++ b/docs/superpowers/specs/2026-04-21-phase1-reward-inventory.md @@ -0,0 +1,116 @@ +# Phase 1 — Reward Term Inventory + +Companion to `2026-04-21-unified-train-val-env-design.md`. Enumerates every +term that contributes to the training or validation reward, classifies each +as P&L-aligned (keeper) or behavioral (remover), and records the location +in source. Evidence from `experience_kernels.cu` line numbers as of merge +commit `9223795c7`. + +## Core P&L-aligned terms (present in both envs — keepers) + +These map directly to the equity change a real trading account would +experience. Removing any of them would make the reward diverge from real P&L. + +| Term | Training location | Validation location | Semantics | +|---|---|---|---| +| Core trade return | `experience_kernels.cu:1687` (`2.0 × vol_normalized_return`) | `backtest_env_kernel.cu:251` (`step_ret = ΔV/V`) | Realized equity change | +| Transaction cost | `trade_physics.cuh:execute_trade` | Same (shared) | Broker fees + spread | +| Drawdown penalty | `experience_kernels.cu:1885` (`compute_drawdown_penalty`) | Implicit via capital floor | Pushback when equity < peak × (1 - dd_threshold) | +| Inventory/holding cost | `experience_kernels.cu:1839` (`-holding_cost_rate × \|pos\|`) | `backtest_env_kernel.cu:257` (same) | Continuous carry cost of holding position | +| Churn penalty | Via commitment_lambda (see below) | `backtest_env_kernel.cu:266` (`churn_penalty_scale`) | Pushback on rapid position flips | +| Capital floor | `experience_kernels.cu:1924` (`reward = -5 - 5×risk_taken`) | `backtest_env_kernel.cu:122-139` (liquidation) | Catastrophic penalty when equity breaks the 25% drawdown floor | + +These are semantically consistent across envs (though magnitudes differ — +training uses larger absolute penalties because of vol normalization). The +unified env should preserve these with identical numerics. + +## Training-only behavioral terms (removers) + +Each of these rewards a *decision-making behavior* rather than a *realized +outcome*. They were added to address specific training pathologies but +break the design goal that training reward should equal realized P&L. + +### Sparse (trade-completion) behavioral terms + +| Term | Location | Formula | Why it's behavioral | +|---|---|---|---| +| `order_credit_weight` | `experience_kernels.cu:1700` | `+= w × (market_cost - taken_cost) × 100 / equity` | Rewards choosing limit-over-market order types regardless of whether the limit actually got a better fill (assumes theoretical savings). Real fills are already captured in actual `tx_cost` via `compute_tx_cost(position, raw_close, ..., order_type_idx, ...)`. | +| `risk_efficiency_weight` | `experience_kernels.cu:1706` | `+= w × segment_return / \|intra_trade_max_dd\|` | Rewards trades with low path-drawdown. The drawdown penalty already exists as a separate P&L-aligned term — this double-counts but asymmetrically (only on winners). | +| `urgency_credit_weight` | `experience_kernels.cu:1717` | `+= w × fill_improvement / vol_proxy` | Rewards favorable fill vs entry price. But the fill price IS the entry price — `fill_improvement = (raw_close - entry_price)/entry_price` is just an alias for the unrealized P&L at the moment of evaluation. Redundant with the core return. | +| `kelly_sizing_weight` | `experience_kernels.cu:1746` | `-= w × \|actual_fraction - kelly_f\| × 0.5` | Penalizes deviation from Kelly-optimal fraction computed with priors over win/loss statistics. Kelly is a derived quantity — rewarding for matching it is rewarding the model for matching a formula, not for making money. | + +### Dense (per-bar) behavioral terms + +| Term | Location | Formula | Why it's behavioral | +|---|---|---|---| +| `micro_reward_scale` | `experience_kernels.cu:1836` | `reward = w × tanh((flow_momentum + price_confirm + book_aggression + hold_quality - spread_penalty) / temp)` | Rewards alignment with OFI momentum signals regardless of realized P&L. The network can score well on this by following the signal dogmatically even when it's misleading. | +| `commitment_lambda` | `experience_kernels.cu:1890-1896` | `-= λ × (1 - exp(-\|Δpos\| / τ)) × cost_scale` | "Action commitment penalty." Discourages oscillation. But transaction cost already penalizes position changes, and churn penalty already handles rapid flips. Triple-counting. | + +### Always-applied behavioral terms + +| Term | Location | Formula | Why it's behavioral | +|---|---|---|---| +| `reward_noise_scale` | `experience_kernels.cu:1874-1881` | `reward += noise_magnitude × pseudo_noise` (label smoothing) | Regularization proxy. Inserts noise into the reward signal. Useful for preventing overfitting, but should be applied at the gradient/target level (e.g., Q-target smoothing), not at the reward level where it changes the objective. | +| `position_entropy_weight` | `experience_kernels.cu:1976` | `+= w × combined_entropy` at episode end | Rewards exploration across action buckets. Exploration is already handled by ε-greedy + noisy nets; rewarding it in the objective changes what "optimal" means. | + +### Dormant / already-removed behavioral terms + +These appear in the config / plumbing but are no longer added to the +reward. Dead weight: + +| Term | Status | +|---|---| +| `w_dsr` (Differential Sharpe Ratio) | EMA is tracked (`experience_kernels.cu:1868-1870`) but no longer added to reward. Dead plumbing. | +| `exit_timing_weight` | Commented out (experience_kernels.cu:1724 says "REMOVED: post_exit_move used raw_next"). Dead plumbing. | +| `ofi_reward_weight` | Not actually applied inside the kernel — replaced by `micro_reward_scale` path. Dead. | + +## Validation-only behavioral terms + +| Term | Status | +|---|---| +| `opp_cost_scale` | Multiplied by `0.0f × is_flat` at `backtest_env_kernel.cu:275` — always zero. Dead. | + +## Summary + +- **7 P&L-aligned terms** to preserve in the unified env +- **8 active behavioral training-only terms** to delete +- **4 dormant/dead terms** to clean up (including cleanup of their config + fields and kernel args) + +## Magnitude comparison (approximate per-bar contribution) + +With production-config weights (`w_dsr=5.0`, `order_credit=1.0`, etc.) +measured from the 50-epoch smoke test: + +| Term | Magnitude per bar | +|---|---| +| Core vol-normalized return | ±0.05 to ±0.5 (bulk of signal) | +| `order_credit_weight × ...` | +0.01 to +0.2 when trade completes (~5% of bars) | +| `risk_efficiency_weight × ...` | 0 to +0.5 on winners only | +| `urgency_credit_weight × ...` | 0 to +1.0 on any positioned bar with entry | +| `micro_reward_scale × tanh(...)` | 0 (disabled: `micro_reward_scale=0.0` in smoke config) | +| `-holding_cost_rate × \|pos\|` | -0.0001 × position per bar | +| `compute_drawdown_penalty` | 0 to -5 (capped) when in drawdown | +| `reward_noise_scale × pseudo_noise` | ±5% of reward (label smoothing) | +| `commitment_lambda × ...` | -0.01 to -0.1 when position changes | +| `position_entropy_weight × entropy` | 0 to +0.02 at episode end | + +The BIG non-P&L contributors are `urgency_credit_weight` (can exceed the +core signal on any positioned bar) and `risk_efficiency_weight` (asymmetric +amplifier of winners). These are prime candidates for the first pass of +deletion because their magnitudes can dominate the core signal. + +## Next step (Phase 2) + +Per the design doc: + +> New file: `crates/ml/src/cuda_pipeline/unified_env_kernel.cu`. One +> `__global__ void unified_env_step(...)` that: +> - Takes the exploration/shaping scale scalars as `const float*` device +> pointers (pinned device-mapped, same pattern as distillation alpha). +> - Computes trade physics once. +> - Computes reward = `pnl_per_step + shaping_scale × shaping_bundle`. +> - Writes `step_return = pnl_per_step` to a separate output buffer. + +With Phase 1 done, Phase 2 has a concrete deletion list rather than a +judgment call. Safe to proceed.