diff --git a/docs/superpowers/specs/2026-04-21-phase1-reward-inventory.md b/docs/superpowers/specs/2026-04-21-phase1-reward-inventory.md index 52391cdf7..0d20351f2 100644 --- a/docs/superpowers/specs/2026-04-21-phase1-reward-inventory.md +++ b/docs/superpowers/specs/2026-04-21-phase1-reward-inventory.md @@ -100,6 +100,164 @@ core signal on any positioned bar) and `risk_efficiency_weight` (asymmetric amplifier of winners). These are prime candidates for the first pass of deletion because their magnitudes can dominate the core signal. +## Gems, pearls, and novel moves + +Reviewing the remover list for ideas worth preserving *in a different form*. +Each "behavioral" term was added to address a real phenomenon; deletion +without replacement regresses to the pathology the term was patching. + +### Gems (preserve the underlying idea, relocate the mechanism) + +#### πŸ”Ή Probabilistic fill model + +Training has saboteur-driven `fill_ioc_fill_prob ∈ [0.65, 0.95]` β€” 15%+ of +limit orders don't fill. Validation always fills deterministically. This +is **real-world execution physics**, not a shaping term. The asymmetry +explains part of the train/val Sharpe gap: training policies are robust to +missed fills; validation policies are brittle. + +**Relocation**: In the unified env, probabilistic fills apply in BOTH modes +via a shared Philox RNG stream. Training uses the full adversarial range +(0.65–0.95); validation uses a single calibrated mean (e.g. 0.80). Neither +uses `1.0` β€” that would create a new mismatch in the opposite direction. + +#### πŸ”Ή Kelly as physics constraint, not reward + +`kelly_sizing_weight` penalizes deviation from Kelly-optimal fraction. +Wrong level: rewards matching a formula instead of rewarding outcomes. But +Kelly IS a mathematically-correct cap on growth-optimal sizing. + +**Relocation**: Delete the reward term. Add a hard position-size constraint +in `trade_physics.cuh`: + +```c +float kelly_cap = kelly_fraction(win_stats) * max_position * safety_multiplier; +target_position = clampf(target_position, -kelly_cap, +kelly_cap); +``` + +The network can't over-lever because the environment physically won't let +it β€” not because the reward penalizes it. Same destination, different +layer of the stack. + +#### πŸ”Ή Path quality via per-step drawdown integration + +`risk_efficiency_weight` rewards trades with low intra-trade drawdown β€” +i.e., path quality beyond final P&L. It double-counts asymmetrically +(only on winners), but the underlying signal is real. + +**Relocation**: The time-integral of `compute_drawdown_penalty` IS path +quality, applied uniformly (not winners-only). Make sure the drawdown +penalty runs every bar, not just at trade completion. No separate reward +term needed β€” integration does the work. + +#### πŸ”Ή Saboteur noise relocated, not deleted + +Saboteur (`spread_mult`, `fill_prob`, `slippage_mult` per-episode) made +the training policy robust. Validation had none of it, so training +learned strategies that rely on clean execution. + +**Relocation**: Saboteur becomes a per-bar Philox-driven noise source +scaled by `exploration_scale ∈ [0, 1]` (pinned device-mapped scalar). +Training: `exploration_scale = 1.0` (full adversarial). Validation: +`exploration_scale = 0.5` (calibrated to realistic execution noise, NOT +zero). Sets a deliberate train/val gap that mirrors deployment-vs-backtest +conditions rather than hiding them. + +### Pearls (subtle insights) + +#### πŸ’Ž Sparse reward is fine if TD propagation works + +The reason `micro_reward_scale` was added: trade-completion reward is +sparse (one reward per completed trade), so gradient signal is weak on +holding bars. But the Q-learning framework handles this via TD bootstrap +β€” IQN, C51, n-step exist specifically to propagate sparse signal across +time. + +**Diagnostic before deleting**: verify whether Q-values for entry bars +respond to trade-completion rewards. If Q(entry) tracks the average of +subsequent trade outcomes, TD works β€” delete `micro_reward_scale` +unconditionally. If Q(entry) is decoupled from outcomes, the bug is in +n-step propagation (possibly `td_lambda = 0.9` being too low, or reward +clipping truncating the signal before TD sees it). Fix that, then delete. + +#### πŸ’Ž Every behavioral term is one of four failure modes + +| Category | Terms | Fix | +|---|---|---| +| Double-count of P&L-aligned terms | `order_credit`, `urgency_credit`, `risk_efficiency` | Just delete | +| Misplaced physics | `commitment_lambda`, `kelly_sizing`, `saboteur` (relocation) | Move to trade_physics / env constraints | +| Wrong-level regularization | `reward_noise_scale`, `position_entropy_weight` | Move to Q-target smoothing / diagnostic logging | +| Compensating for downstream bug | `micro_reward_scale` | Fix the downstream bug (TD propagation) | + +No term is semantically "neutral." Each was a patch for a specific +pathology better solved elsewhere. + +### Novel architectural moves + +#### ⭐ Diagnostic-only entropy tracking + +Don't reward entropy. But TRACK it. `H(dir_hist)` and `H(mag_hist)` per +epoch, logged alongside HEALTH_DIAG. When entropy drops below a +threshold, existing `LOW EXPOSURE DIVERSITY` warnings fire β€” elevate +those from warnings to principled collapse signals. Exploration bonuses +belong in Ξ΅-greedy / noisy-nets, not in the reward. + +#### ⭐ Single `exploration_scale` scalar replaces all mode toggles + +Unified env takes `exploration_scale: f32` via pinned device-mapped +memory (same pattern as distillation alpha β€” see +`2026-04-21-unified-train-val-env-design.md` Β§Option C): + +- `1.0` β†’ full training: saboteur on, Ξ΅-greedy at current Ξ΅, + counterfactual flips at `cf_ratio` +- `0.5` β†’ semi-realistic validation: saboteur half-strength, Ξ΅=0, no + counterfactuals +- `0.0` β†’ deterministic diagnostic: all adversarial off, pure argmax + +One scalar, three behaviors. No boolean flags +(`feedback_no_feature_flags.md` compliant). Kernel reads it once per step +from the pinned pointer. + +#### ⭐ Layered reward with per-term budget caps + +``` +reward = pnl_per_step // core, always present + + shaping_scale Γ— clamp(0, DRAWDOWN_BUDGET, drawdown_penalty) + + shaping_scale Γ— clamp(0, CHURN_BUDGET, churn_penalty) + + shaping_scale Γ— clamp(0, HOLDING_BUDGET, holding_cost) +``` + +Each shaping term has an absolute dollar-equivalent budget. Total shaping +contribution is bounded by `Ξ£ budgets Γ— shaping_scale`. The train/val +Sharpe gap upper bound becomes **computable**: +`gap_upper_bound = Ξ£ budgets Γ— shaping_scale Γ— num_bars / Οƒ(pnl_per_step)`. +Today that gap is uncomputable magic; tomorrow it's a scalar we can +budget against. + +#### ⭐ Q-target smoothing replaces reward noise + +`reward_noise_scale` adds uniform noise to the reward, changing the +objective. Better: add N(0, Οƒ) noise to the Q-target distribution in the +C51 Bellman backward (`c51_loss_kernel.cu`). Same regularization effect +(prevents overfitting to exact target values) without perturbing what +the network believes "reward" means. Applied at gradient level, not +reward level. + +## Updated disposition table (Phase 2 scope) + +| Term | Disposition | Replacement | +|---|---|---| +| `order_credit_weight` | DELETE | Core tx_cost already captures real fill savings | +| `risk_efficiency_weight` | DELETE | Per-step drawdown integration | +| `urgency_credit_weight` | DELETE | Redundant with vol-normalized core return | +| `kelly_sizing_weight` | DELETE β†’ physics | Kelly-fraction position cap in `trade_physics.cuh` | +| `micro_reward_scale` | DIAGNOSE β†’ DELETE | Verify TD propagation first; fix `td_lambda`/n-step if broken | +| `commitment_lambda` | DELETE | Strengthen tx_cost slippage model instead | +| `reward_noise_scale` | MOVE | N(0,Οƒ) on Q-target in C51 Bellman | +| `position_entropy_weight` | MOVE | Diagnostic log via HEALTH_DIAG | +| `saboteur_params` (training-only today) | RELOCATE | Apply in both modes scaled by `exploration_scale` | +| `w_dsr`, `exit_timing_weight`, `ofi_reward_weight`, `opp_cost_scale` | DELETE dead plumbing | N/A β€” already dormant | + ## Next step (Phase 2) Per the design doc: @@ -112,5 +270,27 @@ Per the design doc: > - Computes reward = `pnl_per_step + shaping_scale Γ— shaping_bundle`. > - Writes `step_return = pnl_per_step` to a separate output buffer. -With Phase 1 done, Phase 2 has a concrete deletion list rather than a -judgment call. Safe to proceed. +With Phase 1 done, Phase 2 has a concrete deletion list AND a concrete +relocation plan. The relocations (Kelly cap, saboteur scaling, Q-target +smoothing) can land as independent PRs before the unified kernel rewrite, +reducing Phase 2's change surface. + +## Open questions for Phase 2 + +1. **TD propagation diagnostic**: before deleting `micro_reward_scale`, we + need to verify Q-values at trade-entry bars respond to + trade-completion rewards. Simplest test: train with sparse rewards + only for 10 epochs; measure `cov(Q(s_entry, a), realized_trade_return)`. + If > 0.3, TD is working. If < 0.1, fix TD first. + +2. **Kelly priors**: current `kelly_sizing_weight` uses priors + `win_rate β‰ˆ 0.5, payoff_ratio β‰ˆ 1.0` before any trades. For the + position-cap version, need a conservative initial `kelly_cap` (e.g. + 0.25 Γ— max_position) that loosens as real win/loss stats accumulate. + Open: should the cap be per-episode or per-run? Per-run gives more + data but crosses regime boundaries. + +3. **Saboteur calibration for validation**: `exploration_scale = 0.5` + picks "halfway adversarial" β€” but what's the RIGHT noise level for + honest backtest? Calibrate against live broker execution stats if + available; otherwise use a known-conservative midpoint.