From 7b6641e038e2e7a97695b637abdbfccb36f69afc Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Tue, 21 Apr 2026 00:31:58 +0200 Subject: [PATCH] =?UTF-8?q?docs(design):=20phase-1=20inventory=20=E2=80=94?= =?UTF-8?q?=20gems/pearls/novels=20for=20env=20unification?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Extends the reward inventory with a review pass that preserves the *ideas* behind behavioral terms while deleting the wrong-level mechanics. Gems (relocate, don't delete): - Probabilistic fill model (real physics, not shaping) → apply in both modes via shared Philox RNG + exploration_scale - Kelly-as-physics-constraint → hard position-size cap in trade_physics, not a soft reward - Path quality via per-step drawdown integration → no separate term, just run drawdown penalty every bar - Saboteur relocation → both modes, scaled by exploration_scale (training 1.0, validation 0.5, diagnostic 0.0) — avoids creating a new clean-vs- noisy mismatch in the opposite direction Pearls: - Sparse reward is fine if TD propagation works — diagnose before deleting micro_reward_scale; measure cov(Q(s_entry,a), trade_return) - Every behavioral term maps to one of four failure modes: double-count, misplaced physics, wrong-level regularization, or compensating for a downstream bug. None are semantically "neutral" shaping. Novel architectural moves: - Diagnostic-only entropy tracking (no reward, just HEALTH_DIAG logging) - Single exploration_scale scalar replacing all mode toggles (feedback_no_feature_flags compliant) - Layered reward with per-term budget caps — makes the train/val Sharpe gap computable instead of uncomputable magic - Q-target smoothing replacing reward_noise_scale (regularization at gradient level, not reward level) Updated disposition table: 4 DELETE, 2 MOVE, 2 RELOCATE-to-physics, 1 DIAGNOSE-then-DELETE, 4 delete-dead-plumbing. Three relocations (Kelly cap, saboteur scaling, Q-target smoothing) can land as independent PRs before the unified_env_kernel rewrite, shrinking Phase 2's change surface. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../2026-04-21-phase1-reward-inventory.md | 184 +++++++++++++++++- 1 file changed, 182 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/specs/2026-04-21-phase1-reward-inventory.md b/docs/superpowers/specs/2026-04-21-phase1-reward-inventory.md index 52391cdf7..0d20351f2 100644 --- a/docs/superpowers/specs/2026-04-21-phase1-reward-inventory.md +++ b/docs/superpowers/specs/2026-04-21-phase1-reward-inventory.md @@ -100,6 +100,164 @@ core signal on any positioned bar) and `risk_efficiency_weight` (asymmetric amplifier of winners). These are prime candidates for the first pass of deletion because their magnitudes can dominate the core signal. +## Gems, pearls, and novel moves + +Reviewing the remover list for ideas worth preserving *in a different form*. +Each "behavioral" term was added to address a real phenomenon; deletion +without replacement regresses to the pathology the term was patching. + +### Gems (preserve the underlying idea, relocate the mechanism) + +#### 🔹 Probabilistic fill model + +Training has saboteur-driven `fill_ioc_fill_prob ∈ [0.65, 0.95]` — 15%+ of +limit orders don't fill. Validation always fills deterministically. This +is **real-world execution physics**, not a shaping term. The asymmetry +explains part of the train/val Sharpe gap: training policies are robust to +missed fills; validation policies are brittle. + +**Relocation**: In the unified env, probabilistic fills apply in BOTH modes +via a shared Philox RNG stream. Training uses the full adversarial range +(0.65–0.95); validation uses a single calibrated mean (e.g. 0.80). Neither +uses `1.0` — that would create a new mismatch in the opposite direction. + +#### 🔹 Kelly as physics constraint, not reward + +`kelly_sizing_weight` penalizes deviation from Kelly-optimal fraction. +Wrong level: rewards matching a formula instead of rewarding outcomes. But +Kelly IS a mathematically-correct cap on growth-optimal sizing. + +**Relocation**: Delete the reward term. Add a hard position-size constraint +in `trade_physics.cuh`: + +```c +float kelly_cap = kelly_fraction(win_stats) * max_position * safety_multiplier; +target_position = clampf(target_position, -kelly_cap, +kelly_cap); +``` + +The network can't over-lever because the environment physically won't let +it — not because the reward penalizes it. Same destination, different +layer of the stack. + +#### 🔹 Path quality via per-step drawdown integration + +`risk_efficiency_weight` rewards trades with low intra-trade drawdown — +i.e., path quality beyond final P&L. It double-counts asymmetrically +(only on winners), but the underlying signal is real. + +**Relocation**: The time-integral of `compute_drawdown_penalty` IS path +quality, applied uniformly (not winners-only). Make sure the drawdown +penalty runs every bar, not just at trade completion. No separate reward +term needed — integration does the work. + +#### 🔹 Saboteur noise relocated, not deleted + +Saboteur (`spread_mult`, `fill_prob`, `slippage_mult` per-episode) made +the training policy robust. Validation had none of it, so training +learned strategies that rely on clean execution. + +**Relocation**: Saboteur becomes a per-bar Philox-driven noise source +scaled by `exploration_scale ∈ [0, 1]` (pinned device-mapped scalar). +Training: `exploration_scale = 1.0` (full adversarial). Validation: +`exploration_scale = 0.5` (calibrated to realistic execution noise, NOT +zero). Sets a deliberate train/val gap that mirrors deployment-vs-backtest +conditions rather than hiding them. + +### Pearls (subtle insights) + +#### 💎 Sparse reward is fine if TD propagation works + +The reason `micro_reward_scale` was added: trade-completion reward is +sparse (one reward per completed trade), so gradient signal is weak on +holding bars. But the Q-learning framework handles this via TD bootstrap +— IQN, C51, n-step exist specifically to propagate sparse signal across +time. + +**Diagnostic before deleting**: verify whether Q-values for entry bars +respond to trade-completion rewards. If Q(entry) tracks the average of +subsequent trade outcomes, TD works — delete `micro_reward_scale` +unconditionally. If Q(entry) is decoupled from outcomes, the bug is in +n-step propagation (possibly `td_lambda = 0.9` being too low, or reward +clipping truncating the signal before TD sees it). Fix that, then delete. + +#### 💎 Every behavioral term is one of four failure modes + +| Category | Terms | Fix | +|---|---|---| +| Double-count of P&L-aligned terms | `order_credit`, `urgency_credit`, `risk_efficiency` | Just delete | +| Misplaced physics | `commitment_lambda`, `kelly_sizing`, `saboteur` (relocation) | Move to trade_physics / env constraints | +| Wrong-level regularization | `reward_noise_scale`, `position_entropy_weight` | Move to Q-target smoothing / diagnostic logging | +| Compensating for downstream bug | `micro_reward_scale` | Fix the downstream bug (TD propagation) | + +No term is semantically "neutral." Each was a patch for a specific +pathology better solved elsewhere. + +### Novel architectural moves + +#### ⭐ Diagnostic-only entropy tracking + +Don't reward entropy. But TRACK it. `H(dir_hist)` and `H(mag_hist)` per +epoch, logged alongside HEALTH_DIAG. When entropy drops below a +threshold, existing `LOW EXPOSURE DIVERSITY` warnings fire — elevate +those from warnings to principled collapse signals. Exploration bonuses +belong in ε-greedy / noisy-nets, not in the reward. + +#### ⭐ Single `exploration_scale` scalar replaces all mode toggles + +Unified env takes `exploration_scale: f32` via pinned device-mapped +memory (same pattern as distillation alpha — see +`2026-04-21-unified-train-val-env-design.md` §Option C): + +- `1.0` → full training: saboteur on, ε-greedy at current ε, + counterfactual flips at `cf_ratio` +- `0.5` → semi-realistic validation: saboteur half-strength, ε=0, no + counterfactuals +- `0.0` → deterministic diagnostic: all adversarial off, pure argmax + +One scalar, three behaviors. No boolean flags +(`feedback_no_feature_flags.md` compliant). Kernel reads it once per step +from the pinned pointer. + +#### ⭐ Layered reward with per-term budget caps + +``` +reward = pnl_per_step // core, always present + + shaping_scale × clamp(0, DRAWDOWN_BUDGET, drawdown_penalty) + + shaping_scale × clamp(0, CHURN_BUDGET, churn_penalty) + + shaping_scale × clamp(0, HOLDING_BUDGET, holding_cost) +``` + +Each shaping term has an absolute dollar-equivalent budget. Total shaping +contribution is bounded by `Σ budgets × shaping_scale`. The train/val +Sharpe gap upper bound becomes **computable**: +`gap_upper_bound = Σ budgets × shaping_scale × num_bars / σ(pnl_per_step)`. +Today that gap is uncomputable magic; tomorrow it's a scalar we can +budget against. + +#### ⭐ Q-target smoothing replaces reward noise + +`reward_noise_scale` adds uniform noise to the reward, changing the +objective. Better: add N(0, σ) noise to the Q-target distribution in the +C51 Bellman backward (`c51_loss_kernel.cu`). Same regularization effect +(prevents overfitting to exact target values) without perturbing what +the network believes "reward" means. Applied at gradient level, not +reward level. + +## Updated disposition table (Phase 2 scope) + +| Term | Disposition | Replacement | +|---|---|---| +| `order_credit_weight` | DELETE | Core tx_cost already captures real fill savings | +| `risk_efficiency_weight` | DELETE | Per-step drawdown integration | +| `urgency_credit_weight` | DELETE | Redundant with vol-normalized core return | +| `kelly_sizing_weight` | DELETE → physics | Kelly-fraction position cap in `trade_physics.cuh` | +| `micro_reward_scale` | DIAGNOSE → DELETE | Verify TD propagation first; fix `td_lambda`/n-step if broken | +| `commitment_lambda` | DELETE | Strengthen tx_cost slippage model instead | +| `reward_noise_scale` | MOVE | N(0,σ) on Q-target in C51 Bellman | +| `position_entropy_weight` | MOVE | Diagnostic log via HEALTH_DIAG | +| `saboteur_params` (training-only today) | RELOCATE | Apply in both modes scaled by `exploration_scale` | +| `w_dsr`, `exit_timing_weight`, `ofi_reward_weight`, `opp_cost_scale` | DELETE dead plumbing | N/A — already dormant | + ## Next step (Phase 2) Per the design doc: @@ -112,5 +270,27 @@ Per the design doc: > - Computes reward = `pnl_per_step + shaping_scale × shaping_bundle`. > - Writes `step_return = pnl_per_step` to a separate output buffer. -With Phase 1 done, Phase 2 has a concrete deletion list rather than a -judgment call. Safe to proceed. +With Phase 1 done, Phase 2 has a concrete deletion list AND a concrete +relocation plan. The relocations (Kelly cap, saboteur scaling, Q-target +smoothing) can land as independent PRs before the unified kernel rewrite, +reducing Phase 2's change surface. + +## Open questions for Phase 2 + +1. **TD propagation diagnostic**: before deleting `micro_reward_scale`, we + need to verify Q-values at trade-entry bars respond to + trade-completion rewards. Simplest test: train with sparse rewards + only for 10 epochs; measure `cov(Q(s_entry, a), realized_trade_return)`. + If > 0.3, TD is working. If < 0.1, fix TD first. + +2. **Kelly priors**: current `kelly_sizing_weight` uses priors + `win_rate ≈ 0.5, payoff_ratio ≈ 1.0` before any trades. For the + position-cap version, need a conservative initial `kelly_cap` (e.g. + 0.25 × max_position) that loosens as real win/loss stats accumulate. + Open: should the cap be per-episode or per-run? Per-run gives more + data but crosses regime boundaries. + +3. **Saboteur calibration for validation**: `exploration_scale = 0.5` + picks "halfway adversarial" — but what's the RIGHT noise level for + honest backtest? Calibrate against live broker execution stats if + available; otherwise use a known-conservative midpoint.