diff --git a/docs/superpowers/specs/2026-05-09-sp19-20-wr-first-design.md b/docs/superpowers/specs/2026-05-09-sp19-20-wr-first-design.md index 2e8bfed49..7c16033c2 100644 --- a/docs/superpowers/specs/2026-05-09-sp19-20-wr-first-design.md +++ b/docs/superpowers/specs/2026-05-09-sp19-20-wr-first-design.md @@ -185,13 +185,24 @@ elif hold_pct_ema < target_hold_pct - 0.05: **Buffer sizing:** `hold_baseline_buffer` is a circular buffer of size `LOOKAHEAD_HORIZON_MAX = 30` bars (matches SP19's longest horizon and bounds max trade duration). Bars older than 30 are overwritten — fine, since trades > 30 bars don't sample beyond their last 30 bars. -**Q-scale design note (intentional asymmetry between Hold and trade rewards):** Hold per-bar reward is **uncentered** (raw `-aux_conf × cost_scale`), whereas trade reward is centered via `R_used = alpha - alpha_ema`. This asymmetry is **deliberate**: it produces a marginal Q(trade) > Q(Hold) preference (Q(Hold) accumulates a small negative drift, Q(trade) is zero-mean), which counteracts the Q(Hold) attractor we are trying to break. Centering Hold rewards too would symmetrize the bias and remove this engineered preference. Behavioral test `sp20_pure_noise` validates the Hold attractor is still correctly induced when there is no signal — so the marginal trade preference doesn't override the no-signal Hold default. +**Q-scale comparability — dual emission with centering:** Both Hold and trade rewards are centered via long-run EMA subtraction so Q(Hold) and Q(trade) are scale-comparable. Hold's `R_per_bar_centered` writes to the replay tuple for Q-target consumption; the **uncentered** `R_per_bar` writes to `hold_baseline_buffer` for Component 1's structural baseline subtraction. The two emissions serve different purposes — centered for Q-target stability, uncentered for "what would Hold have paid" baseline. + +``` +hold_reward_ema_t+1 ← (1-α) × hold_reward_ema_t + α × R_per_bar_i # Wiener-α floor 0.4 + # over Hold-state bars only + +R_per_bar_centered_i = R_per_bar_i - hold_reward_ema_t # for Q-target / replay tuple + +hold_baseline_buffer[i] = R_per_bar_i # uncentered, for Component 1 +``` + +The marginal Q(trade) > Q(Hold) preference now comes from the **data** rather than from structural scale asymmetry: in high-aux-confidence states, Hold's variance pulls Q(Hold) more negative (when aux is right); in low-aux states, Q(Hold) is near-zero and Q(trade) is near-zero, model has no preference. This is cleaner than scale-asymmetric drift and robust as cost_scale evolves. **Implementation:** - 1 new producer kernel `hold_cost_scale_compute_kernel.cu` (~50 LoC, follows SP4 Pearls A+D) - Modification at `experience_kernels.cu` per-bar reward path: add `R_per_bar_i` term, write to `hold_baseline_buffer` -**ISV slots (3 new):** `HOLD_COST_SCALE_INDEX`, `TARGET_HOLD_PCT_INDEX`, `HOLD_PCT_EMA_INDEX`. +**ISV slots (4 new):** `HOLD_COST_SCALE_INDEX`, `TARGET_HOLD_PCT_INDEX`, `HOLD_PCT_EMA_INDEX`, `HOLD_REWARD_EMA_INDEX`. ### 4.3 Component 3 — n-step credit distributor @@ -283,6 +294,7 @@ Produces (Wiener-α EMAs with floor 0.4): - `aux_dir_acc_ema` — per-step (extends existing SP14-C signal) - `hold_pct_ema` — per-step - `alpha_ema` — per-trade +- `hold_reward_ema` — per-step on Hold-state bars only (for Q-scale comparability) **Kernel 2: `sp20_controllers_compute`** (~70 LoC, depends on Kernel 1) @@ -301,7 +313,7 @@ Consumes: aux_logits raw. Produces: `aux_conf_p50`, `aux_conf_std` (used by Kernel 1). Uses `sp4_histogram_p99.cuh` pattern for p50 (median, k=128). -**Total new ISV slots: 9 in range [510..520).** +**Total new ISV slots: 10 in range [510..520).** ### 4.6 Component 6 — Behavioral Test Suite @@ -332,9 +344,12 @@ All reuse SP15 Phase 2A LobBar harness. ~250 LoC test code. Each runs <30s on RT [Step] trade_physics(a_t) → step_return, is_close, position state per-bar reward path: - if action == Hold: R_per_bar = -aux_conf × cost_scale (Component 2) - else: R_per_bar = 0 - hold_baseline_buffer[t] = -aux_conf × cost_scale (always, for Component 1) + if action == Hold: + R_per_bar = -aux_conf × cost_scale # raw, for hold_reward_ema update + R_per_bar_centered = R_per_bar - hold_reward_ema # for Q-target via replay tuple + else: + R_per_bar_centered = 0 + hold_baseline_buffer[t] = -aux_conf × cost_scale # uncentered, for Component 1 [Trade close detected] R_event = 4-quadrant reward (Component 1) @@ -385,7 +400,8 @@ Approximate sizing per component (atomic ship per `feedback_no_partial_refactor` | Component | Files modified | New files | Approx LoC | |---|---|---|---| | 1: Reward kernel | `experience_kernels.cu`, `gpu_experience_collector.rs` | — | 100 | -| 2: Hold opp-cost | `experience_kernels.cu` | `hold_cost_scale_compute_kernel.cu` | 70 | +| 2: Hold opp-cost | `experience_kernels.cu` | `hold_cost_scale_compute_kernel.cu` | 90 | +| (Q1(b) added: `hold_reward_ema` producer + dual emission centered/uncentered split — +20 LoC) | | | | | 3: n-step distributor | `gpu_experience_collector.rs` | `n_step_credit_kernel.cu` | 110 | | 4: Aux→Q gate | `gpu_dqn_trainer.rs` | — | 40 | | 5: Adaptive curriculum | `sp14_isv_slots.rs`, `mod.rs`, `build.rs` | `sp20_emas_compute_kernel.cu`, `sp20_controllers_compute_kernel.cu`, `sp20_stats_compute_kernel.cu` | 280 | @@ -394,7 +410,7 @@ Approximate sizing per component (atomic ship per `feedback_no_partial_refactor` | 6: Behavioral tests | — | `sp19_20_behavioral_tests.rs` | 250 | | Per-component unit tests | — | various | 600 | | Audit doc + memory pearls | `audit/sp20_wireup.md` (new) | — | 100 | -| **Total** | | | **~1,600** | +| **Total** | | | **~1,620** | Comparable in scope to SP18 (~2,500 LoC). Smaller than SP15 (~5,000 LoC).