From 9d9ca3e6e7e040fee1219d8bdee2494f6033cfe2 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Sat, 9 May 2026 17:40:00 +0200 Subject: [PATCH] =?UTF-8?q?spec(sp19+20):=20apply=20Q1(b)=20=E2=80=94=20Ho?= =?UTF-8?q?ld-reward=20EMA=20for=20Q-scale=20comparability?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Q1 design decision (b): add explicit hold_reward_ema to center the per-bar Hold reward, so Q(Hold) and Q(trade) targets are scale-comparable. The marginal Q(trade) > Q(Hold) preference now comes from data variance in each state (high-aux states pull Q(Hold) more negative), not from structural scale asymmetry that depends on cost_scale magnitude. Q2 decision: keep 4-quadrant fixed (no ramping partials). Already in spec. Changes: - §4.2 Hold opp-cost: dual emission documented — R_per_bar_centered (= R_per_bar - hold_reward_ema) for Q-target/replay tuple, R_per_bar uncentered for hold_baseline_buffer (Component 1 baseline) - §4.5 Kernel 1: hold_reward_ema added (per-step on Hold-state bars only) - §5 data flow: per-bar reward path shows the centered/uncentered split - ISV slots: 9 → 10 (HOLD_REWARD_EMA_INDEX added) - §8 footprint: Component 2 LoC 70 → 90 (+20 for dual emission) - Total LoC estimate: 1620 --- .../2026-05-09-sp19-20-wr-first-design.md | 32 ++++++++++++++----- 1 file changed, 24 insertions(+), 8 deletions(-) diff --git a/docs/superpowers/specs/2026-05-09-sp19-20-wr-first-design.md b/docs/superpowers/specs/2026-05-09-sp19-20-wr-first-design.md index 2e8bfed49..7c16033c2 100644 --- a/docs/superpowers/specs/2026-05-09-sp19-20-wr-first-design.md +++ b/docs/superpowers/specs/2026-05-09-sp19-20-wr-first-design.md @@ -185,13 +185,24 @@ elif hold_pct_ema < target_hold_pct - 0.05: **Buffer sizing:** `hold_baseline_buffer` is a circular buffer of size `LOOKAHEAD_HORIZON_MAX = 30` bars (matches SP19's longest horizon and bounds max trade duration). Bars older than 30 are overwritten — fine, since trades > 30 bars don't sample beyond their last 30 bars. -**Q-scale design note (intentional asymmetry between Hold and trade rewards):** Hold per-bar reward is **uncentered** (raw `-aux_conf × cost_scale`), whereas trade reward is centered via `R_used = alpha - alpha_ema`. This asymmetry is **deliberate**: it produces a marginal Q(trade) > Q(Hold) preference (Q(Hold) accumulates a small negative drift, Q(trade) is zero-mean), which counteracts the Q(Hold) attractor we are trying to break. Centering Hold rewards too would symmetrize the bias and remove this engineered preference. Behavioral test `sp20_pure_noise` validates the Hold attractor is still correctly induced when there is no signal — so the marginal trade preference doesn't override the no-signal Hold default. +**Q-scale comparability — dual emission with centering:** Both Hold and trade rewards are centered via long-run EMA subtraction so Q(Hold) and Q(trade) are scale-comparable. Hold's `R_per_bar_centered` writes to the replay tuple for Q-target consumption; the **uncentered** `R_per_bar` writes to `hold_baseline_buffer` for Component 1's structural baseline subtraction. The two emissions serve different purposes — centered for Q-target stability, uncentered for "what would Hold have paid" baseline. + +``` +hold_reward_ema_t+1 ← (1-α) × hold_reward_ema_t + α × R_per_bar_i # Wiener-α floor 0.4 + # over Hold-state bars only + +R_per_bar_centered_i = R_per_bar_i - hold_reward_ema_t # for Q-target / replay tuple + +hold_baseline_buffer[i] = R_per_bar_i # uncentered, for Component 1 +``` + +The marginal Q(trade) > Q(Hold) preference now comes from the **data** rather than from structural scale asymmetry: in high-aux-confidence states, Hold's variance pulls Q(Hold) more negative (when aux is right); in low-aux states, Q(Hold) is near-zero and Q(trade) is near-zero, model has no preference. This is cleaner than scale-asymmetric drift and robust as cost_scale evolves. **Implementation:** - 1 new producer kernel `hold_cost_scale_compute_kernel.cu` (~50 LoC, follows SP4 Pearls A+D) - Modification at `experience_kernels.cu` per-bar reward path: add `R_per_bar_i` term, write to `hold_baseline_buffer` -**ISV slots (3 new):** `HOLD_COST_SCALE_INDEX`, `TARGET_HOLD_PCT_INDEX`, `HOLD_PCT_EMA_INDEX`. +**ISV slots (4 new):** `HOLD_COST_SCALE_INDEX`, `TARGET_HOLD_PCT_INDEX`, `HOLD_PCT_EMA_INDEX`, `HOLD_REWARD_EMA_INDEX`. ### 4.3 Component 3 — n-step credit distributor @@ -283,6 +294,7 @@ Produces (Wiener-α EMAs with floor 0.4): - `aux_dir_acc_ema` — per-step (extends existing SP14-C signal) - `hold_pct_ema` — per-step - `alpha_ema` — per-trade +- `hold_reward_ema` — per-step on Hold-state bars only (for Q-scale comparability) **Kernel 2: `sp20_controllers_compute`** (~70 LoC, depends on Kernel 1) @@ -301,7 +313,7 @@ Consumes: aux_logits raw. Produces: `aux_conf_p50`, `aux_conf_std` (used by Kernel 1). Uses `sp4_histogram_p99.cuh` pattern for p50 (median, k=128). -**Total new ISV slots: 9 in range [510..520).** +**Total new ISV slots: 10 in range [510..520).** ### 4.6 Component 6 — Behavioral Test Suite @@ -332,9 +344,12 @@ All reuse SP15 Phase 2A LobBar harness. ~250 LoC test code. Each runs <30s on RT [Step] trade_physics(a_t) → step_return, is_close, position state per-bar reward path: - if action == Hold: R_per_bar = -aux_conf × cost_scale (Component 2) - else: R_per_bar = 0 - hold_baseline_buffer[t] = -aux_conf × cost_scale (always, for Component 1) + if action == Hold: + R_per_bar = -aux_conf × cost_scale # raw, for hold_reward_ema update + R_per_bar_centered = R_per_bar - hold_reward_ema # for Q-target via replay tuple + else: + R_per_bar_centered = 0 + hold_baseline_buffer[t] = -aux_conf × cost_scale # uncentered, for Component 1 [Trade close detected] R_event = 4-quadrant reward (Component 1) @@ -385,7 +400,8 @@ Approximate sizing per component (atomic ship per `feedback_no_partial_refactor` | Component | Files modified | New files | Approx LoC | |---|---|---|---| | 1: Reward kernel | `experience_kernels.cu`, `gpu_experience_collector.rs` | — | 100 | -| 2: Hold opp-cost | `experience_kernels.cu` | `hold_cost_scale_compute_kernel.cu` | 70 | +| 2: Hold opp-cost | `experience_kernels.cu` | `hold_cost_scale_compute_kernel.cu` | 90 | +| (Q1(b) added: `hold_reward_ema` producer + dual emission centered/uncentered split — +20 LoC) | | | | | 3: n-step distributor | `gpu_experience_collector.rs` | `n_step_credit_kernel.cu` | 110 | | 4: Aux→Q gate | `gpu_dqn_trainer.rs` | — | 40 | | 5: Adaptive curriculum | `sp14_isv_slots.rs`, `mod.rs`, `build.rs` | `sp20_emas_compute_kernel.cu`, `sp20_controllers_compute_kernel.cu`, `sp20_stats_compute_kernel.cu` | 280 | @@ -394,7 +410,7 @@ Approximate sizing per component (atomic ship per `feedback_no_partial_refactor` | 6: Behavioral tests | — | `sp19_20_behavioral_tests.rs` | 250 | | Per-component unit tests | — | various | 600 | | Audit doc + memory pearls | `audit/sp20_wireup.md` (new) | — | 100 | -| **Total** | | | **~1,600** | +| **Total** | | | **~1,620** | Comparable in scope to SP18 (~2,500 LoC). Smaller than SP15 (~5,000 LoC).