From defbd0abe159a39a374ecfad017e867b0fe39ce5 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Sat, 9 May 2026 17:35:36 +0200 Subject: [PATCH] spec(sp19+20): patch 7 review issues MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit P1 (must-fix bugs/gaps): 1. ASYM_RATIO_INDEX → LOSS_CAP_INDEX with explicit formula in §4.1 and §4.5 (was double source-of-truth — formula in §4.1, ISV slot orphaned) 2. Replay buffer schema change (per-bar aux_conf) added to §8 implementation footprint (~50 LoC additional, was invisible in original spec) 3. hold_baseline_buffer size specified = LOOKAHEAD_HORIZON_MAX = 30 bars (§4.2) 4. "4-tier gate" typo → "5-tier gate" in §7 P2 (clarifications): 5. alpha_ema centering documented as load-bearing for cold-start learning (§4.1) — not just for Q-target stability. EV is slightly negative at WR=46% with uninformed SP19 label; advantage-style centering rescues cold-start. 6. Q-scale asymmetry between Hold (uncentered) and trade (alpha_ema centered) documented as INTENTIONAL design choice (§4.2) — produces marginal Q(trade) > Q(Hold) preference that counteracts the Q(Hold) attractor. Behavioral test sp20_pure_noise validates the no-signal Hold default still works. P3 (tightening): 7. Behavioral test thresholds tightened (§4.6): - sp20_pure_trend: WR > 70% → > 90%, PF > 2.0 → > 3.0 - sp20_pure_noise: Hold% > 80% → > 95%, trades < 50 → < 20 --- .../2026-05-09-sp19-20-wr-first-design.md | 31 ++++++++++++------- 1 file changed, 20 insertions(+), 11 deletions(-) diff --git a/docs/superpowers/specs/2026-05-09-sp19-20-wr-first-design.md b/docs/superpowers/specs/2026-05-09-sp19-20-wr-first-design.md index 5af0aad1c..2e8bfed49 100644 --- a/docs/superpowers/specs/2026-05-09-sp19-20-wr-first-design.md +++ b/docs/superpowers/specs/2026-05-09-sp19-20-wr-first-design.md @@ -71,8 +71,8 @@ This spec re-grades the model on **closed-trade directional correctness** with * spread R_used over trade_duration bars │ │ │ └──────────► Replay buffer ◄───────────┘ - │ (aux_conf at trade open - │ stored alongside reward) + │ (aux_conf stored per bar, + │ read at the sampled state) ▼ [Bellman target — Component 4] Q_target = aux_gate × full_target + @@ -88,7 +88,7 @@ This spec re-grades the model on **closed-trade directional correctness** with * fires every bar Hold is held; aux_conf high → cost high ──── Adaptive Curriculum (ISV) — Component 5 ──── - asym_ratio ← ramp(wr_ema) + loss_cap ← ramp(wr_ema) aux_threshold ← adapt(aux_dir_acc_ema) n_step ← adapt(trade_duration_ema) cost_scale ← adapt(hold_pct_ema) @@ -108,8 +108,9 @@ Six components, each independently testable, all shipping atomically per `feedba ``` At trade close (is_close=1): win_cap = +1.0 - loss_cap = -1.0 - clamp((wr_ema - 0.50) / 0.05, 0, 1) - # = -1 below WR=50%, ramps to -2 by WR=55% + loss_cap = ISV[LOSS_CAP_INDEX] # produced by Kernel 2 from wr_ema + # value: -1.0 - clamp((wr_ema - 0.50) / 0.05, 0, 1) + # = -1 below WR=50%, ramps to -2 by WR=55% label_at_open = sign(SP19_blended_label[trade_open_bar]) dir_match = (sign(close_PnL) == label_at_open) @@ -133,13 +134,15 @@ At trade close (is_close=1): → distributed n-step credit by Component 3 ``` +**alpha_ema centering is load-bearing for cold-start learning, not just Q-target stability.** At WR=46% with the 4-quadrant table and uninformed SP19 label, raw EV ≈ −0.06 (model would prefer not to trade). The `R_used = alpha − alpha_ema` subtraction transforms absolute EV into advantage-style relative-to-mean. Even when absolute alpha is negative, the model still learns "this trade was BETTER than my average trade → do more like this". Without this centering, the model collapses to Hold during cold start before it can learn directional discrimination. Do not drop this term during implementation as an "optimization". + **Implementation:** - New device function `sp20_compute_event_reward()` in `experience_kernels.cu` (~80 LoC) - Replaces SP18 D-leg `compute_sp18_hold_opportunity_cost` (returned 0 throughout 5rbml — empirically null) - Replaces SP12 v3 reward block (`compute_sp12_reward_with_cost` and per-bar additions) **ISV slots (3 new):** -- `ASYM_RATIO_INDEX` — produced by Kernel 2 from `wr_ema` +- `LOSS_CAP_INDEX` — produced by Kernel 2 from `wr_ema` (was `ASYM_RATIO_INDEX` — renamed for clarity; stores the actual loss_cap value, not a ratio) - `ALPHA_EMA_INDEX` — produced by Kernel 1 (Wiener-α EMA) - `WR_EMA_INDEX` — produced by Kernel 1 (per-trade EMA) @@ -180,6 +183,10 @@ elif hold_pct_ema < target_hold_pct - 0.05: - Aux untrained (warmup): `aux_conf_p50_ema ≈ 0.05` → `target_hold_pct ≈ 0.725` → controller pushes `cost_scale` toward floor 0.01 → per-bar cost effectively zero. **No explicit warmup gate needed.** - Flat action: treated as not-Hold (no cost). Flat is intentional close-all, conceptually different from Hold. +**Buffer sizing:** `hold_baseline_buffer` is a circular buffer of size `LOOKAHEAD_HORIZON_MAX = 30` bars (matches SP19's longest horizon and bounds max trade duration). Bars older than 30 are overwritten — fine, since trades > 30 bars don't sample beyond their last 30 bars. + +**Q-scale design note (intentional asymmetry between Hold and trade rewards):** Hold per-bar reward is **uncentered** (raw `-aux_conf × cost_scale`), whereas trade reward is centered via `R_used = alpha - alpha_ema`. This asymmetry is **deliberate**: it produces a marginal Q(trade) > Q(Hold) preference (Q(Hold) accumulates a small negative drift, Q(trade) is zero-mean), which counteracts the Q(Hold) attractor we are trying to break. Centering Hold rewards too would symmetrize the bias and remove this engineered preference. Behavioral test `sp20_pure_noise` validates the Hold attractor is still correctly induced when there is no signal — so the marginal trade preference doesn't override the no-signal Hold default. + **Implementation:** - 1 new producer kernel `hold_cost_scale_compute_kernel.cu` (~50 LoC, follows SP4 Pearls A+D) - Modification at `experience_kernels.cu` per-bar reward path: add `R_per_bar_i` term, write to `hold_baseline_buffer` @@ -281,7 +288,7 @@ Produces (Wiener-α EMAs with floor 0.4): Consumes: above EMAs. Produces controller outputs: -- `ASYM_RATIO_INDEX` = ramp from wr_ema (Section 4.1) +- `LOSS_CAP_INDEX` = -1.0 - clamp((wr_ema - 0.50) / 0.05, 0, 1) (Section 4.1) - `N_STEP_INDEX` = round(trade_duration_ema), clamp [1, 30] - `AUX_CONF_THRESHOLD` = clamp(aux_dir_acc_ema − 0.50, 0.01, 0.20) - `AUX_GATE_TEMP` = aux_conf_std_ema (with floor) @@ -302,8 +309,8 @@ Uses `sp4_histogram_p99.cuh` pattern for p50 (median, k=128). | Test | Synthetic market | Pass criterion | Proves | |---|---|---|---| -| `sp20_pure_trend` | Strong monotonic trend, low noise | WR > 70%, PF > 2.0 | Model CAN learn directional skill when signal exists | -| `sp20_pure_noise` | i.i.d. zero-mean returns | Hold% > 80%, trades < 50/run | Model doesn't gamble — Hold attractor correctly induced when no edge exists | +| `sp20_pure_trend` | Strong monotonic trend, low noise | WR > 90%, PF > 3.0 | Model CAN learn directional skill when signal exists. Tightened: pure-trend with low noise is near-deterministic; loose threshold hides real failures. | +| `sp20_pure_noise` | i.i.d. zero-mean returns | Hold% > 95%, trades < 20/run | Model doesn't gamble — Hold attractor correctly induced when no edge exists. Tightened for the same reason. | | `sp20_confidence_correlation` | Mixed (50% trend + 50% noise bars) | corr(aux_conf, trade_WR) > 0.4 | Aux gate is functional | | `sp20_asymmetry_no_game` | Trending market, hard symmetric clamp | PF > 1.0 (small but positive) | WR optimization isn't an artifact of asymmetric clamp | | `sp20_n_step_distribution` | 3-bar trade, known close reward = +0.6, n_step = 3 | Each of 3 bars receives credit ≈ 0.2 | n-step distributor is correct | @@ -359,7 +366,7 @@ Verified-natural cases (no explicit handling needed): ## 7. Testing strategy -4-tier gate, in order. Each tier blocks the next: +5-tier gate, in order. Each tier blocks the next: 1. **Per-kernel unit tests** (RTX 3050 Ti, ~50 tests) - Each kernel: oracle test, sentinel-bootstrap test, ISV signal independence test, reset behavior test @@ -382,10 +389,12 @@ Approximate sizing per component (atomic ship per `feedback_no_partial_refactor` | 3: n-step distributor | `gpu_experience_collector.rs` | `n_step_credit_kernel.cu` | 110 | | 4: Aux→Q gate | `gpu_dqn_trainer.rs` | — | 40 | | 5: Adaptive curriculum | `sp14_isv_slots.rs`, `mod.rs`, `build.rs` | `sp20_emas_compute_kernel.cu`, `sp20_controllers_compute_kernel.cu`, `sp20_stats_compute_kernel.cu` | 280 | +| Replay buffer schema | `gpu_experience_collector.rs`, `GpuBatchPtrs` struct, callers | — | 50 | +| (per-bar `aux_conf` field for Component 4 gate consumption — propagated through experience tuple, batch ptrs, replay path)| | | | | 6: Behavioral tests | — | `sp19_20_behavioral_tests.rs` | 250 | | Per-component unit tests | — | various | 600 | | Audit doc + memory pearls | `audit/sp20_wireup.md` (new) | — | 100 | -| **Total** | | | **~1,550** | +| **Total** | | | **~1,600** | Comparable in scope to SP18 (~2,500 LoC). Smaller than SP15 (~5,000 LoC).