diff --git a/docs/superpowers/specs/2026-05-09-sp19-20-wr-first-design.md b/docs/superpowers/specs/2026-05-09-sp19-20-wr-first-design.md new file mode 100644 index 000000000..5af0aad1c --- /dev/null +++ b/docs/superpowers/specs/2026-05-09-sp19-20-wr-first-design.md @@ -0,0 +1,426 @@ +# SP19+20: WR-First Reward + Multi-Horizon Label Utilization + +**Spec date:** 2026-05-09 +**Branch:** `sp19-20-wr-first` forks from `feat/sp18-combined` HEAD `d39005c6f` +**Combines:** SP19 (multi-horizon labels, already landed) + SP20 (WR-first reward design) +**Per memory rule:** combined into one spec/plan per `pearl_no_deferrals_for_complementary_fixes` — non-overlapping refactor scopes (SP19 = producer-side label blend already on disk; SP20 = reward kernel + Q-target plumbing) + +## 1. Goal + +Hard goal: **WR ≥ 55%** closed-trade directional accuracy AND **textbook PF ≥ 2.0**, walk-forward stable across folds, per-regime stable across T/R/V buckets. + +Soft goal: profit comes from honest directional accuracy, not from payoff-asymmetry compensation. SP-chain pattern (WR=46% + omega 1.32:1 → positive Sharpe) is "fragile trading" and unacceptable as the terminal state. + +## 2. Background and motivation + +### 2.1 What we know honestly (post 2026-05-09 metric audit) +- **WR = 46-50% across 16+ SP runs** is honest closed-trade directional accuracy. Verified by audit at `experience_kernels.cu:2857`, `backtest_metrics_kernel.cu:256, 424`, `trade_physics.cuh:515` — all use realized PnL > 0 predicate. +- **Sharpe ≈ 78** (annualized at √(2880 × 252) = 852 for our volume bars) is honest under its per-bar definition. +- **Per-trade EV ≈ +0.07** in avg_loss units. Positive but fragile to slippage/fees. PF ≈ 1.32 (textbook) implies wins compensate losses by 1.55×. +- The **Q(Hold) attractor is structural across all SP-chain runs** (vtj9r → 5rbml → wgdc7). Hold% climbs to 50-62% by epoch 5; SP19 multi-horizon reward delayed but did not break it. +- **Per-regime WR is uniformly ~46% across T/R/V** (wgdc7 instrumentation). No regime-conditional edge surfaces under current reward. + +### 2.2 What this spec attacks +The SP-chain reward shapes (popart, cf, trail_r, micro, opp_cost, bonus, multi-horizon blend) all grade *cumulative per-bar PnL*, not closed-trade directional correctness. The model learns to maximize cumulative reward via payoff asymmetry (1.32:1 with WR=46%) rather than directional skill (would require WR>50%). + +This spec re-grades the model on **closed-trade directional correctness** with **multi-horizon ground-truth check** and **adaptive asymmetric loss aversion** that ramps in only after the model has learned the directional baseline. The mechanism is fundamentally event-driven (per `pearl_event_driven_reward_density_alignment`), not per-bar-shaped. + +### 2.3 What we are deliberately NOT doing +- More architecture changes — capacity is not the constraint (per audit). +- Per-bar reward shaping — `pearl_event_driven_reward_density_alignment` warns this creates exposure-positive bias for event-driven objectives. +- Optimizing Sharpe — Sharpe is honest but compensation-based; targeting it directly perpetuates the fragile-trading failure mode. +- Multi-instrument joint training, cross-asset features, instrument selection — flagged as future work per `project_label_horizon_is_bottleneck`, but addressed only after SP20 establishes whether the model can lift WR with current data. + +## 3. Architecture + +``` + ┌──────── Encoder ────────┐ + │ │ + ▼ ▼ + [Shared Trunk (GRN)] [Aux Trunk (SP14-C, separate)] + │ │ + ┌─────┼──────┐ │ + ▼ ▼ ▼ ▼ + Q_dir Q_mag Q_ord/urg aux_direction (multi-bar, supervised by SP19 labels) + │ │ + ▼ ▼ + Thompson selector aux_conf, aux_dir_acc_ema (ISV) + │ │ + ▼ │ + Action a_t │ + │ │ + ▼ │ + Trade physics ──── step_return │ + │ │ + ┌─────┴──── is_close ? │ + │ │ + ▼ (yes) │ + [Reward Kernel — Component 1] │ + R_event = direction_check(close_PnL, │ + SP19_label, │ + asym_ratio) │ + │ │ + ▼ │ + alpha = R_event − hold_baseline_n_bars │ + │ │ + ▼ │ + R_used = alpha − alpha_ema │ + │ │ + ▼ │ + [n-step distributor — Component 3] │ + spread R_used over trade_duration bars │ + │ │ + └──────────► Replay buffer ◄───────────┘ + │ (aux_conf at trade open + │ stored alongside reward) + ▼ + [Bellman target — Component 4] + Q_target = aux_gate × full_target + + (1 − aux_gate) × baseline_target + where aux_gate = sigmoid((aux_conf − threshold) / temp) + │ + ▼ + Q-loss + Q-gradients + │ + ▼ + ──── Per-bar (in Hold state) — Component 2 ──── + Hold_opp_cost = −aux_conf × cost_scale + fires every bar Hold is held; aux_conf high → cost high + + ──── Adaptive Curriculum (ISV) — Component 5 ──── + asym_ratio ← ramp(wr_ema) + aux_threshold ← adapt(aux_dir_acc_ema) + n_step ← adapt(trade_duration_ema) + cost_scale ← adapt(hold_pct_ema) + alpha_ema_α ← Wiener-α floor 0.4 +``` + +Six components, each independently testable, all shipping atomically per `feedback_no_partial_refactor`. + +## 4. Components + +### 4.1 Component 1 — Reward kernel (event-driven) + +**Fires:** at every trade close (`is_close == 1`) detected by existing trade_physics. + +**Formula:** + +``` +At trade close (is_close=1): + win_cap = +1.0 + loss_cap = -1.0 - clamp((wr_ema - 0.50) / 0.05, 0, 1) + # = -1 below WR=50%, ramps to -2 by WR=55% + + label_at_open = sign(SP19_blended_label[trade_open_bar]) + dir_match = (sign(close_PnL) == label_at_open) + + R_event = match dir_match, sign(close_PnL): + ( true, +1) ⇒ +win_cap # right reason, right outcome + (false, +1) ⇒ +0.5 # wrong reason, right outcome + ( true, -1) ⇒ -0.5 # right reason, wrong outcome + (false, -1) ⇒ loss_cap # wrong reason, wrong outcome + ( *, 0) ⇒ 0.0 # zero PnL = no information + + hold_baseline = Σ over [trade_open .. trade_close] of hold_baseline_buffer[i] + # see Component 2 for the buffer; negative number + + alpha = R_event - hold_baseline # subtraction: hold_baseline is negative, so alpha ≥ R_event + + alpha_ema_t+1 ← (1-α) × alpha_ema_t + α × alpha # Wiener-α floor 0.4 + + R_used = alpha - alpha_ema_t + + → distributed n-step credit by Component 3 +``` + +**Implementation:** +- New device function `sp20_compute_event_reward()` in `experience_kernels.cu` (~80 LoC) +- Replaces SP18 D-leg `compute_sp18_hold_opportunity_cost` (returned 0 throughout 5rbml — empirically null) +- Replaces SP12 v3 reward block (`compute_sp12_reward_with_cost` and per-bar additions) + +**ISV slots (3 new):** +- `ASYM_RATIO_INDEX` — produced by Kernel 2 from `wr_ema` +- `ALPHA_EMA_INDEX` — produced by Kernel 1 (Wiener-α EMA) +- `WR_EMA_INDEX` — produced by Kernel 1 (per-trade EMA) + +### 4.2 Component 2 — Hold opportunity-cost (per-bar) + +**Fires:** every bar. + +Two parallel emissions: + +``` +Per bar i: + aux_conf_i = max(softmax(aux_logits_i)) - 1/3 # range [0, 2/3] + cost_scale = ISV[HOLD_COST_SCALE] # adaptive [0.01, 0.5] + + per_bar_opp_cost_i = -aux_conf_i × cost_scale + + # Path 1 — real reward, fires only on Hold action: + R_per_bar_i = (action_i == Hold) ? per_bar_opp_cost_i : 0 + + # Path 2 — counterfactual buffer, ALWAYS recorded: + hold_baseline_buffer[i] = per_bar_opp_cost_i + # consumed by Component 1 at trade close +``` + +**Adaptive cost_scale (Kernel 2 controller):** + +``` +target_hold_pct = clamp(0.8 - aux_conf_p50_ema × 1.5, 0.1, 0.8) +hold_pct_ema = Wiener-α EMA of (action == Hold) indicator + +if hold_pct_ema > target_hold_pct + 0.05: + cost_scale ← clamp(cost_scale × 1.05, 0.01, 0.5) +elif hold_pct_ema < target_hold_pct - 0.05: + cost_scale ← clamp(cost_scale × 0.95, 0.01, 0.5) +``` + +**Self-stabilizing properties:** +- Aux untrained (warmup): `aux_conf_p50_ema ≈ 0.05` → `target_hold_pct ≈ 0.725` → controller pushes `cost_scale` toward floor 0.01 → per-bar cost effectively zero. **No explicit warmup gate needed.** +- Flat action: treated as not-Hold (no cost). Flat is intentional close-all, conceptually different from Hold. + +**Implementation:** +- 1 new producer kernel `hold_cost_scale_compute_kernel.cu` (~50 LoC, follows SP4 Pearls A+D) +- Modification at `experience_kernels.cu` per-bar reward path: add `R_per_bar_i` term, write to `hold_baseline_buffer` + +**ISV slots (3 new):** `HOLD_COST_SCALE_INDEX`, `TARGET_HOLD_PCT_INDEX`, `HOLD_PCT_EMA_INDEX`. + +### 4.3 Component 3 — n-step credit distributor + +**Fires:** at each trade close, after Component 1 produces `R_used`. + +**Formula (uniform distribution):** + +``` +At trade close (is_close=1): + R_used = α - α_ema # from Component 1 + n_step = ISV[N_STEP_INDEX] # = round(trade_duration_ema), bounded [1, 30] + trade_duration_actual = bar_close - bar_open + + n_distribute = min(n_step, trade_duration_actual) + + for k in [0 .. n_distribute): + bar_idx = bar_close - k + per_bar_credit[bar_idx] += R_used / n_distribute +``` + +**Why uniform, not λ-decay:** +The trade's reward is event-attributed: every bar in the trade contributed equally to the close-PnL outcome (open bar created position, close bar realized it, intermediate bars held). λ-decay would systematically under-credit the open bar — the bar that committed the directional decision. + +**Trade boundary discipline:** +The eligibility trace state must reset at every trade close, not carry across. This fixes the SP18 B-leg `gpu_experience_collector.rs:4154` self-bootstrap bug as part of SP20 (per `pearl_no_deferrals_for_complementary_fixes` — non-overlapping scope, deferred no longer). + +**Implementation:** +- 1 new kernel `n_step_credit_kernel.cu` (~80 LoC) +- Trade-boundary tracking reuses existing `trade_close_event` buffer +- SP18 B-leg fix at `gpu_experience_collector.rs:4154`: trace state resets at trade close (~30 LoC change) + +**ISV slots (1 new):** `N_STEP_INDEX`. Plus `TRADE_DURATION_EMA_INDEX` (for the producer). + +### 4.4 Component 4 — Aux→Q confidence gate + +**Fires:** at each replay batch step, in the Bellman target computation. + +**Formula:** + +``` +At each replay batch step: + aux_conf = max(softmax(aux_logits_at_state)) - 1/3 + threshold = ISV[AUX_CONF_THRESHOLD] # adaptive + temp = ISV[AUX_GATE_TEMP] # adaptive + + gate = sigmoid((aux_conf - threshold) / temp) # ∈ [0, 1] + + Q_target_full = r + γ × max_a' Q(s', a') + Q_target_baseline = mean_a Q(s, a) # neutralizes action preference, doesn't punish + Q_target = gate × Q_target_full + (1 - gate) × Q_target_baseline +``` + +**Why `mean_a Q(s, a)` baseline:** +- `0` baseline punishes all actions for being in noise → drives Hold-everywhere. +- `Q(s, a)` (no update) wastes uncertain-state data. +- `mean_a Q` actively neutralizes action preference at uncertain states without freezing learning. Hold emerges from learned indifference, not from punishment. + +**Adaptive threshold and temperature:** + +``` +threshold = clamp(aux_dir_acc_ema - 0.50, 0.01, 0.20) +temp = aux_conf_std_ema # gate sharpness adapts to actual aux confidence spread +``` + +**Self-stabilizing curriculum:** +- Cold start (aux ≈ random): threshold ≈ 0.01 → gate ≈ 1 always → standard Q-learning +- Aux improves: threshold ramps with `aux_dir_acc_ema − 0.50` → gate becomes selective +- Aux strong: gate sharply discriminates by confidence +- Aux anti-correlated (< 0.50): clamp floor → gate ≈ 1 → fall back to standard Q + +**Implementation:** +- `gpu_dqn_trainer.rs` Bellman target site: ~40 LoC modification +- 2 new ISV slots: `AUX_CONF_THRESHOLD_INDEX`, `AUX_GATE_TEMP_INDEX` +- Producer kernel `aux_gate_params_compute_kernel.cu`: ~60 LoC (fused into Kernel 2) +- Aux logits already exposed at training step (SP14-C wired); no new readback needed + +### 4.5 Component 5 — Adaptive Curriculum Producers + +All ISV-driven parameters come from 3 fused producer kernels (per `pearl_fused_per_group_statistics_oracle`): + +**Kernel 1: `sp20_emas_compute`** (~80 LoC, per-step + per-trade triggers) + +Consumes: trade events, aux logits, action history. +Produces (Wiener-α EMAs with floor 0.4): +- `wr_ema` — per-trade +- `trade_duration_ema` — per-trade +- `aux_conf_p50_ema` — per-step (from Kernel 3) +- `aux_conf_std_ema` — per-step (from Kernel 3) +- `aux_dir_acc_ema` — per-step (extends existing SP14-C signal) +- `hold_pct_ema` — per-step +- `alpha_ema` — per-trade + +**Kernel 2: `sp20_controllers_compute`** (~70 LoC, depends on Kernel 1) + +Consumes: above EMAs. +Produces controller outputs: +- `ASYM_RATIO_INDEX` = ramp from wr_ema (Section 4.1) +- `N_STEP_INDEX` = round(trade_duration_ema), clamp [1, 30] +- `AUX_CONF_THRESHOLD` = clamp(aux_dir_acc_ema − 0.50, 0.01, 0.20) +- `AUX_GATE_TEMP` = aux_conf_std_ema (with floor) +- `TARGET_HOLD_PCT` = clamp(0.8 − aux_conf_p50_ema × 1.5, 0.1, 0.8) +- `HOLD_COST_SCALE` = two-sided controller (hold_pct_ema vs target) + +**Kernel 3: `sp20_stats_compute`** (~60 LoC, per-step) + +Consumes: aux_logits raw. +Produces: `aux_conf_p50`, `aux_conf_std` (used by Kernel 1). +Uses `sp4_histogram_p99.cuh` pattern for p50 (median, k=128). + +**Total new ISV slots: 9 in range [510..520).** + +### 4.6 Component 6 — Behavioral Test Suite + +7 synthetic-market tests on RTX 3050 Ti, gating L40S deployment. + +| Test | Synthetic market | Pass criterion | Proves | +|---|---|---|---| +| `sp20_pure_trend` | Strong monotonic trend, low noise | WR > 70%, PF > 2.0 | Model CAN learn directional skill when signal exists | +| `sp20_pure_noise` | i.i.d. zero-mean returns | Hold% > 80%, trades < 50/run | Model doesn't gamble — Hold attractor correctly induced when no edge exists | +| `sp20_confidence_correlation` | Mixed (50% trend + 50% noise bars) | corr(aux_conf, trade_WR) > 0.4 | Aux gate is functional | +| `sp20_asymmetry_no_game` | Trending market, hard symmetric clamp | PF > 1.0 (small but positive) | WR optimization isn't an artifact of asymmetric clamp | +| `sp20_n_step_distribution` | 3-bar trade, known close reward = +0.6, n_step = 3 | Each of 3 bars receives credit ≈ 0.2 | n-step distributor is correct | +| `sp20_per_regime_wr` | T/R/V segments (40%/40%/20%) | WR_T > 65%, WR_R > 50% with Hold%_R > 60%, WR_V > 50% with low trade count | Model behaviorally adapts to regime | +| `sp20_regime_transition` | Abrupt trend → noise at bar 50 | Trade frequency drops 50% within 10 bars of transition | Model recognizes regime change | + +All reuse SP15 Phase 2A LobBar harness. ~250 LoC test code. Each runs <30s on RTX 3050 Ti. + +**Hard rule:** No L40S dispatch until all 7 behavioral tests pass on dev. Per cross-cutting SP15 discipline. + +## 5. Data flow (one full step) + +``` +[Forward pass] + encoder → trunk → Q-heads, aux head + aux_logits → softmax → aux_conf (read at every step into sp20_stats_compute) + Thompson selector → action a_t + +[Step] + trade_physics(a_t) → step_return, is_close, position state + per-bar reward path: + if action == Hold: R_per_bar = -aux_conf × cost_scale (Component 2) + else: R_per_bar = 0 + hold_baseline_buffer[t] = -aux_conf × cost_scale (always, for Component 1) + +[Trade close detected] + R_event = 4-quadrant reward (Component 1) + hold_baseline = sum(hold_baseline_buffer over trade duration) + alpha = R_event - hold_baseline + R_used = alpha - alpha_ema + n_step_credit_kernel: distribute R_used over trade_duration bars (Component 3) + Update wr_ema, trade_duration_ema, alpha_ema (Kernel 1) + +[Replay sample] + Read state, action, distributed_per_bar_credit, aux_conf_at_state + Q_target = gate(aux_conf, threshold) × Q_full + (1 - gate) × Q_baseline (Component 4) + Loss, backprop, Adam update + +[Per epoch] + Recompute controllers (Kernel 2) + Update aux_conf_p50/std (Kernel 3) +``` + +## 6. Error handling philosophy + +Per CLAUDE.md: trust internal contracts. No defensive validation between trusted modules. Validate only at system boundaries (data input, action emission). Use `feedback_no_stubs`: every kernel wired for real or deleted. Edge cases surfaced during design that turn out to be subsumed by natural controller dynamics are **not** explicitly handled (per `feedback_no_quickfixes`). + +Verified-natural cases (no explicit handling needed): +- Aux warmup → controller produces low cost_scale automatically +- Aux anti-correlated (< 0.50 dir_acc) → threshold clamp produces gate ≈ 1 +- First observation → universal Pearl A bootstrap pattern +- Aux NaN → indicates training crash, separate concern + +## 7. Testing strategy + +4-tier gate, in order. Each tier blocks the next: + +1. **Per-kernel unit tests** (RTX 3050 Ti, ~50 tests) + - Each kernel: oracle test, sentinel-bootstrap test, ISV signal independence test, reset behavior test +2. **Component integration tests** (RTX 3050 Ti, ~20 tests) + - Components 1+2 coupling, 1+3 n-step distribution, 4+aux signal flow +3. **Behavioral synthetic tests** (RTX 3050 Ti, 7 tests from §4.6) +4. **L40S integration smoke** (5-epoch × 2-fold) — ~75 min, reads HEALTH_DIAG behavioral metrics +5. **Full validation** (L40S 50-epoch × 3-seed × 3-fold) + +Anomaly at any tier: stop, root-cause, re-run from current tier. Per `feedback_kill_runs_on_anomaly_quickly`. + +## 8. Implementation footprint + +Approximate sizing per component (atomic ship per `feedback_no_partial_refactor`): + +| Component | Files modified | New files | Approx LoC | +|---|---|---|---| +| 1: Reward kernel | `experience_kernels.cu`, `gpu_experience_collector.rs` | — | 100 | +| 2: Hold opp-cost | `experience_kernels.cu` | `hold_cost_scale_compute_kernel.cu` | 70 | +| 3: n-step distributor | `gpu_experience_collector.rs` | `n_step_credit_kernel.cu` | 110 | +| 4: Aux→Q gate | `gpu_dqn_trainer.rs` | — | 40 | +| 5: Adaptive curriculum | `sp14_isv_slots.rs`, `mod.rs`, `build.rs` | `sp20_emas_compute_kernel.cu`, `sp20_controllers_compute_kernel.cu`, `sp20_stats_compute_kernel.cu` | 280 | +| 6: Behavioral tests | — | `sp19_20_behavioral_tests.rs` | 250 | +| Per-component unit tests | — | various | 600 | +| Audit doc + memory pearls | `audit/sp20_wireup.md` (new) | — | 100 | +| **Total** | | | **~1,550** | + +Comparable in scope to SP18 (~2,500 LoC). Smaller than SP15 (~5,000 LoC). + +## 9. Cross-references + +- `pearl_event_driven_reward_density_alignment` — design principle for §4.1, §4.3 +- `pearl_audit_unboundedness_for_implicit_asymmetry` — design principle for asymmetric clamp in §4.1 +- `pearl_one_unbounded_signal_per_reward` — informs that asym_ratio is the only unbounded multiplicand +- `pearl_separate_aux_trunk_when_shared_starves` — §4.4 leverages the separate aux trunk (SP14-C) +- `pearl_no_deferrals_for_complementary_fixes` — combining SP19+SP20 + SP18 B-leg into one spec +- `pearl_controller_anchors_isv_driven` — every adaptive parameter in §4.5 +- `pearl_first_observation_bootstrap` — sentinel = 0, replace on first observation for all EMAs +- `pearl_wiener_alpha_floor_for_nonstationary` — α floor 0.4 for control loops +- `feedback_no_partial_refactor` — atomic ship across all 6 components +- `feedback_isv_for_adaptive_bounds` — every adaptive bound in ISV +- `feedback_no_atomicadd` — sp4_histogram_p99 pattern for p50 +- `feedback_no_stubs` — every kernel wired or deleted +- `feedback_kill_runs_on_anomaly_quickly` — tier gates +- `project_metric_pipeline_inflation_audit` — context establishing Sharpe is honest, WR is real +- `project_goal_wr_55_pf_2` — the goal this spec implements +- `project_label_horizon_is_bottleneck` — the now-partially-invalidated claim that triggered SP19 + +## 10. Success criteria + +A successful SP20 final L40S run (50-epoch × 3-seed × 3-fold) produces: + +- **WR ≥ 55%** (mean across folds), with no fold below 52% +- **Textbook PF ≥ 2.0** (mean across folds), with no fold below 1.5 +- **Per-regime WR_T ≥ 60%, WR_R ≥ 52%, WR_V ≥ 52%** (no regime degenerate) +- **Sharpe still positive** (don't care about magnitude — WR/PF are the goals) +- **Hold% in [20%, 50%] across training** (not Hold-everywhere, not over-trading) +- **Behavioral tests stable** (all 7 pass at every checkpoint, not just at start) + +A failure mode: WR plateaus at 46% but Hold% drops to 30% (model trades more without learning direction). This indicates aux gating is firing too freely; fix by tightening `AUX_CONF_THRESHOLD` floor. + +A success-with-caveat: WR=55% but PF=1.6. Means model has directional skill but exit timing isn't aligned with directional duration. Acceptable as a milestone; follow-up work focuses on exit heuristics. + +A failure that demands stop-the-presses: WR stays at 46% across all 3 seeds × 3 folds. This invalidates the hypothesis that training-side leverage is exhausted only because of reward shape — implies bottleneck is genuinely upstream (data composition, instrument selection). Trigger pivot to multi-instrument joint training per `project_label_horizon_is_bottleneck` action list.