diff --git a/docs/superpowers/specs/2026-05-04-sp12-quality-over-quantity-reward-shaping.md b/docs/superpowers/specs/2026-05-04-sp12-quality-over-quantity-reward-shaping.md new file mode 100644 index 000000000..0a263bb64 --- /dev/null +++ b/docs/superpowers/specs/2026-05-04-sp12-quality-over-quantity-reward-shaping.md @@ -0,0 +1,275 @@ +# SP12 — Per-Trade Event-Driven Reward Composition + +**Date**: 2026-05-04 (v3 — unified single-spec solution) +**Branch**: continues on `sp11-reward-as-controlled-subsystem` (SP12 is architectural continuation of SP11; no fork) +**Builds on**: SP11 B1b symmetric reward cap (commits `35db31089`, `348f6078b`, `6a259942e`) +**Anchor pearls**: +- `pearl_event_driven_reward_density_alignment` (the architectural insight) +- `pearl_audit_unboundedness_for_implicit_asymmetry` (the asymmetric-cap component) + +**Supersedes**: SP12 v1 (kitchen sink) and v2 (asymmetric cap only). v3 is the unified +architectural design that addresses all three causes of HFT-pull as a single coherent +solution. + +## TL;DR + +Reshape reward composition to be **purely per-trade event-driven** (matches objective +density). Three changes in one commit: + +1. **Asymmetric bounded cap** (loss aversion at exit events) +2. **Min-hold soft penalty with temperature** (commitment at exit events) +3. **Zero per-bar shaping rewards** (no continuous reward gradient) + +Combined: model only gets reward on trade events, with prospect-theory loss aversion and +patience requirement. ~50 LOC. Direct attack on MFT goal. + +## Background + +50-epoch validation `train-multi-seed-pmbwn` on commit `6a259942e` revealed sharpe-gaming: + +| Epoch | sharpe | total_pnl | trade_count | win_rate | +|-------|--------|-----------|-------------|----------| +| 0 | 75.19 | 0.5015 | 134,348 | 0.4601 | +| 8 | 75.42 | 0.3502 | 123,970 | 0.4637 | + +Sharpe held while PnL declined 30%. Trade rate 62% (134k/214k bars) — pure HFT noise +extraction territory. User goal: MFT-leaning trading (~20-30% trade rate, 55-60% win rate, +larger per-trade PnL, longer holds). + +## Three causes of HFT pull (architecturally distinct) + +| Cause | Mechanism | Pearl | Phase 1 fix | +|-------|-----------|-------|-------------| +| 1. Lost loss aversion | SP11 symmetric cap erased pre-fix downside-asymmetry → risk-neutral policy | `pearl_audit_unboundedness_for_implicit_asymmetry` | Asymmetric cap | +| 2. Per-bar reward gradient | `micro_reward` + `opp_cost` fire every bar → model rewards continuous exposure | `pearl_event_driven_reward_density_alignment` | Zero per-bar shaping | +| 3. No commitment requirement | Same-bar entry/exit possible without penalty → fast scalps | (none yet — may emerge from validation) | Min-hold soft penalty | + +Each acts at a different point in the reward pipeline. Together they constitute the unified +per-trade event-driven design. + +## The unified design + +### Design 1 — Asymmetric bounded cap (loss aversion) + +**File**: `crates/ml/src/cuda_pipeline/experience_kernels.cu` line ~2788 +**File**: `crates/ml/src/cuda_pipeline/backtest_plan_kernel.cu` matching site + +```cuda +// SP11 (current): symmetric bounded +float capped_pnl = fmaxf(-10.0f, fminf(base_reward, 10.0f)); + +// SP12 v3: asymmetric — 2:1 prospect theory loss aversion +float capped_pnl = fmaxf(REWARD_NEG_CAP, fminf(base_reward, REWARD_POS_CAP)); +// -10.0f +5.0f +``` + +**Rationale**: Kahneman/Tversky meta-analysis pegs human loss aversion at ~2.0-2.25. +Calibration to a known psychological constant, not arbitrary tuning. Preserves SP11's +stability work (still bounded both sides). Restores selectivity without requiring controllers. + +### Design 2 — Min-hold soft penalty with temperature curriculum + +**File**: `crates/ml/src/cuda_pipeline/experience_kernels.cu` — at exit branch + +```cuda +if (exiting_trade && hold_time < min_hold_target) { + float deficit = min_hold_target - hold_time; + /* Soft saturation: factor → 1 as deficit → ∞, factor → 0 as deficit → 0. + * Temperature controls smoothness — higher T = wider transition. */ + float soft_factor = deficit / (deficit + temperature); + float penalty = penalty_max * soft_factor; + reward -= penalty; +} +``` + +**Parameters** (Phase 1 — static constants): +- `min_hold_target = 30` bars (MFT-leaning; 100 = full MFT, 10 = HFT-tolerant) +- `temperature` = annealing schedule: `T_start=50, T_end=5, decay over 50 epochs` +- `penalty_max = 3.0` (60% of asymmetric pos_cap; meaningful but not paralyzing) + +**Why temperature annealing**: early training needs forgiveness (model can experiment with +short trades to learn reward gradients). Late training needs sharpness (enforce MFT +commitment). Standard simulated-annealing curriculum: +``` +T(epoch) = T_end + (T_start - T_end) * exp(-epoch / decay_rate) + ≈ 50 at epoch 0, 5 at epoch 50, smooth transition between +``` + +ISV slot for temperature is constructor-set per epoch (not adaptive controller). Avoids +oscillation, gives clean curriculum. + +### Design 3 — Zero per-bar shaping (event-driven density) + +**File**: `crates/ml/src/cuda_pipeline/experience_kernels.cu` + +In the per-bar reward composition path, gate `micro_reward` and `opp_cost` on `exiting_trade +|| entering_trade`: + +```cuda +// SP11 (current): fires every bar with position +r_micro = -shaping_scale * holding_cost_rate * fabsf(position); + +// SP12 v3: fires only on trade events +if (exiting_trade || entering_trade) { + r_micro = -shaping_scale * holding_cost_rate * fabsf(position) * hold_time; + /* Apply accumulated cost as one-shot at trade boundary instead of per-bar. + * Mathematically equivalent under γ=1, slightly different under γ<1 due to + * discount, but discount difference is small for short trades. */ +} +``` + +Same pattern for `r_micro` (micro_reward) — gate on event, not continuous. + +**Rationale**: per-bar shaping creates the implicit "exposure-positive" gradient +(`pearl_event_driven_reward_density_alignment`). Removing it eliminates HFT pull at the +source. The `opp_cost` accumulation effect is preserved (lump-sum at exit) without the +per-bar density bias. + +### Combined: per-trade event-driven reward chain + +After all three changes, reward fires at: + +| Event | Reward | +|-------|--------| +| Entering trade (hold_time = 0) | 0 (or small entry friction in Phase 2) | +| Within trade (no event) | **0** (this is the architectural change) | +| Trail-fire forced exit | `capped_pnl_asymmetric ± min_hold_penalty + accumulated_opp_cost` | +| Voluntary exit | `capped_pnl_asymmetric ± min_hold_penalty + accumulated_opp_cost` | +| Stop-loss / capital floor | existing penalty (unchanged) | + +This is the cleanest possible per-trade event-driven reward design. + +## Implementation phases + +### Phase 1 — single atomic commit (~50 LOC) + +All three changes together as one atomic commit per `feedback_no_partial_refactor`: + +**Files**: +- `crates/ml/src/cuda_pipeline/experience_kernels.cu` — asymmetric cap (line 2788) + min-hold + penalty at exit + zero per-bar shaping (gate micro/opp_cost on events) +- `crates/ml/src/cuda_pipeline/backtest_plan_kernel.cu` — asymmetric cap matching change +- `crates/ml/src/cuda_pipeline/state_layout.cuh` (or equivalent header) — new constants: + `REWARD_POS_CAP`, `REWARD_NEG_CAP`, `MIN_HOLD_TARGET`, `MIN_HOLD_PENALTY_MAX`, + `MIN_HOLD_TEMPERATURE_START`, `MIN_HOLD_TEMPERATURE_END`, `MIN_HOLD_TEMPERATURE_DECAY` +- `crates/ml/src/trainers/dqn/trainer/training_loop.rs` — pass current temperature to env_step + kernel based on epoch (annealing computation) + +**LOC estimate**: ~50 across all files. + +**Tests**: +- GPU oracle test confirming `capped_pnl(20) = 5`, `capped_pnl(-20) = -10` +- GPU oracle test confirming min-hold penalty: `hold_time=0, target=30, T=10 → penalty=penalty_max × 30/40 = 0.75 × penalty_max` +- GPU oracle test confirming per-bar reward = 0 when `!exiting && !entering` and in-position + +### Phase 2 (deferred — only if Phase 1 results indicate need) + +ISV-driven adaptation: +- `MIN_HOLD_TARGET_INDEX` — adaptive based on observed best-PF hold-time bucket +- `LOSS_AVERSION_RATIO_INDEX` — adaptive based on observed trade rate + +Don't build speculatively. Phase 1 results dictate. + +## Validation + +### Smoke (5-epoch L40S, ~€0.30) + +**Must pass**: +- Reward distribution: `max(reward) ≤ +5 + ε`, `min(reward) ≥ -10 - max_min_hold_penalty - ε` +- Slot 63 PopArt EMA stays bounded (will be lower than SP11 since most bars now have reward=0) +- Mean(weights) ≈ 1.0 +- No NaN/inf +- Training converges (loss not exploding) +- Trade rate visibly different (smoke can't fully assess MFT but trend should be visible) + +### Full validation (30-epoch L40S, ~€1.00) + +Compare against SP11 baseline `train-multi-seed-pmbwn` 8-epoch trajectory (the only baseline +we have): + +**Behavioral targets** (success): +- trade_count drops 40-70% (target: 40-80k vs 134k baseline) +- mean hold_time increases (target: 30-100 bars vs current ~5-10 bars) +- win_rate ≥ 52% (vs SP11 baseline 46%) +- profit_factor ≥ 1.4 (vs SP11 baseline 1.25) +- total_pnl ≥ 0.45 (vs SP11 baseline declining toward 0.35; longer trades should yield bigger per-trade PnL) +- max_drawdown ≤ 0.0030 (loss aversion may slightly increase per-trade DD, fine) + +**Acceptable**: +- sharpe in 30-80 range (sharpe-gaming was the problem; absolute PnL is the goal) + +**Failure modes**: +- Training diverges or loss explodes → reward density change too aggressive; need gentler + transition (e.g., ramp shaping_scale instead of zero-ing) +- Model paralyzed (trade_count → 0) → constraints too strong; reduce min_hold_target or + pos_cap aggressiveness +- No behavior change → constraints not reaching policy; debug ISV plumbing or smaller + Q-target effects than expected +- Slot 63 inflation OR NaN → re-introduces SP11 pathology. STOP, debug. + +## Operating principles (NON-NEGOTIABLE) + +- **feedback_no_atomicadd**: not relevant, no new producers +- **feedback_no_partial_refactor**: Phase 1 single atomic commit +- **feedback_isv_for_adaptive_bounds**: Phase 1 uses Invariant-1 numerical anchors as + static constants. Phase 2 (if triggered) lifts to ISV-driven controllers per pearl. +- **feedback_no_quickfixes**: this IS the proper architectural fix per both pearls +- **feedback_no_feature_flags**: behavior change unconditional +- **pearl_event_driven_reward_density_alignment**: drives the per-bar removal +- **pearl_audit_unboundedness_for_implicit_asymmetry**: drives the asymmetric cap + +## Estimated cost + +**Phase 1**: +- Implementation: ~1-2 hours (50 LOC across 3-4 files + 3 GPU oracle tests) +- Smoke validation: ~€0.30 (5 epochs) +- Full validation: ~€1.00 (30 epochs) +- **Total: ~€1.30** + +**Phase 2 (if triggered)**: +- ISV slots + 1-2 controllers: ~€2-3 additional + +vs original v1 (€4) and v2 (€1.30 partial). v3 is comparable cost to v2 but addresses ALL +three causes in one shot. + +## Open questions + +1. **Static constants in Phase 1 — values?** + - Asymmetric cap: -10 / +5 (2:1 prospect theory). My take: keep. + - min_hold_target: 30 bars (MFT-leaning). User goal "between HFT and MFT" suggests this + is the right ballpark. Could be 20 (more HFT) or 50 (more MFT). My take: 30 as default. + - penalty_max: 3.0 (60% of pos_cap). My take: keep. + - Temperature schedule: T_start=50, T_end=5, decay=20 epochs. My take: keep. + +2. **Should opp_cost preserve as lump-sum at exit, or zero entirely?** + - Lump-sum: `accumulated_opp_cost = -k × |position| × hold_time` applied at exit. Preserves + the carry-cost concept. + - Zero: just remove opp_cost entirely. Simpler. + - My take: lump-sum at exit. Carry cost is a real economic concept; preserving it as + event-driven respects pearl_event_driven_reward_density_alignment without losing the + signal. + +3. **micro_reward — zero entirely or move to event?** + - micro_reward was a per-bar "looking good" shaping reward. Its role is dubious. + - Zero entirely: simplest, aligned with pearl. + - Move to event: complex, signal value unclear post-removal. + - My take: zero entirely. Per pearl, per-bar shaping for credit assignment is the + anti-pattern. Q-learning will handle credit assignment via TD targets. + +4. **Min-hold penalty applied to ALL exits or only voluntary?** + - All exits: trail-fire forced exits ALSO get penalty if early. Could conflict with + stop-loss design. + - Voluntary only: trail-fire / stop-loss exempted. + - My take: voluntary only. Trail-stop is a forced exit by design — penalizing it would + fight the existing risk-management logic. + +5. **Worktree fork strategy?** + - SP12 builds on SP11 cap fix. Fork `sp12-event-driven-reward` from + `sp11-reward-as-controlled-subsystem` HEAD (`6a259942e`). + - Keep SP11 worktree clean for any retroactive amendments. + +## Decision needed + +Approve v3 spec → write minimal implementation plan → dispatch implementer with worktree +fork → smoke → 30-epoch validate → close-out commit (audit doc + memory pearl +empirical-validation update).