# SP12 — Per-Trade Event-Driven Reward Composition

**Date**: 2026-05-04 (v3 — unified single-spec solution)
**Branch**: continues on `sp11-reward-as-controlled-subsystem` (SP12 is architectural continuation of SP11; no fork)
**Builds on**: SP11 B1b symmetric reward cap (commits `35db31089`, `348f6078b`, `6a259942e`)
**Anchor pearls**:
- `pearl_event_driven_reward_density_alignment` (the architectural insight)
- `pearl_audit_unboundedness_for_implicit_asymmetry` (the asymmetric-cap component)

**Supersedes**: SP12 v1 (kitchen sink) and v2 (asymmetric cap only). v3 is the unified
architectural design that addresses all three causes of HFT-pull as a single coherent
solution.

## TL;DR

Reshape reward composition to be **purely per-trade event-driven** (matches objective
density). Three changes in one commit:

1. **Asymmetric bounded cap** (loss aversion at exit events)
2. **Min-hold soft penalty with temperature** (commitment at exit events)
3. **Zero per-bar shaping rewards** (no continuous reward gradient)

Combined: model only gets reward on trade events, with prospect-theory loss aversion and
patience requirement. ~50 LOC. Direct attack on MFT goal.

## Background

50-epoch validation `train-multi-seed-pmbwn` on commit `6a259942e` revealed sharpe-gaming:

| Epoch | sharpe | total_pnl | trade_count | win_rate |
|-------|--------|-----------|-------------|----------|
| 0     | 75.19  | 0.5015    | 134,348     | 0.4601   |
| 8     | 75.42  | 0.3502    | 123,970     | 0.4637   |

Sharpe held while PnL declined 30%. Trade rate 62% (134k/214k bars) — pure HFT noise
extraction territory. User goal: MFT-leaning trading (~20-30% trade rate, 55-60% win rate,
larger per-trade PnL, longer holds).

## Three causes of HFT pull (architecturally distinct)

| Cause | Mechanism | Pearl | Phase 1 fix |
|-------|-----------|-------|-------------|
| 1. Lost loss aversion | SP11 symmetric cap erased pre-fix downside-asymmetry → risk-neutral policy | `pearl_audit_unboundedness_for_implicit_asymmetry` | Asymmetric cap |
| 2. Per-bar reward gradient | `micro_reward` + `opp_cost` fire every bar → model rewards continuous exposure | `pearl_event_driven_reward_density_alignment` | Zero per-bar shaping |
| 3. No commitment requirement | Same-bar entry/exit possible without penalty → fast scalps | (none yet — may emerge from validation) | Min-hold soft penalty |

Each acts at a different point in the reward pipeline. Together they constitute the unified
per-trade event-driven design.

## The unified design

### Design 1 — Asymmetric bounded cap (loss aversion)

**File**: `crates/ml/src/cuda_pipeline/experience_kernels.cu` line ~2788
**File**: `crates/ml/src/cuda_pipeline/backtest_plan_kernel.cu` matching site

```cuda
// SP11 (current): symmetric bounded
float capped_pnl = fmaxf(-10.0f, fminf(base_reward, 10.0f));

// SP12 v3: asymmetric — 2:1 prospect theory loss aversion
float capped_pnl = fmaxf(REWARD_NEG_CAP, fminf(base_reward, REWARD_POS_CAP));
//                       -10.0f                              +5.0f
```

**Rationale**: Kahneman/Tversky meta-analysis pegs human loss aversion at ~2.0-2.25.
Calibration to a known psychological constant, not arbitrary tuning. Preserves SP11's
stability work (still bounded both sides). Restores selectivity without requiring controllers.

### Design 2 — Min-hold soft penalty with temperature curriculum

**File**: `crates/ml/src/cuda_pipeline/experience_kernels.cu` — at exit branch

```cuda
if (exiting_trade && hold_time < min_hold_target) {
    float deficit = min_hold_target - hold_time;
    /* Soft saturation: factor → 1 as deficit → ∞, factor → 0 as deficit → 0.
     * Temperature controls smoothness — higher T = wider transition. */
    float soft_factor = deficit / (deficit + temperature);
    float penalty = penalty_max * soft_factor;
    reward -= penalty;
}
```

**Parameters** (Phase 1 — static constants):
- `min_hold_target = 30` bars (MFT-leaning; 100 = full MFT, 10 = HFT-tolerant)
- `temperature` = annealing schedule: `T_start=50, T_end=5, decay over 50 epochs`
- `penalty_max = 3.0` (60% of asymmetric pos_cap; meaningful but not paralyzing)

**Why temperature annealing**: early training needs forgiveness (model can experiment with
short trades to learn reward gradients). Late training needs sharpness (enforce MFT
commitment). Standard simulated-annealing curriculum:
```
T(epoch) = T_end + (T_start - T_end) * exp(-epoch / decay_rate)
         ≈ 50 at epoch 0, 5 at epoch 50, smooth transition between
```

ISV slot for temperature is constructor-set per epoch (not adaptive controller). Avoids
oscillation, gives clean curriculum.

### Design 3 — Zero per-bar shaping (event-driven density)

**File**: `crates/ml/src/cuda_pipeline/experience_kernels.cu`

In the per-bar reward composition path, gate `micro_reward` and `opp_cost` on `exiting_trade
|| entering_trade`:

```cuda
// SP11 (current): fires every bar with position
r_micro = -shaping_scale * holding_cost_rate * fabsf(position);

// SP12 v3: fires only on trade events
if (exiting_trade || entering_trade) {
    r_micro = -shaping_scale * holding_cost_rate * fabsf(position) * hold_time;
    /* Apply accumulated cost as one-shot at trade boundary instead of per-bar.
     * Mathematically equivalent under γ=1, slightly different under γ<1 due to
     * discount, but discount difference is small for short trades. */
}
```

Same pattern for `r_micro` (micro_reward) — gate on event, not continuous.

**Rationale**: per-bar shaping creates the implicit "exposure-positive" gradient
(`pearl_event_driven_reward_density_alignment`). Removing it eliminates HFT pull at the
source. The `opp_cost` accumulation effect is preserved (lump-sum at exit) without the
per-bar density bias.

### Combined: per-trade event-driven reward chain

After all three changes, reward fires at:

| Event | Reward |
|-------|--------|
| Entering trade (hold_time = 0) | 0 (or small entry friction in Phase 2) |
| Within trade (no event) | **0** (this is the architectural change) |
| Trail-fire forced exit | `capped_pnl_asymmetric ± min_hold_penalty + accumulated_opp_cost` |
| Voluntary exit | `capped_pnl_asymmetric ± min_hold_penalty + accumulated_opp_cost` |
| Stop-loss / capital floor | existing penalty (unchanged) |

This is the cleanest possible per-trade event-driven reward design.

## Implementation phases

### Phase 1 — single atomic commit (~50 LOC)

All three changes together as one atomic commit per `feedback_no_partial_refactor`:

**Files**:
- `crates/ml/src/cuda_pipeline/experience_kernels.cu` — asymmetric cap (line 2788) + min-hold
  penalty at exit + zero per-bar shaping (gate micro/opp_cost on events)
- `crates/ml/src/cuda_pipeline/backtest_plan_kernel.cu` — asymmetric cap matching change
- `crates/ml/src/cuda_pipeline/state_layout.cuh` (or equivalent header) — new constants:
  `REWARD_POS_CAP`, `REWARD_NEG_CAP`, `MIN_HOLD_TARGET`, `MIN_HOLD_PENALTY_MAX`,
  `MIN_HOLD_TEMPERATURE_START`, `MIN_HOLD_TEMPERATURE_END`, `MIN_HOLD_TEMPERATURE_DECAY`
- `crates/ml/src/trainers/dqn/trainer/training_loop.rs` — pass current temperature to env_step
  kernel based on epoch (annealing computation)

**LOC estimate**: ~50 across all files.

**Tests**:
- GPU oracle test confirming `capped_pnl(20) = 5`, `capped_pnl(-20) = -10`
- GPU oracle test confirming min-hold penalty: `hold_time=0, target=30, T=10 → penalty=penalty_max × 30/40 = 0.75 × penalty_max`
- GPU oracle test confirming per-bar reward = 0 when `!exiting && !entering` and in-position

### Phase 2 (deferred — only if Phase 1 results indicate need)

ISV-driven adaptation:
- `MIN_HOLD_TARGET_INDEX` — adaptive based on observed best-PF hold-time bucket
- `LOSS_AVERSION_RATIO_INDEX` — adaptive based on observed trade rate

Don't build speculatively. Phase 1 results dictate.

## Validation

### Smoke (5-epoch L40S, ~€0.30)

**Must pass**:
- Reward distribution: `max(reward) ≤ +5 + ε`, `min(reward) ≥ -10 - max_min_hold_penalty - ε`
- Slot 63 PopArt EMA stays bounded (will be lower than SP11 since most bars now have reward=0)
- Mean(weights) ≈ 1.0
- No NaN/inf
- Training converges (loss not exploding)
- Trade rate visibly different (smoke can't fully assess MFT but trend should be visible)

### Full validation (30-epoch L40S, ~€1.00)

Compare against SP11 baseline `train-multi-seed-pmbwn` 8-epoch trajectory (the only baseline
we have):

**Behavioral targets** (success):
- trade_count drops 40-70% (target: 40-80k vs 134k baseline)
- mean hold_time increases (target: 30-100 bars vs current ~5-10 bars)
- win_rate ≥ 52% (vs SP11 baseline 46%)
- profit_factor ≥ 1.4 (vs SP11 baseline 1.25)
- total_pnl ≥ 0.45 (vs SP11 baseline declining toward 0.35; longer trades should yield bigger per-trade PnL)
- max_drawdown ≤ 0.0030 (loss aversion may slightly increase per-trade DD, fine)

**Acceptable**:
- sharpe in 30-80 range (sharpe-gaming was the problem; absolute PnL is the goal)

**Failure modes**:
- Training diverges or loss explodes → reward density change too aggressive; need gentler
  transition (e.g., ramp shaping_scale instead of zero-ing)
- Model paralyzed (trade_count → 0) → constraints too strong; reduce min_hold_target or
  pos_cap aggressiveness
- No behavior change → constraints not reaching policy; debug ISV plumbing or smaller
  Q-target effects than expected
- Slot 63 inflation OR NaN → re-introduces SP11 pathology. STOP, debug.

## Operating principles (NON-NEGOTIABLE)

- **feedback_no_atomicadd**: not relevant, no new producers
- **feedback_no_partial_refactor**: Phase 1 single atomic commit
- **feedback_isv_for_adaptive_bounds**: Phase 1 uses Invariant-1 numerical anchors as
  static constants. Phase 2 (if triggered) lifts to ISV-driven controllers per pearl.
- **feedback_no_quickfixes**: this IS the proper architectural fix per both pearls
- **feedback_no_feature_flags**: behavior change unconditional
- **pearl_event_driven_reward_density_alignment**: drives the per-bar removal
- **pearl_audit_unboundedness_for_implicit_asymmetry**: drives the asymmetric cap

## Estimated cost

**Phase 1**:
- Implementation: ~1-2 hours (50 LOC across 3-4 files + 3 GPU oracle tests)
- Smoke validation: ~€0.30 (5 epochs)
- Full validation: ~€1.00 (30 epochs)
- **Total: ~€1.30**

**Phase 2 (if triggered)**:
- ISV slots + 1-2 controllers: ~€2-3 additional

vs original v1 (€4) and v2 (€1.30 partial). v3 is comparable cost to v2 but addresses ALL
three causes in one shot.

## Open questions

1. **Static constants in Phase 1 — values?**
   - Asymmetric cap: -10 / +5 (2:1 prospect theory). My take: keep.
   - min_hold_target: 30 bars (MFT-leaning). User goal "between HFT and MFT" suggests this
     is the right ballpark. Could be 20 (more HFT) or 50 (more MFT). My take: 30 as default.
   - penalty_max: 3.0 (60% of pos_cap). My take: keep.
   - Temperature schedule: T_start=50, T_end=5, decay=20 epochs. My take: keep.

2. **Should opp_cost preserve as lump-sum at exit, or zero entirely?**
   - Lump-sum: `accumulated_opp_cost = -k × |position| × hold_time` applied at exit. Preserves
     the carry-cost concept.
   - Zero: just remove opp_cost entirely. Simpler.
   - My take: lump-sum at exit. Carry cost is a real economic concept; preserving it as
     event-driven respects pearl_event_driven_reward_density_alignment without losing the
     signal.

3. **micro_reward — zero entirely or move to event?**
   - micro_reward was a per-bar "looking good" shaping reward. Its role is dubious.
   - Zero entirely: simplest, aligned with pearl.
   - Move to event: complex, signal value unclear post-removal.
   - My take: zero entirely. Per pearl, per-bar shaping for credit assignment is the
     anti-pattern. Q-learning will handle credit assignment via TD targets.

4. **Min-hold penalty applied to ALL exits or only voluntary?**
   - All exits: trail-fire forced exits ALSO get penalty if early. Could conflict with
     stop-loss design.
   - Voluntary only: trail-fire / stop-loss exempted.
   - My take: voluntary only. Trail-stop is a forced exit by design — penalizing it would
     fight the existing risk-management logic.

5. **Worktree fork strategy?**
   - SP12 builds on SP11 cap fix. Fork `sp12-event-driven-reward` from
     `sp11-reward-as-controlled-subsystem` HEAD (`6a259942e`).
   - Keep SP11 worktree clean for any retroactive amendments.

## Decision needed

Approve v3 spec → write minimal implementation plan → dispatch implementer with worktree
fork → smoke → 30-epoch validate → close-out commit (audit doc + memory pearl
empirical-validation update).
