spec(sp12): per-trade event-driven reward composition (v3)
Three architectural changes in one unified design to push the model from
HFT noise extraction (62% trade rate, sharpe-gaming) toward MFT alpha-hunting:
1. Asymmetric bounded cap (-10/+5) — restores loss aversion erased by
SP11 symmetric cap. 2:1 ratio matches Kahneman/Tversky prospect theory.
Anchor: pearl_audit_unboundedness_for_implicit_asymmetry.
2. Min-hold soft penalty with temperature curriculum — patience requirement
at exit. Soft factor = deficit/(deficit+T), T anneals 50→5 over 50 epochs.
Forces commitment without paralysis in early training.
3. Zero per-bar shaping (gate micro/opp_cost on events) — eliminates
continuous-reward gradient that pulls toward continuous exposure.
Anchor: pearl_event_driven_reward_density_alignment.
Combined: reward fires only on trade events with prospect-theory loss
aversion + commitment requirement. Pure per-trade event-driven Q-learning
properly aligned with per-trade P&L objective.
~50 LOC across 3-4 files. No new ISV slots in Phase 1 (constants only).
Cost ~€1.30 (€0.30 smoke + €1.00 30-epoch validation).
Empirical motivation: train-multi-seed-pmbwn 50-epoch on commit 6a259942e
showed sharpe-gaming pattern (PnL -30% over 8 epochs while sharpe held).
SP11 cap fix unmasked the per-bar shaping bias plus erased loss-aversion
that the unbounded loss path was implicitly providing.
Continues on sp11-reward-as-controlled-subsystem branch — SP12 is
architectural continuation of SP11, not separate work.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
@@ -0,0 +1,275 @@
|
||||
# SP12 — Per-Trade Event-Driven Reward Composition
|
||||
|
||||
**Date**: 2026-05-04 (v3 — unified single-spec solution)
|
||||
**Branch**: continues on `sp11-reward-as-controlled-subsystem` (SP12 is architectural continuation of SP11; no fork)
|
||||
**Builds on**: SP11 B1b symmetric reward cap (commits `35db31089`, `348f6078b`, `6a259942e`)
|
||||
**Anchor pearls**:
|
||||
- `pearl_event_driven_reward_density_alignment` (the architectural insight)
|
||||
- `pearl_audit_unboundedness_for_implicit_asymmetry` (the asymmetric-cap component)
|
||||
|
||||
**Supersedes**: SP12 v1 (kitchen sink) and v2 (asymmetric cap only). v3 is the unified
|
||||
architectural design that addresses all three causes of HFT-pull as a single coherent
|
||||
solution.
|
||||
|
||||
## TL;DR
|
||||
|
||||
Reshape reward composition to be **purely per-trade event-driven** (matches objective
|
||||
density). Three changes in one commit:
|
||||
|
||||
1. **Asymmetric bounded cap** (loss aversion at exit events)
|
||||
2. **Min-hold soft penalty with temperature** (commitment at exit events)
|
||||
3. **Zero per-bar shaping rewards** (no continuous reward gradient)
|
||||
|
||||
Combined: model only gets reward on trade events, with prospect-theory loss aversion and
|
||||
patience requirement. ~50 LOC. Direct attack on MFT goal.
|
||||
|
||||
## Background
|
||||
|
||||
50-epoch validation `train-multi-seed-pmbwn` on commit `6a259942e` revealed sharpe-gaming:
|
||||
|
||||
| Epoch | sharpe | total_pnl | trade_count | win_rate |
|
||||
|-------|--------|-----------|-------------|----------|
|
||||
| 0 | 75.19 | 0.5015 | 134,348 | 0.4601 |
|
||||
| 8 | 75.42 | 0.3502 | 123,970 | 0.4637 |
|
||||
|
||||
Sharpe held while PnL declined 30%. Trade rate 62% (134k/214k bars) — pure HFT noise
|
||||
extraction territory. User goal: MFT-leaning trading (~20-30% trade rate, 55-60% win rate,
|
||||
larger per-trade PnL, longer holds).
|
||||
|
||||
## Three causes of HFT pull (architecturally distinct)
|
||||
|
||||
| Cause | Mechanism | Pearl | Phase 1 fix |
|
||||
|-------|-----------|-------|-------------|
|
||||
| 1. Lost loss aversion | SP11 symmetric cap erased pre-fix downside-asymmetry → risk-neutral policy | `pearl_audit_unboundedness_for_implicit_asymmetry` | Asymmetric cap |
|
||||
| 2. Per-bar reward gradient | `micro_reward` + `opp_cost` fire every bar → model rewards continuous exposure | `pearl_event_driven_reward_density_alignment` | Zero per-bar shaping |
|
||||
| 3. No commitment requirement | Same-bar entry/exit possible without penalty → fast scalps | (none yet — may emerge from validation) | Min-hold soft penalty |
|
||||
|
||||
Each acts at a different point in the reward pipeline. Together they constitute the unified
|
||||
per-trade event-driven design.
|
||||
|
||||
## The unified design
|
||||
|
||||
### Design 1 — Asymmetric bounded cap (loss aversion)
|
||||
|
||||
**File**: `crates/ml/src/cuda_pipeline/experience_kernels.cu` line ~2788
|
||||
**File**: `crates/ml/src/cuda_pipeline/backtest_plan_kernel.cu` matching site
|
||||
|
||||
```cuda
|
||||
// SP11 (current): symmetric bounded
|
||||
float capped_pnl = fmaxf(-10.0f, fminf(base_reward, 10.0f));
|
||||
|
||||
// SP12 v3: asymmetric — 2:1 prospect theory loss aversion
|
||||
float capped_pnl = fmaxf(REWARD_NEG_CAP, fminf(base_reward, REWARD_POS_CAP));
|
||||
// -10.0f +5.0f
|
||||
```
|
||||
|
||||
**Rationale**: Kahneman/Tversky meta-analysis pegs human loss aversion at ~2.0-2.25.
|
||||
Calibration to a known psychological constant, not arbitrary tuning. Preserves SP11's
|
||||
stability work (still bounded both sides). Restores selectivity without requiring controllers.
|
||||
|
||||
### Design 2 — Min-hold soft penalty with temperature curriculum
|
||||
|
||||
**File**: `crates/ml/src/cuda_pipeline/experience_kernels.cu` — at exit branch
|
||||
|
||||
```cuda
|
||||
if (exiting_trade && hold_time < min_hold_target) {
|
||||
float deficit = min_hold_target - hold_time;
|
||||
/* Soft saturation: factor → 1 as deficit → ∞, factor → 0 as deficit → 0.
|
||||
* Temperature controls smoothness — higher T = wider transition. */
|
||||
float soft_factor = deficit / (deficit + temperature);
|
||||
float penalty = penalty_max * soft_factor;
|
||||
reward -= penalty;
|
||||
}
|
||||
```
|
||||
|
||||
**Parameters** (Phase 1 — static constants):
|
||||
- `min_hold_target = 30` bars (MFT-leaning; 100 = full MFT, 10 = HFT-tolerant)
|
||||
- `temperature` = annealing schedule: `T_start=50, T_end=5, decay over 50 epochs`
|
||||
- `penalty_max = 3.0` (60% of asymmetric pos_cap; meaningful but not paralyzing)
|
||||
|
||||
**Why temperature annealing**: early training needs forgiveness (model can experiment with
|
||||
short trades to learn reward gradients). Late training needs sharpness (enforce MFT
|
||||
commitment). Standard simulated-annealing curriculum:
|
||||
```
|
||||
T(epoch) = T_end + (T_start - T_end) * exp(-epoch / decay_rate)
|
||||
≈ 50 at epoch 0, 5 at epoch 50, smooth transition between
|
||||
```
|
||||
|
||||
ISV slot for temperature is constructor-set per epoch (not adaptive controller). Avoids
|
||||
oscillation, gives clean curriculum.
|
||||
|
||||
### Design 3 — Zero per-bar shaping (event-driven density)
|
||||
|
||||
**File**: `crates/ml/src/cuda_pipeline/experience_kernels.cu`
|
||||
|
||||
In the per-bar reward composition path, gate `micro_reward` and `opp_cost` on `exiting_trade
|
||||
|| entering_trade`:
|
||||
|
||||
```cuda
|
||||
// SP11 (current): fires every bar with position
|
||||
r_micro = -shaping_scale * holding_cost_rate * fabsf(position);
|
||||
|
||||
// SP12 v3: fires only on trade events
|
||||
if (exiting_trade || entering_trade) {
|
||||
r_micro = -shaping_scale * holding_cost_rate * fabsf(position) * hold_time;
|
||||
/* Apply accumulated cost as one-shot at trade boundary instead of per-bar.
|
||||
* Mathematically equivalent under γ=1, slightly different under γ<1 due to
|
||||
* discount, but discount difference is small for short trades. */
|
||||
}
|
||||
```
|
||||
|
||||
Same pattern for `r_micro` (micro_reward) — gate on event, not continuous.
|
||||
|
||||
**Rationale**: per-bar shaping creates the implicit "exposure-positive" gradient
|
||||
(`pearl_event_driven_reward_density_alignment`). Removing it eliminates HFT pull at the
|
||||
source. The `opp_cost` accumulation effect is preserved (lump-sum at exit) without the
|
||||
per-bar density bias.
|
||||
|
||||
### Combined: per-trade event-driven reward chain
|
||||
|
||||
After all three changes, reward fires at:
|
||||
|
||||
| Event | Reward |
|
||||
|-------|--------|
|
||||
| Entering trade (hold_time = 0) | 0 (or small entry friction in Phase 2) |
|
||||
| Within trade (no event) | **0** (this is the architectural change) |
|
||||
| Trail-fire forced exit | `capped_pnl_asymmetric ± min_hold_penalty + accumulated_opp_cost` |
|
||||
| Voluntary exit | `capped_pnl_asymmetric ± min_hold_penalty + accumulated_opp_cost` |
|
||||
| Stop-loss / capital floor | existing penalty (unchanged) |
|
||||
|
||||
This is the cleanest possible per-trade event-driven reward design.
|
||||
|
||||
## Implementation phases
|
||||
|
||||
### Phase 1 — single atomic commit (~50 LOC)
|
||||
|
||||
All three changes together as one atomic commit per `feedback_no_partial_refactor`:
|
||||
|
||||
**Files**:
|
||||
- `crates/ml/src/cuda_pipeline/experience_kernels.cu` — asymmetric cap (line 2788) + min-hold
|
||||
penalty at exit + zero per-bar shaping (gate micro/opp_cost on events)
|
||||
- `crates/ml/src/cuda_pipeline/backtest_plan_kernel.cu` — asymmetric cap matching change
|
||||
- `crates/ml/src/cuda_pipeline/state_layout.cuh` (or equivalent header) — new constants:
|
||||
`REWARD_POS_CAP`, `REWARD_NEG_CAP`, `MIN_HOLD_TARGET`, `MIN_HOLD_PENALTY_MAX`,
|
||||
`MIN_HOLD_TEMPERATURE_START`, `MIN_HOLD_TEMPERATURE_END`, `MIN_HOLD_TEMPERATURE_DECAY`
|
||||
- `crates/ml/src/trainers/dqn/trainer/training_loop.rs` — pass current temperature to env_step
|
||||
kernel based on epoch (annealing computation)
|
||||
|
||||
**LOC estimate**: ~50 across all files.
|
||||
|
||||
**Tests**:
|
||||
- GPU oracle test confirming `capped_pnl(20) = 5`, `capped_pnl(-20) = -10`
|
||||
- GPU oracle test confirming min-hold penalty: `hold_time=0, target=30, T=10 → penalty=penalty_max × 30/40 = 0.75 × penalty_max`
|
||||
- GPU oracle test confirming per-bar reward = 0 when `!exiting && !entering` and in-position
|
||||
|
||||
### Phase 2 (deferred — only if Phase 1 results indicate need)
|
||||
|
||||
ISV-driven adaptation:
|
||||
- `MIN_HOLD_TARGET_INDEX` — adaptive based on observed best-PF hold-time bucket
|
||||
- `LOSS_AVERSION_RATIO_INDEX` — adaptive based on observed trade rate
|
||||
|
||||
Don't build speculatively. Phase 1 results dictate.
|
||||
|
||||
## Validation
|
||||
|
||||
### Smoke (5-epoch L40S, ~€0.30)
|
||||
|
||||
**Must pass**:
|
||||
- Reward distribution: `max(reward) ≤ +5 + ε`, `min(reward) ≥ -10 - max_min_hold_penalty - ε`
|
||||
- Slot 63 PopArt EMA stays bounded (will be lower than SP11 since most bars now have reward=0)
|
||||
- Mean(weights) ≈ 1.0
|
||||
- No NaN/inf
|
||||
- Training converges (loss not exploding)
|
||||
- Trade rate visibly different (smoke can't fully assess MFT but trend should be visible)
|
||||
|
||||
### Full validation (30-epoch L40S, ~€1.00)
|
||||
|
||||
Compare against SP11 baseline `train-multi-seed-pmbwn` 8-epoch trajectory (the only baseline
|
||||
we have):
|
||||
|
||||
**Behavioral targets** (success):
|
||||
- trade_count drops 40-70% (target: 40-80k vs 134k baseline)
|
||||
- mean hold_time increases (target: 30-100 bars vs current ~5-10 bars)
|
||||
- win_rate ≥ 52% (vs SP11 baseline 46%)
|
||||
- profit_factor ≥ 1.4 (vs SP11 baseline 1.25)
|
||||
- total_pnl ≥ 0.45 (vs SP11 baseline declining toward 0.35; longer trades should yield bigger per-trade PnL)
|
||||
- max_drawdown ≤ 0.0030 (loss aversion may slightly increase per-trade DD, fine)
|
||||
|
||||
**Acceptable**:
|
||||
- sharpe in 30-80 range (sharpe-gaming was the problem; absolute PnL is the goal)
|
||||
|
||||
**Failure modes**:
|
||||
- Training diverges or loss explodes → reward density change too aggressive; need gentler
|
||||
transition (e.g., ramp shaping_scale instead of zero-ing)
|
||||
- Model paralyzed (trade_count → 0) → constraints too strong; reduce min_hold_target or
|
||||
pos_cap aggressiveness
|
||||
- No behavior change → constraints not reaching policy; debug ISV plumbing or smaller
|
||||
Q-target effects than expected
|
||||
- Slot 63 inflation OR NaN → re-introduces SP11 pathology. STOP, debug.
|
||||
|
||||
## Operating principles (NON-NEGOTIABLE)
|
||||
|
||||
- **feedback_no_atomicadd**: not relevant, no new producers
|
||||
- **feedback_no_partial_refactor**: Phase 1 single atomic commit
|
||||
- **feedback_isv_for_adaptive_bounds**: Phase 1 uses Invariant-1 numerical anchors as
|
||||
static constants. Phase 2 (if triggered) lifts to ISV-driven controllers per pearl.
|
||||
- **feedback_no_quickfixes**: this IS the proper architectural fix per both pearls
|
||||
- **feedback_no_feature_flags**: behavior change unconditional
|
||||
- **pearl_event_driven_reward_density_alignment**: drives the per-bar removal
|
||||
- **pearl_audit_unboundedness_for_implicit_asymmetry**: drives the asymmetric cap
|
||||
|
||||
## Estimated cost
|
||||
|
||||
**Phase 1**:
|
||||
- Implementation: ~1-2 hours (50 LOC across 3-4 files + 3 GPU oracle tests)
|
||||
- Smoke validation: ~€0.30 (5 epochs)
|
||||
- Full validation: ~€1.00 (30 epochs)
|
||||
- **Total: ~€1.30**
|
||||
|
||||
**Phase 2 (if triggered)**:
|
||||
- ISV slots + 1-2 controllers: ~€2-3 additional
|
||||
|
||||
vs original v1 (€4) and v2 (€1.30 partial). v3 is comparable cost to v2 but addresses ALL
|
||||
three causes in one shot.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. **Static constants in Phase 1 — values?**
|
||||
- Asymmetric cap: -10 / +5 (2:1 prospect theory). My take: keep.
|
||||
- min_hold_target: 30 bars (MFT-leaning). User goal "between HFT and MFT" suggests this
|
||||
is the right ballpark. Could be 20 (more HFT) or 50 (more MFT). My take: 30 as default.
|
||||
- penalty_max: 3.0 (60% of pos_cap). My take: keep.
|
||||
- Temperature schedule: T_start=50, T_end=5, decay=20 epochs. My take: keep.
|
||||
|
||||
2. **Should opp_cost preserve as lump-sum at exit, or zero entirely?**
|
||||
- Lump-sum: `accumulated_opp_cost = -k × |position| × hold_time` applied at exit. Preserves
|
||||
the carry-cost concept.
|
||||
- Zero: just remove opp_cost entirely. Simpler.
|
||||
- My take: lump-sum at exit. Carry cost is a real economic concept; preserving it as
|
||||
event-driven respects pearl_event_driven_reward_density_alignment without losing the
|
||||
signal.
|
||||
|
||||
3. **micro_reward — zero entirely or move to event?**
|
||||
- micro_reward was a per-bar "looking good" shaping reward. Its role is dubious.
|
||||
- Zero entirely: simplest, aligned with pearl.
|
||||
- Move to event: complex, signal value unclear post-removal.
|
||||
- My take: zero entirely. Per pearl, per-bar shaping for credit assignment is the
|
||||
anti-pattern. Q-learning will handle credit assignment via TD targets.
|
||||
|
||||
4. **Min-hold penalty applied to ALL exits or only voluntary?**
|
||||
- All exits: trail-fire forced exits ALSO get penalty if early. Could conflict with
|
||||
stop-loss design.
|
||||
- Voluntary only: trail-fire / stop-loss exempted.
|
||||
- My take: voluntary only. Trail-stop is a forced exit by design — penalizing it would
|
||||
fight the existing risk-management logic.
|
||||
|
||||
5. **Worktree fork strategy?**
|
||||
- SP12 builds on SP11 cap fix. Fork `sp12-event-driven-reward` from
|
||||
`sp11-reward-as-controlled-subsystem` HEAD (`6a259942e`).
|
||||
- Keep SP11 worktree clean for any retroactive amendments.
|
||||
|
||||
## Decision needed
|
||||
|
||||
Approve v3 spec → write minimal implementation plan → dispatch implementer with worktree
|
||||
fork → smoke → 30-epoch validate → close-out commit (audit doc + memory pearl
|
||||
empirical-validation update).
|
||||
Reference in New Issue
Block a user