# Trade-Level Reward Attribution + Exploration Risk Budget — Design Spec

**Goal:** Fix two fundamental reward problems: (1) per-bar reward is 99% noise — replace with trade-level P&L attribution for 10-50× higher SNR, (2) the risk protection stack prevents the model from discovering profitable strategies that require temporary risk-taking.

**Problem 1 — Noisy Reward:** The reward is `next_close - close` (1-bar price change). For ES futures, single-bar returns are dominated by random walk noise. The model receives 100 bars of noise per trade, but the TRADE-LEVEL P&L (sum of those bars) is the actual signal. C51's 51 atoms model the distribution of per-bar noise instead of the distribution of trade outcomes.

**Problem 2 — Over-Protection:** The risk protection stack (CVaR + epistemic gate + commitment penalty + cost curriculum + adaptive DSR + Q-anchoring) collectively pushes the model toward inactivity. During EXPLORATION, the model needs to take risky trades to discover which edges survive costs. The protection stack kills exploration before the model can evaluate trade outcomes.

**Architecture:** Two complementary components:
1. **Trade-Level Reward Attribution**: accumulate P&L over trades, assign to entry decision. Per-bar holding reward = 0.
2. **Exploration Risk Budget**: during exploration (high epsilon / low iqn_readiness), relax the protection stack. As the model converges (low epsilon / high readiness), tighten protection.

---

## Component 1: Trade-Level Reward Attribution

### The Problem in Detail

Current reward flow per bar:
```
Bar 1: enter Long  → reward = +0.02% (noise)
Bar 2: hold        → reward = -0.08% (noise)
Bar 3: hold        → reward = +0.15% (noise)
Bar 4: hold        → reward = -0.03% (noise)
Bar 5: exit        → reward = +0.01% (noise)
                     ─────────────────
                     Total: +0.07% (signal, but model never sees this as one number)
```

The model gets 5 separate rewards. The C51 distribution learns: "this state produces returns centered at +0.02% with std=0.08%" — that's the PER-BAR noise distribution, not the trade outcome distribution.

### The Solution

Track trade state in the env_step kernel. When a trade closes (position changes or episode ends):
1. Compute total trade P&L = (exit_price - entry_price) × direction × size - costs
2. Assign trade P&L as reward to the ENTRY bar
3. Holding bars get reward = 0 (no decision to evaluate)

```
Bar 1: enter Long  → reward = 0 (pending — trade not closed yet)
Bar 2: hold        → reward = 0
Bar 3: hold        → reward = 0
Bar 4: hold        → reward = 0
Bar 5: exit        → reward = +0.07% - costs (assigned to bar 1 retroactively via n-step)
```

### Implementation

In `experience_kernels.cu` env_step kernel:

```cuda
// Per-episode trade tracking state (in episode_states buffer):
//   trade_entry_price: float   — price at position entry
//   trade_entry_bar: int       — bar index of entry (for retroactive attribution)
//   trade_direction: int       — +1 Long, -1 Short, 0 Flat

// When position changes:
if (new_position != old_position) {
    if (old_position != 0) {
        // Close existing trade
        float trade_pnl = (current_price - trade_entry_price) 
                        * trade_direction * fabsf(old_position)
                        * contract_multiplier;
        trade_pnl -= spread_cost * 2.0f * cost_scale;  // entry + exit costs
        
        // Trade-level reward replaces per-bar reward
        reward = trade_pnl;
    }
    if (new_position != 0) {
        // Open new trade
        trade_entry_price = current_price;
        trade_direction = (new_position > 0) ? 1 : -1;
    }
} else {
    // Holding — reward = small holding cost (opportunity cost of capital)
    reward = -0.0001f * fabsf(old_position);  // tiny negative to discourage idle holding
}
```

### State Buffers

Need 3 new per-episode floats in the episode state buffer:
- `trade_entry_price: f32` — saved at position entry
- `trade_direction: f32` — +1/-1/0
- `bars_held: i32` — counter for holding period statistics

These are already tracked by the portfolio simulator (`portfolio_states` buffer). The implementation reuses the existing portfolio state rather than adding new buffers.

### Interaction with Existing Reward Pipeline

- **DSR shaping**: operates on trade-level P&L (amplifies drawdown penalty per trade, not per bar)
- **Rank normalization (G13)**: ranks trade P&Ls instead of per-bar noise — MUCH higher signal quality
- **Cost curriculum (G2)**: costs already subtracted from trade P&L
- **Commitment penalty (G15)**: fires at position change (entry/exit) — synergistic
- **Counterfactual flip (G7)**: flips the entire trade, not individual bars — cleaner symmetry

### Backwards Compatibility

The per-bar reward is not removed — it's replaced for bars where a trade closes. Holding bars get reward=0 instead of `next_close - close`. The n-step return computation accumulates these zeros naturally (gamma^k × 0 = 0 for holding bars).

---

## Component 2: Exploration Risk Budget

### The Problem in Detail

The risk protection stack collectively applies:
```
CVaR: alpha=0.3-0.5 → targets bottom 30-50% of returns
Epistemic gate: scales magnitude toward Small when uncertain
Commitment penalty: -0.01 per position change
Cost curriculum: 0-100% spread costs
Adaptive DSR: w_dsr up to 3× when Sharpe negative
Q-anchoring: anchors to Flat (doing nothing = zero penalty)
```

During exploration (epsilon=0.05, iqn_readiness=0.1), the model takes random actions. The protection stack punishes ALL random actions heavily. The model learns: "random actions are bad → stay Flat." But random actions are HOW the model discovers which state→action pairs are profitable.

### The Solution

Scale the protection stack by exploration readiness:

```
protection_scale = iqn_readiness^2  // 0 at start, 1 when converged

CVaR alpha: 0.5 * (1 + protection_scale * 0.8)  // 0.5→0.9 (balanced→risk-averse)
Epistemic gate: threshold *= (1 + 2 * protection_scale)  // looser early, tighter late
Commitment penalty: *= protection_scale  // zero early, full late
DSR multiplier: clamp(1 - sharpe_ema, 0.5, 0.5 + 2.5 * protection_scale)  // narrower early
```

When iqn_readiness=0 (exploring): CVaR=0.5 (balanced), no commitment penalty, no epistemic gate, narrow DSR. The model freely explores.

When iqn_readiness=1 (converged): CVaR=0.9 (risk-averse), full commitment penalty, tight epistemic gate, wide DSR. The model is protected.

### Implementation

Single pinned device-mapped scalar `protection_scale` written from CPU (from iqn_readiness, already computed).

In the CUDA kernels that apply protection:

```cuda
// CVaR (c51_loss_kernel.cu):
float alpha = 0.5f - 0.4f * protection_scale;  // already uses iqn_readiness, just rename

// Epistemic gate (experience_kernels.cu):
float effective_threshold = var_ema * (1.0f + 2.0f * protection_scale);

// Commitment penalty (experience_kernels.cu):
float effective_commitment = 0.01f * protection_scale;

// DSR (training_loop.rs):
let dsr_range = 0.5 + 2.5 * protection_scale as f64;
let adaptive_mult = (1.0 - sharpe_ema).clamp(0.5, 0.5 + dsr_range);
```

Most of these are 1-line changes to existing kernels/Rust code.

---

## Combined Effect

```
Epoch 1-5 (exploring, readiness=0.0-0.2):
  - Trade-level rewards: model sees clean trade P&L signal
  - Protection: minimal — model freely discovers edges
  - Learns: "Long in this regime, held 8 bars, made +$300 net of costs"

Epoch 5-15 (learning, readiness=0.2-0.7):
  - Trade-level rewards: refines trade timing and sizing
  - Protection: ramping — starts penalizing bad trades
  - Learns: "this edge survives costs, this one doesn't"

Epoch 15+ (converged, readiness=0.7-1.0):
  - Trade-level rewards: fine-tunes around profitable strategies
  - Protection: full — conservative risk management
  - Result: OOS-ready strategies that survived cost + risk pressure
```

---

## Success Criteria

| Metric | Current (per-bar, over-protected) | Target (trade-level, exploration budget) |
|--------|----------------------------------|----------------------------------------|
| Training Sharpe by epoch 10 | oscillating ±0.3 | > 1.0 sustained |
| Reward SNR | ~0.01 (per-bar noise) | ~0.1-0.5 (trade-level signal) |
| Trade count | ~60K/epoch (churn) | ~5-15K/epoch (meaningful trades) |
| val/OOS gap | unknown (can't measure at Sharpe~0) | < 15% |
| Exploration trades tried | few (protection kills) | many early, few late |

## Implementation Order

1. Trade-level reward attribution (~30 lines CUDA in env_step)
2. Exploration risk budget (~10 lines across 4 files: 1 line each in CVaR, epistemic gate, commitment, DSR)
3. Trade state tracking (reuse existing portfolio_states buffer)

## Risks

| Risk | Mitigation |
|------|-----------|
| Trade-level reward = sparse signal (only at trade close) | Holding bars get small negative (opportunity cost), not exactly zero |
| Exploration budget too loose → model learns bad habits | protection_scale ramps with iqn_readiness² (quadratic), bites quickly |
| Long holding periods → delayed reward attribution | n-step return (n=config) handles multi-bar attribution naturally |
| Trade tracking adds state to env_step | Reuse portfolio_states buffer (already tracks position, entry price) |
