# Reward v6 Metrics Fixes — Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Fix 5 root causes that prevent the DQN model from learning profitable trading — all in the reward/metrics computation within the CUDA experience kernel.

**Architecture:** Single file change (`experience_kernels.cu`) plus minor config cleanup. All fixes are in the GPU kernel — no CPU paths.

**Tech Stack:** CUDA kernel, compiled via nvcc.

---

## Root Causes (from audit)

| # | Bug | Impact |
|---|-----|--------|
| 1 | CUSUM direction used as vol proxy (binary [-1,1] not volatility) | 10000× reward amplification in calm markets |
| 2 | Loss aversion applied AFTER clamp (breaks ±10 symmetry to ±15) | Expected reward negative even for fair strategy |
| 3 | Hard clamp at ±10 destroys tail information | Model can't distinguish 1% win from 5% win |
| 4 | Turnover penalty double-counts tx costs | Over-penalizes necessary rebalancing |
| 5 | Feature[9] is log-normalized ATR — needs exp() to recover price units | Vol proxy computation needs the right feature |

## The Fix: Replace vol normalization + fix reward ordering

All changes in `crates/ml/src/cuda_pipeline/experience_kernels.cu`, lines ~893-920.

### Before (broken):
```cuda
float vol_proxy = fmaxf(cusum_raw * 0.01f, 0.0001f);           // WRONG: CUSUM is direction, not vol
float vol_norm = vol_proxy * sqrtf(fmaxf(segment_hold_time, 1.0f));
float vol_normalized_return = segment_return / vol_norm;
reward = 10.0f * vol_normalized_return;                          // Can explode to 10000+
reward = fmaxf(-10.0f, fminf(10.0f, reward));                   // Hard clamp kills tails
// ... later ...
if (reward < 0.0f) reward *= loss_aversion;                      // Applied AFTER clamp → -15
// ... later ...
reward -= 0.05f * turnover;                                      // Double-counts tx costs
```

### After (fixed):
```cuda
/* Vol normalization using ATR(14) — proper realized volatility.
 * Feature[9] is log-normalized ATR. exp() recovers price units.
 * Divide by close to get percentage vol. Scale by sqrt(hold_time)
 * for time-adjusted normalization (annualization principle). */
float log_atr = 0.0f;
if (features != NULL && bar_idx < total_bars && market_dim > 9) {
    log_atr = features[(long long)bar_idx * market_dim + 9];
}
float atr_pct = expf(log_atr) / fmaxf(raw_close, 1.0f);
float vol_proxy = fmaxf(atr_pct, 0.0001f);
float vol_norm = vol_proxy * sqrtf(fmaxf(segment_hold_time, 1.0f));
float vol_normalized_return = segment_return / vol_norm;

/* Scale to learnable magnitude. Typical vol-normalized return ≈ 0.1-0.5.
 * REWARD_SCALE=10 puts rewards in [-5, +5] range (SNR ≈ 5000). */
reward = 10.0f * vol_normalized_return;

/* Loss aversion BEFORE clamp (prospect theory: losses loom 1.5x larger).
 * Must happen before clamp so the asymmetry is preserved correctly. */
if (reward < 0.0f) {
    reward *= loss_aversion;
}

/* Soft clamp: tanh squashing preserves tail information.
 * Maps [-inf,+inf] → [-10,+10] smoothly.
 * A 2x-larger win still produces a larger reward (not clamped flat). */
reward = 10.0f * tanhf(reward / 10.0f);
```

And remove the turnover penalty (tx costs already deducted from cash):
```cuda
/* Turnover penalty REMOVED — tx costs are already deducted from cash
 * at position change (line 682). Adding a second penalty double-counts
 * and over-penalizes necessary rebalancing. */
```

---

## Task 1: Implement all 5 fixes

**Files:**
- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu`

- [ ] **Step 1:** Replace CUSUM vol proxy (line ~897) with ATR(14)-based vol from features[9]
- [ ] **Step 2:** Move loss aversion BEFORE the clamp
- [ ] **Step 3:** Replace hard clamp `fmaxf(-10, fminf(10, ...))` with `10 * tanhf(reward / 10)`
- [ ] **Step 4:** Remove the turnover penalty block (lines ~910-913)
- [ ] **Step 5:** Verify feature[9] is indeed log-ATR by checking `crates/ml/src/features/extraction.rs`
- [ ] **Step 6:** Build: `SQLX_OFFLINE=true cargo check -p ml`
- [ ] **Step 7:** Run 50-epoch smoke test and compare metrics
- [ ] **Step 8:** Commit

---

## Expected Impact

| Metric | Before (broken v6) | After (fixed v6) |
|--------|-------------------|-------------------|
| Vol normalization | CUSUM (binary) → 10000× in calm markets | ATR (continuous) → consistent 1-10× |
| Reward symmetry | +10 max, -15 max (loss aversion after clamp) | Smooth tanh, loss aversion applied first |
| Tail information | Hard clamp → all large wins identical | Tanh → larger wins still distinguishable |
| Turnover penalty | -0.05 × delta (double-counts tx) | Removed (tx cost in cash is sufficient) |
