Reward v6 Metrics Fixes — Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Fix 5 root causes that prevent the DQN model from learning profitable trading — all in the reward/metrics computation within the CUDA experience kernel.

Architecture: Single file change (experience_kernels.cu) plus minor config cleanup. All fixes are in the GPU kernel — no CPU paths.

Tech Stack: CUDA kernel, compiled via nvcc.

Root Causes (from audit)

#	Bug	Impact
1	CUSUM direction used as vol proxy (binary [-1,1] not volatility)	10000× reward amplification in calm markets
2	Loss aversion applied AFTER clamp (breaks ±10 symmetry to ±15)	Expected reward negative even for fair strategy
3	Hard clamp at ±10 destroys tail information	Model can't distinguish 1% win from 5% win
4	Turnover penalty double-counts tx costs	Over-penalizes necessary rebalancing
5	Feature[9] is log-normalized ATR — needs exp() to recover price units	Vol proxy computation needs the right feature

The Fix: Replace vol normalization + fix reward ordering

All changes in crates/ml/src/cuda_pipeline/experience_kernels.cu, lines ~893-920.

Before (broken):

float vol_proxy = fmaxf(cusum_raw * 0.01f, 0.0001f);           // WRONG: CUSUM is direction, not vol
float vol_norm = vol_proxy * sqrtf(fmaxf(segment_hold_time, 1.0f));
float vol_normalized_return = segment_return / vol_norm;
reward = 10.0f * vol_normalized_return;                          // Can explode to 10000+
reward = fmaxf(-10.0f, fminf(10.0f, reward));                   // Hard clamp kills tails
// ... later ...
if (reward < 0.0f) reward *= loss_aversion;                      // Applied AFTER clamp → -15
// ... later ...
reward -= 0.05f * turnover;                                      // Double-counts tx costs

After (fixed):

/* Vol normalization using ATR(14) — proper realized volatility.
 * Feature[9] is log-normalized ATR. exp() recovers price units.
 * Divide by close to get percentage vol. Scale by sqrt(hold_time)
 * for time-adjusted normalization (annualization principle). */
float log_atr = 0.0f;
if (features != NULL && bar_idx < total_bars && market_dim > 9) {
    log_atr = features[(long long)bar_idx * market_dim + 9];
}
float atr_pct = expf(log_atr) / fmaxf(raw_close, 1.0f);
float vol_proxy = fmaxf(atr_pct, 0.0001f);
float vol_norm = vol_proxy * sqrtf(fmaxf(segment_hold_time, 1.0f));
float vol_normalized_return = segment_return / vol_norm;

/* Scale to learnable magnitude. Typical vol-normalized return ≈ 0.1-0.5.
 * REWARD_SCALE=10 puts rewards in [-5, +5] range (SNR ≈ 5000). */
reward = 10.0f * vol_normalized_return;

/* Loss aversion BEFORE clamp (prospect theory: losses loom 1.5x larger).
 * Must happen before clamp so the asymmetry is preserved correctly. */
if (reward < 0.0f) {
    reward *= loss_aversion;
}

/* Soft clamp: tanh squashing preserves tail information.
 * Maps [-inf,+inf] → [-10,+10] smoothly.
 * A 2x-larger win still produces a larger reward (not clamped flat). */
reward = 10.0f * tanhf(reward / 10.0f);

And remove the turnover penalty (tx costs already deducted from cash):

/* Turnover penalty REMOVED — tx costs are already deducted from cash
 * at position change (line 682). Adding a second penalty double-counts
 * and over-penalizes necessary rebalancing. */

Task 1: Implement all 5 fixes

Files:

Modify: crates/ml/src/cuda_pipeline/experience_kernels.cu
Step 1: Replace CUSUM vol proxy (line ~897) with ATR(14)-based vol from features[9]
Step 2: Move loss aversion BEFORE the clamp
Step 3: Replace hard clamp fmaxf(-10, fminf(10, ...)) with 10 * tanhf(reward / 10)
Step 4: Remove the turnover penalty block (lines ~910-913)
Step 5: Verify feature[9] is indeed log-ATR by checking crates/ml/src/features/extraction.rs
Step 6: Build: SQLX_OFFLINE=true cargo check -p ml
Step 7: Run 50-epoch smoke test and compare metrics
Step 8: Commit

Expected Impact

Metric	Before (broken v6)	After (fixed v6)
Vol normalization	CUSUM (binary) → 10000× in calm markets	ATR (continuous) → consistent 1-10×
Reward symmetry	+10 max, -15 max (loss aversion after clamp)	Smooth tanh, loss aversion applied first
Tail information	Hard clamp → all large wins identical	Tanh → larger wins still distinguishable
Turnover penalty	-0.05 × delta (double-counts tx)	Removed (tx cost in cash is sufficient)

4.7 KiB Raw Blame History Unescape Escape