# Advantage Standardization + Distribution Entropy — Design Spec

## Problem

The magnitude branch (Small/Medium/Full position sizing) collapses to all-Small within 4 epochs. Root cause: dueling mean subtraction + C51 cross-entropy loss creates a self-reinforcing winner-take-all dynamic. Small positions have lower PnL variance → tighter C51 atom distributions → lower cross-entropy loss → more gradient signal → advantages diverge further. The mean subtraction accelerates this: raising Small's advantage forces Medium/Full down.

Previous fixes (epsilon, counterfactual gradient, gradient amplification, risk-scaling-before-magnitude) address symptoms but not the root cause.

## Solution: Two-Pronged

### Prong 1: Advantage Standardization (magnitude branch only)

**What:** After computing dueling advantages `A[a,j] - mean_a(A[*,j])` for the magnitude branch (d==1), divide by `std_a(A[*,j]) + 1e-6`. This normalizes the advantage scale per-atom to unit variance.

**Before (current):**
```
Q_mag[j] = V[j] + A_mag[a_d, j] - mean_a(A_mag[*, j])
```

**After:**
```
a_mean[j] = mean_a(A_mag[*, j])
a_std[j]  = std_a(A_mag[*, j])
Q_mag[j]  = V[j] + (A_mag[a_d, j] - a_mean[j]) / (a_std[j] + 1e-6)
```

**Why this works:**
- Self-adapting: no temperature to tune. The normalization automatically adjusts based on actual advantage variance.
- Preserves relative ordering: if Full truly has higher advantage than Small, it stays highest.
- Prevents scale runaway: advantages are bounded to O(1), breaking the winner-take-all feedback loop.
- The gradient through the std normalization creates a natural "repulsive" force that prevents collapse — when advantages converge (low std), the normalization amplifies small differences.

**Where:** Inside the branch loop in `c51_loss_kernel.cu` and `mse_loss_kernel.cu`, only when `d == 1` (magnitude branch). Applied AFTER loading advantages into shmem_adv, BEFORE log-softmax.

**Shared memory:** Requires computing std over n_d actions per atom. With n_d=3, this is trivial — just 3 values per atom. The mean is already computed. Std = sqrt(mean(x²) - mean(x)²), computable from the existing shmem_adv values with no extra shared memory.

### Prong 2: Distribution Entropy Regularization (magnitude branch only)

**What:** In the C51 grad kernel, multiply the entropy gradient term by 5× for d==1 (magnitude). Current entropy coefficient = 0.001; magnitude gets effective 0.005.

**Before:**
```
if (entropy_coeff > 0.0f) {
    d_combined += entropy_coeff * (1.0f + lp_clamped);
}
```

**After:**
```
float ent_scale = (d == 1) ? 5.0f : 1.0f;
if (entropy_coeff > 0.0f) {
    d_combined += ent_scale * entropy_coeff * (1.0f + lp_clamped);
}
```

**Why:** Standardization prevents Q-value scale collapse. Entropy prevents atom distribution sharpness collapse. Both feedback loops need to be broken.

## Files Modified

- `crates/ml/src/cuda_pipeline/c51_loss_kernel.cu` — advantage standardization in branch loop
- `crates/ml/src/cuda_pipeline/mse_loss_kernel.cu` — same
- `crates/ml/src/cuda_pipeline/c51_grad_kernel.cu` — entropy boost for d==1
- `crates/ml/src/cuda_pipeline/mse_grad_kernel.cu` — same

## No New Parameters

All changes use existing values (advantage tensors, entropy_coeff) and constants (5.0f, 1e-6). No new kernel parameters, no Rust launch changes, no config fields. CUDA graph capture is unaffected (no buffer size changes, no new allocations).

## Expected Outcome

- Magnitude diversity should hold at 5-7/9 through late epochs (not collapse to 3/9)
- Small/Medium/Full should show state-dependent preferences (Full in high-conviction, Small in uncertain)
- No impact on direction/order/urgency branches (standardization is magnitude-only)
- PF should improve as the model learns to size positions appropriately
