feat(rl): FRD softmax + CE + dL/dlogits backward stage 1 (F.3a)

Per-(batch, horizon) softmax + cross-entropy loss + gradient w.r.t.
the 21 atom logits. First of three backward stages — F.3b adds layer-2
weight grads (dW2, db2, dhidden), F.3c adds layer-1 weight grads
(dW1, db1, dh_t with ReLU mask).

Kernel `cuda/rl_frd_softmax_ce_grad.cu`:
  * grid_dim = (B, FRD_N_HORIZONS, 1), block_dim = (FRD_N_ATOMS=21, 1, 1)
    — one block per (batch, horizon) pair, threads cooperate over the
    21 atoms via shared mem
  * Standard numerically-stable softmax: shift by row_max, exponentiate,
    normalize by row_sum (thread 0 does the serial reductions — 21
    atoms is small enough warp-shuffle overhead isn't worth it)
  * Gradient: (p[a] - 1{a==label}) / B at the source per v_head_bwd
    convention (mean-reduce over batch)
  * Loss: -log(p[label]) with 1e-30 floor against log(0)
  * Sentinel label (-1) zeros both gradient row and loss — for the
    missing-horizon case at the rightmost edge of the snapshot stream
    (forward returns at h=300 ticks aren't realized for the last
    300 snapshots; loader marks those labels with -1)
  * Per feedback_no_atomicadd: per-(b, h, a) sole-writer pattern

Rust wiring `src/rl/frd.rs::FrdHead::softmax_ce_grad`:
  * Second cubin loaded alongside fwd (separate module per the
    aux_heads pattern; small handle, no impact on init time)
  * Caller provides labels_d [B, FRD_N_HORIZONS] of i32 and gets back
    grad_logits + per-(b, h) raw CE; sum + λ_frd scaling left to the
    caller (F.4 will hook this into stats.l_total + Adam step)

Tests `tests/frd_head.rs` — 3 new GPU-oracle tests (6/6 file total),
all PASS on RTX 3050 Ti:
  1. frd_softmax_ce_grad_uniform_logits_match_log_n_atoms — for any
     label, uniform logits → CE = ln(FRD_N_ATOMS) = ln(21) ≈ 3.0445.
     Also asserts per-row Σ grad_logits = 0 (softmax-CE invariant).
  2. frd_softmax_ce_grad_sentinel_label_zeros_row — label=-1 with
     non-trivial random logits produces exactly zero loss + grad
     for every row (no leak through the sentinel path).
  3. frd_softmax_ce_grad_finite_diff_matches_analytical — perturbs
     one logit slot by ±ε=1e-3, compares (L(+ε) - L(-ε))/(2ε) to
     the kernel's analytical gradient. rel_err ≈ 1.3e-3 (fp32
     finite-diff is rounding-error-limited at this ε; tolerance
     set to 5e-3 with explanatory comment).

The first two tests provide strong analytical oracles (no CPU
reference impl per feedback_no_cpu_test_fallbacks). The finite-diff
test cross-validates the full softmax+CE chain via a numerical
gradient — the standard ground-truth for autodiff kernels.
This commit is contained in:
jgrusewski
2026-05-24 18:31:03 +02:00
parent 119c3a15f4
commit 6cfd7e6691
4 changed files with 342 additions and 3 deletions

View File

@@ -76,6 +76,7 @@ const KERNELS: &[&str] = &[
"rl_trail_mutate", // SP20 P1+P5 audit fix (a7/a8 dead): TrailTighten/TrailLoosen mutate unit_trail_distance bounded MIN/MAX, symmetric reciprocal adjust
"rl_trail_stop_check", // SP20 P1+P5 audit fix: per-unit trail breach check; OVERRIDE action to FlatFromLong/Short on breach (close routes through existing flat plumbing)
"rl_frd_fwd", // SP20 P3: Forward-Return-Distribution head fwd — 2-layer MLP [HIDDEN_DIM → FRD_HIDDEN_DIM → FRD_N_HORIZONS × FRD_N_ATOMS]; ReLU hidden cached for bwd; softmax + CE happen in bwd
"rl_frd_softmax_ce_grad", // SP20 P3 F.3a: per-(batch, horizon) softmax + CE loss + dL/dlogits; 1 block per (b, h), 21 threads; label = -1 sentinel masks the row
];
// Cache bust v31 — five new reduce / derive kernels populate the input