2d498bec3a
rdgzl follow-up — chain hypothesis layer 2:
reward clamp lift unlocked V regression + PPO advantage (R/done
-$1.39 → -$0.48), but Q's distributional learning was structurally
capped at hardcoded V_MAX=1.0 in bellman_target_projection.cu —
any Bellman target > 1.0 categorically projected to atom 20 (top)
regardless of clamp. Even with WIN=3.8 clamp, Q never saw a +3.8
reward signal as distinct from a +1.0 reward signal.
This commit makes V_MIN/V_MAX ISV-driven with monotone-grow ratchet
coupled to the reward clamp. The C51 distribution support adapts
WITHOUT destabilising Q's learned values — atom 20 always represents
at least the widest WIN we've ever admitted (only grows, never shrinks).
Implementation:
- 2 new ISV slots (484 V_MAX, 485 V_MIN) with [-1, +1] floors
seeded by rl_isv_write
- rl_reward_clamp_controller.cu also ratchets these slots:
V_MAX_new = max(V_MAX_prev, max(1.0, WIN_clamp))
V_MIN_new = min(V_MIN_prev, min(-1.0, -LOSS_clamp))
- bellman_target_projection.cu reads V_MIN/V_MAX from ISV, derives
DELTA_Z inline (was #define)
- New rl_atom_support_update.cu (21-thread block) refreshes
atom_supports_d = linspace(V_MIN, V_MAX, 21) per step so
downstream C51 kernels (argmax_expected_q, rl_action_kernel,
dqn_distributional_q) see the current span
- Trainer launches atom-support updater after each reward-clamp
controller launch (both helper + step_with_lobsim inline paths)
- Diag exposes c51_v_max + c51_v_min for adaptation visibility
Floors at [-1, +1] preserve original C51 design as hard minimum —
the atom support can only become wider, never narrower than the
baseline.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
37 lines
1.5 KiB
Plaintext
37 lines
1.5 KiB
Plaintext
// rl_atom_support_update.cu — refresh `atom_supports_d` from the
|
|
// ISV-driven [V_MIN, V_MAX] span (audit 2026-05-24 second follow-up).
|
|
//
|
|
// Companion to the C51 atom-span ratchet in rl_reward_clamp_controller.
|
|
// `atom_supports_d` is the device buffer of 21 float values that the
|
|
// non-projection C51 kernels (`argmax_expected_q`, `rl_action_kernel`,
|
|
// `dqn_distributional_q`) read instead of recomputing the per-atom
|
|
// values themselves. When V_MIN/V_MAX adapt, this buffer must be
|
|
// rewritten or those kernels see a stale span.
|
|
//
|
|
// One block, Q_N_ATOMS threads. Each thread writes one atom value:
|
|
// atom_supports[i] = V_MIN + i * (V_MAX - V_MIN) / (N_ATOMS - 1)
|
|
//
|
|
// Per `feedback_no_atomicadd`: no atomics. Per `feedback_cpu_is_read_only`:
|
|
// pure device write. Per `feedback_no_htod_htoh_only_mapped_pinned`: no
|
|
// host transfer — values come from ISV slots, written from the trainer
|
|
// per-step launch right after the reward clamp controller refreshes
|
|
// V_MIN/V_MAX.
|
|
|
|
#define Q_N_ATOMS 21
|
|
#define RL_C51_V_MAX_INDEX 484
|
|
#define RL_C51_V_MIN_INDEX 485
|
|
|
|
extern "C" __global__ void rl_atom_support_update(
|
|
const float* __restrict__ isv, // ≥ RL_C51_V_MAX_INDEX + 1
|
|
float* __restrict__ atom_supports // [Q_N_ATOMS]
|
|
) {
|
|
const int tid = threadIdx.x;
|
|
if (tid >= Q_N_ATOMS) return;
|
|
|
|
const float v_min = isv[RL_C51_V_MIN_INDEX];
|
|
const float v_max = isv[RL_C51_V_MAX_INDEX];
|
|
const float delta = (v_max - v_min) / (float)(Q_N_ATOMS - 1);
|
|
|
|
atom_supports[tid] = v_min + (float)tid * delta;
|
|
}
|