6d433784f4
Closes defect #5 from the flawed Phase F+G arc (feedback_cpu_is_read_only violation in step_with_lobsim's host advantage + EMA loops) by landing the GPU primitives those loops will become in R6. Three new kernels, each with a GPU-oracle gate test: 1. ema_update_on_done.cu — done-gated EMA producer. - Slot-parameterised (one kernel, 3 callers in R5 covering mean_abs_pnl_ema, q_divergence_ema, td_kurtosis_ema). - Shared-mem tree reduce, no atomicAdd (feedback_no_atomicadd). - Per pearl_first_observation_bootstrap: sentinel-zero ISV → first observation replaces directly. Defers bootstrap if mean_obs == 0 to avoid writing a degenerate sentinel that would be re-bootstrapped next call. - Per pearl_wiener_alpha_floor_for_nonstationary: Wiener-α blend on subsequent calls; caller pre-floors α at 0.4. 2. ema_update_per_step.cu — per-step EMA producer (no done-gate). - Slot-parameterised (kl_pi_ema, entropy_observed_ema, advantage_var_ratio_ema, mean_trade_duration_ema in R5). - Same shared-mem tree reduce + bootstrap discipline as ema_update_on_done. 3. compute_advantage_return.cu — element-wise returns[b] = r + γ(1-done)·V(s_{t+1}); advantages[b] = returns − V(s_t). - Reads γ from ISV[400] (R1 bootstrap = 0.99). - Trivially parallel, one thread per batch entry; no atomics. Rust launchers added to IntegratedTrainer: - launch_ema_update_on_done(slot, alpha, obs_d, dones_d, b_size) - launch_ema_update_per_step(slot, alpha, obs_d, b_size) - launch_compute_advantage_return(rewards_d, dones_d, v_t_d, v_tp1_d, returns_d, advantages_d, b_size) 3 cubin includes, 3 module/function fields, loaders in new() between the rl_reward_scale_controller load and the with_controllers_bootstrapped call so the new fields are populated by struct construction. GPU-oracle tests in tests/r3_ema_advantage.rs (per feedback_no_cpu_test_fallbacks every oracle is either the kernel's documented bootstrap behaviour or an analytical property of the formula, not a CPU reference): R3.1: ema_update_on_done bootstrap path — sentinel-zero ISV + one observation k → ISV[slot] == k exactly. Negative invariant: hold-only step (dones all zero) preserves the EMA. R3.2: ema_update_per_step convergence — feed obs=5.0 for 50 steps with α=0.4 → ISV[slot] → 5.0 within 1e-4 (EMA of constant = constant). R3.3: compute_advantage_return formula — r=0, done=0, v_t=v_tp1=k, γ=0.99 → returns=γk=4.95, advantages=(γ−1)k=−0.05. Negative invariant: done=1 + r=0 zeros the future-value bootstrap (returns=0, advantages=−k). Build cache-bust v26. cargo check + cargo build --test r3_ema_advantage on ml-alpha green. Pre-existing heads_bit_equiv.rs index-out-of-bounds failure persists (unrelated; pre-Phase E). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
66 lines
2.6 KiB
Plaintext
66 lines
2.6 KiB
Plaintext
// ema_update_per_step.cu — per-step EMA producer for ISV-resident
|
||
// adaptive-control EMAs that update every step regardless of episode
|
||
// boundary (Phase R3 of the integrated RL trainer rebuild;
|
||
// see docs/superpowers/plans/2026-05-23-integrated-rl-trainer-rebuild.md).
|
||
//
|
||
// Companion to `ema_update_on_done.cu`. This variant has NO done-gate
|
||
// — it consumes the per-batch raw signal on every step. Used for EMAs
|
||
// that should update continuously rather than only at trade closes:
|
||
//
|
||
// * KL(π_new ‖ π_old) EMA (input to rl_ppo_clip_controller, ISV[419])
|
||
// * Observed action entropy H(π) EMA (input to rl_entropy_coef_controller,
|
||
// ISV[420])
|
||
// * Advantage variance ratio EMA (input to rl_rollout_steps_controller,
|
||
// ISV[421])
|
||
//
|
||
// Generic over the ISV slot — caller passes `slot_index` + Wiener-α.
|
||
// Bootstrap discipline + reduction strategy are identical to
|
||
// `ema_update_on_done.cu`; only the gate differs (here, every batch
|
||
// entry contributes, so the per-batch mean is the unweighted average).
|
||
//
|
||
// Per `feedback_no_atomicadd`: shared-memory tree reduction in a
|
||
// single block; no atomics. Caller passes
|
||
// `shared_mem_bytes = b_size * sizeof(float)`.
|
||
|
||
extern "C" __global__ void ema_update_per_step(
|
||
float* __restrict__ isv, // ISV bus (≥ slot_index + 1)
|
||
int slot_index, // which ISV slot to update
|
||
float alpha, // Wiener-α (host-floored ≥ 0.4)
|
||
const float* __restrict__ obs, // [b_size] per-batch raw signal
|
||
int b_size
|
||
) {
|
||
extern __shared__ float smem[]; // [b_size]
|
||
|
||
const int tid = threadIdx.x;
|
||
if (tid >= b_size) return;
|
||
|
||
smem[tid] = obs[tid];
|
||
__syncthreads();
|
||
|
||
// Tree reduction (block-level, no atomicAdd per
|
||
// feedback_no_atomicadd).
|
||
for (int stride = 1; stride < b_size; stride *= 2) {
|
||
if ((tid % (stride * 2)) == 0 && (tid + stride) < b_size) {
|
||
smem[tid] += smem[tid + stride];
|
||
}
|
||
__syncthreads();
|
||
}
|
||
|
||
if (tid == 0) {
|
||
const float mean_obs = smem[0] / (float)b_size;
|
||
const float prev = isv[slot_index];
|
||
if (prev == 0.0f) {
|
||
// First-observation bootstrap per pearl_first_observation_bootstrap.
|
||
// Defer if mean_obs == 0 so we don't write a sentinel that
|
||
// would be re-bootstrapped on the next call.
|
||
if (mean_obs != 0.0f) {
|
||
isv[slot_index] = mean_obs;
|
||
}
|
||
} else {
|
||
// Wiener-α blend; caller pre-floored α at 0.4 per
|
||
// pearl_wiener_alpha_floor_for_nonstationary.
|
||
isv[slot_index] = (1.0f - alpha) * prev + alpha * mean_obs;
|
||
}
|
||
}
|
||
}
|