aba8ec61b2
Honors R6's commit-message promise to close the host-work boundary
the partial GPU-purity left behind. After R7a, step_with_lobsim's
post-fill pipeline is GPU-resident through the entire training step
except for 3 small host-slice uploads (actions, next_actions,
log_pi_old) that R7b lifts via R4's Thompson/argmax/log_pi kernels.
CHANGES:
1. New cuda/abs_copy.cu — element-wise dst[b] = fabsf(src[b]).
Feeds the |reward| signal into ema_update_on_done for the
MEAN_ABS_PNL_EMA slot without mutating signed rewards_d.
2. New trainer-owned per-step device buffers (allocated once in
new(), reused every step — no per-call churn):
reward_abs_d, actions_d, next_actions_d, log_pi_old_d,
advantages_d, returns_d
3. step_synthetic signature change: drops the 7 synthetic_* host
slice args (synthetic_actions, synthetic_rewards, synthetic_dones,
synthetic_next_actions, synthetic_advantages, synthetic_returns,
synthetic_log_pi_old). The body now reads from trainer-owned
device buffers (self.actions_d, self.rewards_d, etc.) via the
disjoint-field borrow rule. The 7 upload_i32/upload_f32 calls
are deleted — caller (step_with_lobsim) populates the buffers
via GPU kernels before invoking step_synthetic.
4. step_with_lobsim Step 6 rewritten end-to-end:
- Upload host Thompson outputs (actions, next_actions, log_pi_old)
to trainer buffers — R7b removes these via R4's GPU kernels.
- GPU abs_copy(rewards_d) → reward_abs_d.
- GPU ema_update_on_done(MEAN_ABS_PNL_EMA, reward_abs_d, dones_d).
- GPU launch_rl_controllers_per_step (R5) — all 7 controllers
adapt to this step's EMA inputs.
- GPU apply_reward_scale(rewards_d) in-place — reads the freshly
updated ISV[406] from the controller.
- GPU compute_advantage_return(rewards_d, dones_d, v_t, v_tp1)
→ returns_d, advantages_d.
- step_synthetic(snapshots) consumes all trainer device buffers,
runs the training kernel chain, returns stats.
- dqn_head.soft_update_target(isv_d) — R5 target-net Polyak
update with τ from ISV[401].
R7a PARTIAL — work R7b lifts:
- Host Thompson sampler still produces actions/next_actions/log_pi
(R7b replaces with R4 rl_action_kernel + argmax_expected_q +
log_pi_at_action — eliminating the 3 remaining HtoD uploads).
- v_pred_host → v_pred_d HtoD round-trip (the host already has
v_pred_host from the action-sampling Thompson read; R7b keeps V
on device throughout).
- v_tp1_d uses v_pred_host (V(s_t) approximated as V(s_{t+1}) until
R7b wires next_snapshots + forward_encoder(next_snapshots)).
The 4 final DtoH copies the R6 commit message warned about are
GONE. Hot path: snapshot upload (boundary HtoD), ISV diagnostic
readback (HEALTH_DIAG), and 3 host-Thompson uploads (R7b removes).
cargo check + cargo build --tests on ml-alpha green. Tests still
compile against the new step_synthetic signature (no test calls it
directly — only step_with_lobsim does).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
24 lines
942 B
Plaintext
24 lines
942 B
Plaintext
// abs_copy.cu — element-wise absolute-value copy: dst[b] = fabsf(src[b])
|
|
// (Phase R7a of the integrated RL trainer rebuild;
|
|
// see docs/superpowers/plans/2026-05-23-integrated-rl-trainer-rebuild.md).
|
|
//
|
|
// Feeds the |reward| signal into `ema_update_on_done` for the
|
|
// `MEAN_ABS_PNL_EMA` slot (ISV[423], input to
|
|
// rl_reward_scale_controller). The trainer keeps signed `rewards_d`
|
|
// for the advantage / return / reward-scale-apply pipeline; this
|
|
// kernel writes the magnitude into a separate `reward_abs_d` scratch
|
|
// so the EMA producer reads the magnitude without mutating the signed
|
|
// reward stream.
|
|
//
|
|
// Element-wise, trivially parallel. No reduction, no atomics.
|
|
|
|
extern "C" __global__ void abs_copy(
|
|
const float* __restrict__ src,
|
|
float* __restrict__ dst,
|
|
int b_size
|
|
) {
|
|
const int b = blockIdx.x * blockDim.x + threadIdx.x;
|
|
if (b >= b_size) return;
|
|
dst[b] = fabsf(src[b]);
|
|
}
|