refactor(rl): per-K hidden-grad buffer + split fused K-loop (E.3a)
Refactors the encoder backward to make its INPUT contract explicit so
the integrated RL trainer (and any future caller) can seed it without
going through the supervised GRN/aux backward path.
Refactor:
- New field PerceptionTrainer::grad_h_new_per_k_d: [K × B × HIDDEN_DIM]
buffer. The Loop-1 GRN head backward kernel writes pure per-K head
contributions into this buffer (called with grad_h_carry = nullptr,
the kernel's existing null-check path); the Loop-2 CfC step backward
reads slot k after folding in the recurrent grad_h_carry_d via
aux_vec_add_inplace. Replaces the pre-E.3a single-slot
grad_h_new_d ping-pong (field removed).
- dispatch_train_step's fused (GRN bwd → CfC bwd) reverse K-loop split
into two passes:
Loop 1 (head backwards) → grad_h_new_per_k_d (slot k, no carry)
Loop 2 (encoder backward) reads grad_h_new_per_k_d, folds in
grad_h_carry_d via the existing aux_vec_add_inplace kernel, then
runs cfc_step_bwd; CfC carry semantics byte-identical to before.
- New private helper dispatch_encoder_backward containing the entire
encoder backward kernel-launch chain: Loop-2 K-loop CfC bwd → 3D
transpose → attn-pool bwd + reducer → LN_b bwd + 2 reducers →
Mamba2 L2 bwd → LN_a bwd + 2 reducers → Mamba2 L1 bwd → VSN bwd +
2 reducers → reduce_axis0 for CfC param grads (4 launches) →
encoder Adam steps (CfC ×4 + Controller B + LN ×2 + VSN + attn_q +
Mamba2 L1/L2 grouped). Both supervised and RL paths call this same
helper — single source of truth per
feedback_single_source_of_truth_no_duplicates.
- New public method backward_encoder_with_grad_h_t seeds the per-K
buffer from a caller-provided grad_h_t at slot K-1 (the only slot
the RL heads consumed h_t from) and dispatches the shared encoder
backward helper. All other slots stay zero — those positions had
no downstream loss signal.
- Reducer closure at the old fused-loop site split per Option (a) in
the SDD: reduce_encoder lives in the helper (CfC ×4), reduce_heads
stays in dispatch_train_step (GRN ×10).
Integration:
- IntegratedTrainer::step_synthetic now calls
backward_encoder_with_grad_h_t with the accumulated
grad_h_t_combined_d (Q + π + V contributions with loss-balance λ
scaling). The encoder now learns from the full per-head gradient —
closes the last deferred item from Phase E.2-DEFER (item 1: encoder
backward integration).
Byte-identical supervised behavior preserved: all 63 ml-alpha lib
tests pass after the refactor (baseline 63, post-refactor 63). The
K-loop split + per-K buffer seeding contract is mathematically
equivalent to the prior fused-loop + single-slot ping-pong:
Old: GRN bwd writes grad_h[k] = head_contrib[k] + carry[k+1] (folded
in via kernel's grad_h_carry arg); cfc bwd reads grad_h.
New: Loop 1 writes grad_h_new_per_k[k] = head_contrib[k] (nullptr
carry); Loop 2 at slot k does grad_h_new_per_k[k] += carry[k+1]
via aux_vec_add_inplace, then cfc bwd reads the slot. Same final
value before cfc_step_bwd consumes it.
Capture-graph safety preserved: the new aux_vec_add launches per K
iteration are kernel-only (no host branches, no host mallocs, no
event tracking) per pearl_no_host_branches_in_captured_graph.
Companion to E.2 (commit 2665669b5) and E.2-DEFER (commit 7356e3c7b).
Phase E.3b follows with LobSim integration + toy bandit activation +
2 more controller kernels (rl_rollout_steps + rl_per_alpha).
This commit is contained in:
@@ -635,12 +635,10 @@ impl IntegratedTrainer {
|
||||
|
||||
// ── Step 10: encoder grad combine (Phase E.2-DEFER item 1) ───
|
||||
// Zero the combined slot, fold Q+π+V grad_h_t with λ-weights.
|
||||
// The combined grad slot is ready for an upcoming follow-up
|
||||
// commit that lands `PerceptionTrainer::backward_encoder_with_grad_h_t`
|
||||
// — the encoder-side entry point is a multi-thousand-line refactor
|
||||
// of `dispatch_train_step`'s backward portion and is being
|
||||
// sequenced separately to keep this commit focused on the kernel
|
||||
// signature changes (items 2, 3, 4) that unlock it.
|
||||
// The combined grad slot feeds `backward_encoder_with_grad_h_t`
|
||||
// below, which runs the shared encoder backward chain that
|
||||
// `dispatch_train_step` also uses (single source of truth per
|
||||
// feedback_single_source_of_truth_no_duplicates).
|
||||
self.stream
|
||||
.memset_zeros(&mut self.grad_h_t_combined_d)
|
||||
.map_err(|e| anyhow::anyhow!("zero grad_h_t_combined_d: {e}"))?;
|
||||
@@ -666,6 +664,21 @@ impl IntegratedTrainer {
|
||||
&mut self.grad_h_t_combined_d,
|
||||
)?;
|
||||
|
||||
// ── Step 11: encoder backward (Phase E.3a) ───────────────────
|
||||
// Seed the perception trainer's per-K hidden-state grad buffer
|
||||
// from `grad_h_t_combined_d` at slot K-1 (the only slot the
|
||||
// RL heads consumed `h_t` from) and run the shared encoder
|
||||
// backward kernel chain. This updates ALL encoder params:
|
||||
// CfC ×4, LN ×2, VSN, attn_q, Mamba2 L1+L2 — closes the last
|
||||
// deferred item from Phase E.2.
|
||||
self.perception
|
||||
.backward_encoder_with_grad_h_t(
|
||||
&self.grad_h_t_combined_d,
|
||||
self.cfg.perception.n_batch,
|
||||
self.cfg.perception.seq_len,
|
||||
)
|
||||
.context("perception.backward_encoder_with_grad_h_t")?;
|
||||
|
||||
// ── Step 12: compose stats ───────────────────────────────────
|
||||
// BCE / aux losses are NOT read this phase — perception is driven
|
||||
// separately by callers via its existing step_batched path. The
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user