feat(sp6): Pearl 5 — IQN τ per-branch schedule (4 forward passes, ÷4 budget normalization)

GpuIqnHead gains 12 new CudaSlice<f32> buffers (online_taus_branch[4],
target_taus_branch[4], cos_features_branch[4]) allocated at construction time
via alloc_f32. Each slab is [B,N] for taus and [D,N] for cos_features — same
sizes as the existing main buffers.

refresh_taus_for_branch(branch_idx, tau5): uploads one branch's 5-quantile
τ schedule from ISV[IQN_TAU_BASE + b*5 .. +5] to per-branch slabs with cold-start
floor (FIXED_TAUS[q] when ISV slot is zero). No cross-branch averaging.

activate_branch_taus(b) / deactivate_branch_taus(b): symmetric mem::swap helpers
install/restore one branch's slab into self.online_taus/target_taus/cos_features
for a per-branch IQN forward pass. activate→deactivate(b) is its own inverse.

fused_training.rs:
- Tau refresh block calls refresh_taus_for_branch(b, tau5) for all 4 branches,
  then refresh_taus_from_isv for the averaged main buffer (CVaR backward compat).
- grad_decomp_snapshot_iqn() moved BEFORE the parallel/sequential fork so the
  snapshot is taken before any of the 4 per-branch apply_iqn_trunk_gradient calls.
- Parallel path: 4 sequential IQN passes on iqn_stream; after each pass, event
  sync to main stream, apply_iqn_trunk_gradient(iqn_budget/4), re-fork so next
  pass starts after main has consumed d_h_s2_buf. iqn_done_event recorded after
  all 4 passes.
- Sequential path: 4 sequential IQN passes on main stream; apply_iqn_trunk_gradient
  (iqn_budget/4) inline after each pass while d_h_s2_buf holds that branch's result.
- Post-join: single apply_iqn_trunk_gradient removed (now inline); target_ema_update
  and PER loss cast remain.

÷4 normalization: both apply_iqn_trunk_gradient call sites use iqn_budget_per_branch
= iqn_budget / 4.0_f32 so 4 × (budget/4) = budget total — matching SP5 Layer B
gradient magnitude contract exactly.

docs/isv-slots.md: add SP6 Pearl 5 consumer wiring section under the SP5 table.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-02 02:08:45 +02:00
parent 6b01b70c3e
commit 89fadec24a
3 changed files with 252 additions and 60 deletions

View File

@@ -325,6 +325,14 @@ pub struct GpuIqnHead {
/// Precomputed cosine features [D, N] — cos(π·(d+1)·τ_i)
/// Fixed at construction (τ are midpoints). Column-major.
cos_features: CudaSlice<f32>,
/// SP6 Pearl 5: per-branch τ tensors [B, N], one slab per branch (4 total).
/// Populated by `refresh_taus_for_branch`; activated via `activate_branch_taus`
/// before each per-branch IQN forward pass. Same allocation size as `online_taus`.
online_taus_branch: [CudaSlice<f32>; 4],
/// SP6 Pearl 5: per-branch target τ tensors [B, N], one slab per branch.
target_taus_branch: [CudaSlice<f32>; 4],
/// SP6 Pearl 5: per-branch cosine features [D, N], one slab per branch.
cos_features_branch: [CudaSlice<f32>; 4],
/// Branch actions decoded from flat actions [B, 4]
branch_actions: CudaSlice<i32>,
/// Target h_s2 computed from next_states + target trunk weights [B, H]
@@ -582,6 +590,21 @@ impl GpuIqnHead {
// IQR buffer
let iqr_buf = alloc_f32(&stream, tba, "iqn_iqr")?;
// SP6 Pearl 5: allocate per-branch tau/cos buffer triplets (same size as main buffers).
// 4 branches × ([B*N] online + [B*N] target + [D*N] cos).
let online_taus_b0 = alloc_f32(&stream, b * n, "iqn_online_taus_b0")?;
let online_taus_b1 = alloc_f32(&stream, b * n, "iqn_online_taus_b1")?;
let online_taus_b2 = alloc_f32(&stream, b * n, "iqn_online_taus_b2")?;
let online_taus_b3 = alloc_f32(&stream, b * n, "iqn_online_taus_b3")?;
let target_taus_b0 = alloc_f32(&stream, b * n, "iqn_target_taus_b0")?;
let target_taus_b1 = alloc_f32(&stream, b * n, "iqn_target_taus_b1")?;
let target_taus_b2 = alloc_f32(&stream, b * n, "iqn_target_taus_b2")?;
let target_taus_b3 = alloc_f32(&stream, b * n, "iqn_target_taus_b3")?;
let cos_features_b0 = alloc_f32(&stream, d * n, "iqn_cos_features_b0")?;
let cos_features_b1 = alloc_f32(&stream, d * n, "iqn_cos_features_b1")?;
let cos_features_b2 = alloc_f32(&stream, d * n, "iqn_cos_features_b2")?;
let cos_features_b3 = alloc_f32(&stream, d * n, "iqn_cos_features_b3")?;
let vram_bytes = (total_params * 6 + h * bq * 8 + tba * bq * 4
+ d * bq + b * h * 2 + b * 4 + tba + b * 2 + 2) * 4;
@@ -683,6 +706,9 @@ impl GpuIqnHead {
online_taus,
target_taus,
cos_features,
online_taus_branch: [online_taus_b0, online_taus_b1, online_taus_b2, online_taus_b3],
target_taus_branch: [target_taus_b0, target_taus_b1, target_taus_b2, target_taus_b3],
cos_features_branch: [cos_features_b0, cos_features_b1, cos_features_b2, cos_features_b3],
branch_actions,
target_h_s2,
cached_target_h_s2_ptr: None,
@@ -2016,6 +2042,89 @@ impl GpuIqnHead {
}
}
/// SP6 Pearl 5: Refresh the τ tensors for a single branch from its 5-quantile
/// schedule `tau5` (from ISV[IQN_TAU_BASE + branch_idx*5 .. +5]).
///
/// Uploads to `online_taus_branch[branch_idx]`, `target_taus_branch[branch_idx]`,
/// and `cos_features_branch[branch_idx]`. Called 4 times per epoch refresh (once
/// per branch) from `FusedTrainingCtx::run_full_step`.
///
/// Cold-start floor: if a tau value is below 1e-6 (ISV not yet populated),
/// uses `FIXED_TAUS[q]` to preserve the interpretable quantile distribution.
pub fn refresh_taus_for_branch(
&mut self,
branch_idx: usize,
tau5: &[f32; 5],
) {
debug_assert!(branch_idx < 4, "refresh_taus_for_branch: branch_idx {branch_idx} >= 4");
let n = self.config.num_quantiles; // 5
let b = self.config.batch_size;
let d = self.config.embed_dim;
// Apply cold-start floor: if ISV slot is zero (not yet populated),
// fall back to FIXED_TAUS[q] so we don't collapse the quantile distribution.
let mut taus = [0.0_f32; 5];
for q in 0..n {
taus[q] = if tau5[q] < 1e-6_f32 { FIXED_TAUS[q] } else { tau5[q].min(1.0_f32) };
}
// Recompute cosine features [D, N]: cos_feat[dim + q*D] = cos(π·(d+1)·τ_q)
let mut cos_feat_host = vec![0.0_f32; d * n];
for q in 0..n {
let tau_q = taus[q];
for dim in 0..d {
cos_feat_host[dim + q * d] =
(std::f32::consts::PI * ((dim + 1) as f32) * tau_q).cos();
}
}
// Tile taus [B, N].
let mut tiled = Vec::with_capacity(b * n);
for _ in 0..b {
tiled.extend_from_slice(&taus[..n]);
}
if let Err(e) = super::mapped_pinned::upload_f32_via_pinned(
&self.stream, &tiled, &mut self.online_taus_branch[branch_idx],
) {
tracing::warn!("SP6 Pearl 5 online_taus_branch[{branch_idx}] upload failed: {e}");
return;
}
if let Err(e) = super::mapped_pinned::upload_f32_via_pinned(
&self.stream, &tiled, &mut self.target_taus_branch[branch_idx],
) {
tracing::warn!("SP6 Pearl 5 target_taus_branch[{branch_idx}] upload failed: {e}");
return;
}
if let Err(e) = super::mapped_pinned::upload_f32_via_pinned(
&self.stream, &cos_feat_host, &mut self.cos_features_branch[branch_idx],
) {
tracing::warn!("SP6 Pearl 5 cos_features_branch[{branch_idx}] upload failed: {e}");
}
}
/// SP6 Pearl 5: Swap the main tau/cos buffers with branch `branch_idx`'s per-branch
/// buffers so the IQN forward pass uses that branch's τ schedule.
///
/// Must be paired with `deactivate_branch_taus(branch_idx)` after the forward pass.
/// `activate` followed by `deactivate` with the same index is its own inverse.
pub fn activate_branch_taus(&mut self, branch_idx: usize) {
debug_assert!(branch_idx < 4, "activate_branch_taus: branch_idx {branch_idx} >= 4");
std::mem::swap(&mut self.online_taus, &mut self.online_taus_branch[branch_idx]);
std::mem::swap(&mut self.target_taus, &mut self.target_taus_branch[branch_idx]);
std::mem::swap(&mut self.cos_features, &mut self.cos_features_branch[branch_idx]);
}
/// SP6 Pearl 5: Restore the main tau/cos buffers after a per-branch IQN forward pass.
///
/// Symmetric inverse of `activate_branch_taus(branch_idx)`.
pub fn deactivate_branch_taus(&mut self, branch_idx: usize) {
debug_assert!(branch_idx < 4, "deactivate_branch_taus: branch_idx {branch_idx} >= 4");
std::mem::swap(&mut self.online_taus, &mut self.online_taus_branch[branch_idx]);
std::mem::swap(&mut self.target_taus, &mut self.target_taus_branch[branch_idx]);
std::mem::swap(&mut self.cos_features, &mut self.cos_features_branch[branch_idx]);
}
/// Compute CVaR-based position scaling from IQN quantiles.
///
/// Runs the IQN forward-only kernel on `h_s2` (trunk activation),

View File

@@ -2011,15 +2011,25 @@ impl FusedTrainingCtx {
// ── IQN prep: decode actions + DtoD copies (main stream, before fork) ──
// Must complete before execute_training_pipeline runs on iqn_stream.
if let Some(ref mut iqn) = self.gpu_iqn {
// SP5 Pearl 5: refresh τ schedule from ISV[IQN_TAU_BASE=250..270).
// Read 4 branches × 5 quantiles, average per quantile, apply cold-start
// floor from FIXED_TAUS, re-upload online_taus/target_taus/cos_features.
// Must run before prepare_buffers so the kernel sees updated tau values.
// SP6 Pearl 5: refresh per-branch τ schedules from ISV[250..270).
// ISV layout: branch b, quantile q → IQN_TAU_BASE + b*5 + q.
// Each branch gets its own [B,N] tau slab; the main (averaged) buffer
// is also kept current via refresh_taus_from_isv for CVaR and any
// consumer that has not migrated to the per-branch path.
{
let mut isv_taus = [0.0_f32; 20];
for i in 0..20 {
isv_taus[i] = self.trainer.read_isv_signal_at(IQN_TAU_BASE + i);
}
// Populate per-branch slabs first.
for branch_idx in 0..4_usize {
let mut tau5 = [0.0_f32; 5];
for q in 0..5_usize {
tau5[q] = isv_taus[branch_idx * 5 + q];
}
iqn.refresh_taus_for_branch(branch_idx, &tau5);
}
// Keep the averaged main buffer current for CVaR and backward-compat consumers.
iqn.refresh_taus_from_isv(&isv_taus);
}
iqn.set_cached_target_h_s2(self.trainer.tg_h_s2_ptr());
@@ -2042,6 +2052,13 @@ impl FusedTrainingCtx {
let parallel = self.iqn_stream.is_some();
let mut iqn_ok = false;
// SP6 Pearl 5 / Task 2.0: snapshot grad_buf branch 0+1 slice BEFORE any
// IQN trunk gradient fires. The parallel and sequential paths both call
// apply_iqn_trunk_gradient inline (4 passes × budget/4), so the snapshot
// must precede the fork to correctly isolate the IQN contribution delta.
self.trainer.grad_decomp_snapshot_iqn()
.map_err(|e| anyhow::anyhow!("Task 2.0 grad_decomp_snapshot_iqn: {e}"))?;
if parallel {
// Record fork-point event on main stream.
let iql_ev = self.iql_done_event.as_ref()
@@ -2050,6 +2067,10 @@ impl FusedTrainingCtx {
.map_err(|e| anyhow::anyhow!("iql_done_event record: {e}"))?;
// ── Fork IQN to iqn_stream ──
// SP6 Pearl 5: run 4 sequential IQN forward passes on iqn_stream, each
// with its per-branch τ schedule active. After each pass, synchronize to
// the main stream and apply iqn_budget/4 trunk gradient contribution so
// all 4 branches' d_h_s2 values are folded into grad_buf equally.
if has_iqn {
let iqn_s = self.iqn_stream.as_ref().unwrap();
iqn_s.wait(iql_ev)
@@ -2059,10 +2080,6 @@ impl FusedTrainingCtx {
let ws_override = Some((iqn_ws.raw_ptr(), 32 * 1024 * 1024_usize));
let iqn_stream_ref = Arc::clone(iqn_s);
let iqn = self.gpu_iqn.as_mut().unwrap();
let online_h_s2 = self.trainer.save_h_s2();
let next_states_buf = self.trainer.next_states_buf();
// SP4 Layer B (Mech 9): ISV-driven post-Adam |p| clamp for
// `iqn_adam_kernel`. IQN params live in group 3
// (`ParamGroup::Iqn`); `read_group_adam_bounds` returns
@@ -2077,24 +2094,62 @@ impl FusedTrainingCtx {
let iqn_beta2 = self.trainer.read_isv_signal_at(ADAM_BETA2_BASE + iqn_g).clamp(0.99_f32, 0.9995_f32);
let iqn_epsilon = self.trainer.read_isv_signal_at(ADAM_EPS_BASE + iqn_g).clamp(1e-10_f32, 1e-6_f32);
match iqn.execute_training_pipeline(
online_h_s2, next_states_buf, &self.target_dueling,
Some(&iqn_stream_ref), ws_override,
weight_clamp_max_abs,
weight_decay_value,
iqn_beta1,
iqn_beta2,
iqn_epsilon,
) {
Ok(_) => { iqn_ok = true; }
Err(e) => {
tracing::warn!("IQN parallel step failed (non-fatal): {e}");
// ÷4 normalization: each branch pass contributes iqn_budget/4 to
// grad_buf via apply_iqn_trunk_gradient; 4 passes × (budget/4) =
// budget total, preserving SP5 Layer B gradient magnitude contract.
let iqn_budget_per_branch = iqn_budget / 4.0_f32;
let iqn_ev = self.iqn_done_event.as_ref().unwrap();
for branch_idx in 0..4_usize {
let online_h_s2 = self.trainer.save_h_s2();
let next_states_buf = self.trainer.next_states_buf();
let iqn = self.gpu_iqn.as_mut().unwrap();
iqn.activate_branch_taus(branch_idx);
let pass_ok = match iqn.execute_training_pipeline(
online_h_s2, next_states_buf, &self.target_dueling,
Some(&iqn_stream_ref), ws_override,
weight_clamp_max_abs,
weight_decay_value,
iqn_beta1,
iqn_beta2,
iqn_epsilon,
) {
Ok(_) => true,
Err(e) => {
tracing::warn!("IQN parallel branch {branch_idx} step failed (non-fatal): {e}");
false
}
};
iqn.deactivate_branch_taus(branch_idx);
if pass_ok {
// Synchronize iqn_stream → main stream so apply_iqn_trunk_gradient
// (cuBLAS, main stream) sees the completed d_h_s2 from this pass.
iqn_ev.record(&iqn_stream_ref)
.map_err(|e| anyhow::anyhow!("iqn branch{branch_idx} event record: {e}"))?;
self.stream.wait(iqn_ev)
.map_err(|e| anyhow::anyhow!("main wait iqn branch{branch_idx}: {e}"))?;
let d_h_s2_ptr = iqn.d_h_s2_raw_ptr();
self.trainer.apply_iqn_trunk_gradient(
d_h_s2_ptr, &mut self.online_dueling, iqn_budget_per_branch,
).map_err(|e| anyhow::anyhow!("IQN trunk gradient branch {branch_idx}: {e}"))?;
// Re-fork iqn_stream off main so the next branch pass starts
// after main has consumed the gradient (prevents use-after-free
// of d_h_s2_buf if iqn_stream were to get ahead of main).
iql_ev.record(&self.stream)
.map_err(|e| anyhow::anyhow!("re-fork iqn branch{branch_idx}: {e}"))?;
iqn_stream_ref.wait(iql_ev)
.map_err(|e| anyhow::anyhow!("iqn_stream re-fork wait {branch_idx}: {e}"))?;
iqn_ok = true;
}
}
// Record completion on iqn_stream.
let iqn_ev = self.iqn_done_event.as_ref().unwrap();
iqn_ev.record(self.iqn_stream.as_ref().unwrap())
// Record final iqn_stream completion for the join below.
iqn_ev.record(&iqn_stream_ref)
.map_err(|e| anyhow::anyhow!("iqn_done_event record: {e}"))?;
}
@@ -2216,13 +2271,11 @@ impl FusedTrainingCtx {
.map_err(|e| anyhow::anyhow!("Attention Adam: {e}"))?;
}
// IQN training pipeline (sequential on main stream).
if let Some(ref mut iqn) = self.gpu_iqn {
let online_h_s2 = self.trainer.save_h_s2();
let next_states_buf = self.trainer.next_states_buf();
// SP4 Layer B (Mech 9): same ISV-driven bound as the
// parallel arm above. Mirror the dqn_adam pattern.
// SP6 Pearl 5: IQN training pipeline — 4 sequential passes on main stream,
// each using a per-branch τ schedule. ÷4 budget normalization preserves
// total IQN gradient magnitude at iqn_budget (same as SP5 Layer B contract).
if has_iqn {
// SP4 Layer B (Mech 9): same ISV-driven bound as the parallel arm.
let (weight_clamp_max_abs, weight_decay_value) =
self.trainer.read_group_adam_bounds(crate::cuda_pipeline::ParamGroup::Iqn);
// SP5 Layer B (Pearl 4): per-group Adam β1/β2/ε for Iqn=3.
@@ -2231,44 +2284,50 @@ impl FusedTrainingCtx {
let iqn_beta2 = self.trainer.read_isv_signal_at(ADAM_BETA2_BASE + iqn_g).clamp(0.99_f32, 0.9995_f32);
let iqn_epsilon = self.trainer.read_isv_signal_at(ADAM_EPS_BASE + iqn_g).clamp(1e-10_f32, 1e-6_f32);
match iqn.execute_training_pipeline(
online_h_s2, next_states_buf, &self.target_dueling,
None, None,
weight_clamp_max_abs,
weight_decay_value,
iqn_beta1,
iqn_beta2,
iqn_epsilon,
) {
Ok(_) => { iqn_ok = true; }
Err(e) => {
tracing::warn!("IQN step failed (non-fatal): {e}");
let iqn_budget_per_branch = iqn_budget / 4.0_f32;
for branch_idx in 0..4_usize {
let online_h_s2 = self.trainer.save_h_s2();
let next_states_buf = self.trainer.next_states_buf();
let iqn = self.gpu_iqn.as_mut().unwrap();
iqn.activate_branch_taus(branch_idx);
let pass_ok = match iqn.execute_training_pipeline(
online_h_s2, next_states_buf, &self.target_dueling,
None, None,
weight_clamp_max_abs,
weight_decay_value,
iqn_beta1,
iqn_beta2,
iqn_epsilon,
) {
Ok(_) => true,
Err(e) => {
tracing::warn!("IQN sequential branch {branch_idx} step failed (non-fatal): {e}");
false
}
};
iqn.deactivate_branch_taus(branch_idx);
if pass_ok {
// Apply this branch's gradient contribution immediately while
// d_h_s2_buf still holds this pass's result.
let d_h_s2_ptr = iqn.d_h_s2_raw_ptr();
self.trainer.apply_iqn_trunk_gradient(
d_h_s2_ptr, &mut self.online_dueling, iqn_budget_per_branch,
).map_err(|e| anyhow::anyhow!("IQN trunk gradient branch {branch_idx}: {e}"))?;
iqn_ok = true;
}
}
}
}
// ── Post-join IQN processing (main stream) ──
// trunk gradient, EMA, IQR run on self.stream (main) because they use
// the trainer's cuBLAS handle which is bound to the main stream.
//
// Task 2.0 — snapshot `grad_buf` branch 0+1 slice BEFORE the IQN
// trunk gradient fires, so the post-IQN reduction sees only the
// additive delta. Snapshot runs unconditionally (even when
// `iqn_ok=false` or `gpu_iqn` is None) so the reduction below can
// also run unconditionally and keep the pinned result slot's IQN
// field (slots [0..2]) populated every step (all-zero delta → 0.0
// ratio, which is the honest signal for a step that skipped IQN).
self.trainer.grad_decomp_snapshot_iqn()
.map_err(|e| anyhow::anyhow!("Task 2.0 grad_decomp_snapshot_iqn: {e}"))?;
// SP6 Pearl 5: apply_iqn_trunk_gradient is now called inline per branch
// (4 × budget/4) inside the parallel/sequential loops above. The snapshot
// was moved to before the fork. Only EMA + IQR run here post-join.
if iqn_ok {
if let Some(ref mut iqn) = self.gpu_iqn {
let d_h_s2_ptr = iqn.d_h_s2_raw_ptr();
self.trainer.apply_iqn_trunk_gradient(
d_h_s2_ptr, &mut self.online_dueling, iqn_budget,
).map_err(|e| anyhow::anyhow!("IQN trunk gradient: {e}"))?;
let tau = self.trainer.read_isv_signal_at(TAU_EFF_INDEX);
iqn.target_ema_update(tau)
.map_err(|e| anyhow::anyhow!("IQN EMA update: {e}"))?;

View File

@@ -159,3 +159,27 @@ Constants live in `crates/ml/src/cuda_pipeline/sp5_isv_slots.rs`.
Kelly slots `[280..286)` are NOT in the fold-reset registry. All other SP5 slots
are per-fold and reset at fold boundaries. Task A0 is reservation-only; producers
and consumers land in subsequent SP5 tasks.
## SP6 Pearl 5: IQN τ per-branch consumer wiring
SP6 Pearl 5 (commit in `sp6-pearl-5` worktree) upgrades the consumer side of
`ISV[250..270)` (`IQN_TAU_BASE`, 4 branches × 5 quantiles).
**Before SP6 (SP5 Layer B contract):** `fused_training.rs` read all 20 slots once
per epoch, averaged 4 branches per quantile to a single `[5]` tau array, and
uploaded it to the shared `online_taus/target_taus/cos_features` buffers.
One IQN forward pass per training step used the averaged schedule.
**After SP6 Pearl 5:** `GpuIqnHead` holds 12 additional `CudaSlice<f32>` buffers:
`online_taus_branch[4]`, `target_taus_branch[4]`, `cos_features_branch[4]`.
`refresh_taus_for_branch(b, tau5)` populates one branch slab per call — no
averaging. `activate_branch_taus(b)` / `deactivate_branch_taus(b)` swap the main
buffers for a per-branch pass via `mem::swap` (no allocation).
Training step: 4 sequential IQN forward passes, each with one branch's τ active.
Each pass calls `apply_iqn_trunk_gradient(iqn_budget / 4.0)` so total IQN
gradient contribution = `iqn_budget` (same as SP5 Layer B). The averaged
`refresh_taus_from_isv` is kept for CVaR and backward-compat consumers.
**Files changed:** `crates/ml/src/cuda_pipeline/gpu_iqn_head.rs`,
`crates/ml/src/trainers/dqn/fused_training.rs`.