feat(sp6): Pearl 5 — IQN τ per-branch schedule (4 forward passes, ÷4 budget normalization)
GpuIqnHead gains 12 new CudaSlice<f32> buffers (online_taus_branch[4], target_taus_branch[4], cos_features_branch[4]) allocated at construction time via alloc_f32. Each slab is [B,N] for taus and [D,N] for cos_features — same sizes as the existing main buffers. refresh_taus_for_branch(branch_idx, tau5): uploads one branch's 5-quantile τ schedule from ISV[IQN_TAU_BASE + b*5 .. +5] to per-branch slabs with cold-start floor (FIXED_TAUS[q] when ISV slot is zero). No cross-branch averaging. activate_branch_taus(b) / deactivate_branch_taus(b): symmetric mem::swap helpers install/restore one branch's slab into self.online_taus/target_taus/cos_features for a per-branch IQN forward pass. activate→deactivate(b) is its own inverse. fused_training.rs: - Tau refresh block calls refresh_taus_for_branch(b, tau5) for all 4 branches, then refresh_taus_from_isv for the averaged main buffer (CVaR backward compat). - grad_decomp_snapshot_iqn() moved BEFORE the parallel/sequential fork so the snapshot is taken before any of the 4 per-branch apply_iqn_trunk_gradient calls. - Parallel path: 4 sequential IQN passes on iqn_stream; after each pass, event sync to main stream, apply_iqn_trunk_gradient(iqn_budget/4), re-fork so next pass starts after main has consumed d_h_s2_buf. iqn_done_event recorded after all 4 passes. - Sequential path: 4 sequential IQN passes on main stream; apply_iqn_trunk_gradient (iqn_budget/4) inline after each pass while d_h_s2_buf holds that branch's result. - Post-join: single apply_iqn_trunk_gradient removed (now inline); target_ema_update and PER loss cast remain. ÷4 normalization: both apply_iqn_trunk_gradient call sites use iqn_budget_per_branch = iqn_budget / 4.0_f32 so 4 × (budget/4) = budget total — matching SP5 Layer B gradient magnitude contract exactly. docs/isv-slots.md: add SP6 Pearl 5 consumer wiring section under the SP5 table. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -325,6 +325,14 @@ pub struct GpuIqnHead {
|
||||
/// Precomputed cosine features [D, N] — cos(π·(d+1)·τ_i)
|
||||
/// Fixed at construction (τ are midpoints). Column-major.
|
||||
cos_features: CudaSlice<f32>,
|
||||
/// SP6 Pearl 5: per-branch τ tensors [B, N], one slab per branch (4 total).
|
||||
/// Populated by `refresh_taus_for_branch`; activated via `activate_branch_taus`
|
||||
/// before each per-branch IQN forward pass. Same allocation size as `online_taus`.
|
||||
online_taus_branch: [CudaSlice<f32>; 4],
|
||||
/// SP6 Pearl 5: per-branch target τ tensors [B, N], one slab per branch.
|
||||
target_taus_branch: [CudaSlice<f32>; 4],
|
||||
/// SP6 Pearl 5: per-branch cosine features [D, N], one slab per branch.
|
||||
cos_features_branch: [CudaSlice<f32>; 4],
|
||||
/// Branch actions decoded from flat actions [B, 4]
|
||||
branch_actions: CudaSlice<i32>,
|
||||
/// Target h_s2 computed from next_states + target trunk weights [B, H]
|
||||
@@ -582,6 +590,21 @@ impl GpuIqnHead {
|
||||
// IQR buffer
|
||||
let iqr_buf = alloc_f32(&stream, tba, "iqn_iqr")?;
|
||||
|
||||
// SP6 Pearl 5: allocate per-branch tau/cos buffer triplets (same size as main buffers).
|
||||
// 4 branches × ([B*N] online + [B*N] target + [D*N] cos).
|
||||
let online_taus_b0 = alloc_f32(&stream, b * n, "iqn_online_taus_b0")?;
|
||||
let online_taus_b1 = alloc_f32(&stream, b * n, "iqn_online_taus_b1")?;
|
||||
let online_taus_b2 = alloc_f32(&stream, b * n, "iqn_online_taus_b2")?;
|
||||
let online_taus_b3 = alloc_f32(&stream, b * n, "iqn_online_taus_b3")?;
|
||||
let target_taus_b0 = alloc_f32(&stream, b * n, "iqn_target_taus_b0")?;
|
||||
let target_taus_b1 = alloc_f32(&stream, b * n, "iqn_target_taus_b1")?;
|
||||
let target_taus_b2 = alloc_f32(&stream, b * n, "iqn_target_taus_b2")?;
|
||||
let target_taus_b3 = alloc_f32(&stream, b * n, "iqn_target_taus_b3")?;
|
||||
let cos_features_b0 = alloc_f32(&stream, d * n, "iqn_cos_features_b0")?;
|
||||
let cos_features_b1 = alloc_f32(&stream, d * n, "iqn_cos_features_b1")?;
|
||||
let cos_features_b2 = alloc_f32(&stream, d * n, "iqn_cos_features_b2")?;
|
||||
let cos_features_b3 = alloc_f32(&stream, d * n, "iqn_cos_features_b3")?;
|
||||
|
||||
let vram_bytes = (total_params * 6 + h * bq * 8 + tba * bq * 4
|
||||
+ d * bq + b * h * 2 + b * 4 + tba + b * 2 + 2) * 4;
|
||||
|
||||
@@ -683,6 +706,9 @@ impl GpuIqnHead {
|
||||
online_taus,
|
||||
target_taus,
|
||||
cos_features,
|
||||
online_taus_branch: [online_taus_b0, online_taus_b1, online_taus_b2, online_taus_b3],
|
||||
target_taus_branch: [target_taus_b0, target_taus_b1, target_taus_b2, target_taus_b3],
|
||||
cos_features_branch: [cos_features_b0, cos_features_b1, cos_features_b2, cos_features_b3],
|
||||
branch_actions,
|
||||
target_h_s2,
|
||||
cached_target_h_s2_ptr: None,
|
||||
@@ -2016,6 +2042,89 @@ impl GpuIqnHead {
|
||||
}
|
||||
}
|
||||
|
||||
/// SP6 Pearl 5: Refresh the τ tensors for a single branch from its 5-quantile
|
||||
/// schedule `tau5` (from ISV[IQN_TAU_BASE + branch_idx*5 .. +5]).
|
||||
///
|
||||
/// Uploads to `online_taus_branch[branch_idx]`, `target_taus_branch[branch_idx]`,
|
||||
/// and `cos_features_branch[branch_idx]`. Called 4 times per epoch refresh (once
|
||||
/// per branch) from `FusedTrainingCtx::run_full_step`.
|
||||
///
|
||||
/// Cold-start floor: if a tau value is below 1e-6 (ISV not yet populated),
|
||||
/// uses `FIXED_TAUS[q]` to preserve the interpretable quantile distribution.
|
||||
pub fn refresh_taus_for_branch(
|
||||
&mut self,
|
||||
branch_idx: usize,
|
||||
tau5: &[f32; 5],
|
||||
) {
|
||||
debug_assert!(branch_idx < 4, "refresh_taus_for_branch: branch_idx {branch_idx} >= 4");
|
||||
let n = self.config.num_quantiles; // 5
|
||||
let b = self.config.batch_size;
|
||||
let d = self.config.embed_dim;
|
||||
|
||||
// Apply cold-start floor: if ISV slot is zero (not yet populated),
|
||||
// fall back to FIXED_TAUS[q] so we don't collapse the quantile distribution.
|
||||
let mut taus = [0.0_f32; 5];
|
||||
for q in 0..n {
|
||||
taus[q] = if tau5[q] < 1e-6_f32 { FIXED_TAUS[q] } else { tau5[q].min(1.0_f32) };
|
||||
}
|
||||
|
||||
// Recompute cosine features [D, N]: cos_feat[dim + q*D] = cos(π·(d+1)·τ_q)
|
||||
let mut cos_feat_host = vec![0.0_f32; d * n];
|
||||
for q in 0..n {
|
||||
let tau_q = taus[q];
|
||||
for dim in 0..d {
|
||||
cos_feat_host[dim + q * d] =
|
||||
(std::f32::consts::PI * ((dim + 1) as f32) * tau_q).cos();
|
||||
}
|
||||
}
|
||||
|
||||
// Tile taus [B, N].
|
||||
let mut tiled = Vec::with_capacity(b * n);
|
||||
for _ in 0..b {
|
||||
tiled.extend_from_slice(&taus[..n]);
|
||||
}
|
||||
|
||||
if let Err(e) = super::mapped_pinned::upload_f32_via_pinned(
|
||||
&self.stream, &tiled, &mut self.online_taus_branch[branch_idx],
|
||||
) {
|
||||
tracing::warn!("SP6 Pearl 5 online_taus_branch[{branch_idx}] upload failed: {e}");
|
||||
return;
|
||||
}
|
||||
if let Err(e) = super::mapped_pinned::upload_f32_via_pinned(
|
||||
&self.stream, &tiled, &mut self.target_taus_branch[branch_idx],
|
||||
) {
|
||||
tracing::warn!("SP6 Pearl 5 target_taus_branch[{branch_idx}] upload failed: {e}");
|
||||
return;
|
||||
}
|
||||
if let Err(e) = super::mapped_pinned::upload_f32_via_pinned(
|
||||
&self.stream, &cos_feat_host, &mut self.cos_features_branch[branch_idx],
|
||||
) {
|
||||
tracing::warn!("SP6 Pearl 5 cos_features_branch[{branch_idx}] upload failed: {e}");
|
||||
}
|
||||
}
|
||||
|
||||
/// SP6 Pearl 5: Swap the main tau/cos buffers with branch `branch_idx`'s per-branch
|
||||
/// buffers so the IQN forward pass uses that branch's τ schedule.
|
||||
///
|
||||
/// Must be paired with `deactivate_branch_taus(branch_idx)` after the forward pass.
|
||||
/// `activate` followed by `deactivate` with the same index is its own inverse.
|
||||
pub fn activate_branch_taus(&mut self, branch_idx: usize) {
|
||||
debug_assert!(branch_idx < 4, "activate_branch_taus: branch_idx {branch_idx} >= 4");
|
||||
std::mem::swap(&mut self.online_taus, &mut self.online_taus_branch[branch_idx]);
|
||||
std::mem::swap(&mut self.target_taus, &mut self.target_taus_branch[branch_idx]);
|
||||
std::mem::swap(&mut self.cos_features, &mut self.cos_features_branch[branch_idx]);
|
||||
}
|
||||
|
||||
/// SP6 Pearl 5: Restore the main tau/cos buffers after a per-branch IQN forward pass.
|
||||
///
|
||||
/// Symmetric inverse of `activate_branch_taus(branch_idx)`.
|
||||
pub fn deactivate_branch_taus(&mut self, branch_idx: usize) {
|
||||
debug_assert!(branch_idx < 4, "deactivate_branch_taus: branch_idx {branch_idx} >= 4");
|
||||
std::mem::swap(&mut self.online_taus, &mut self.online_taus_branch[branch_idx]);
|
||||
std::mem::swap(&mut self.target_taus, &mut self.target_taus_branch[branch_idx]);
|
||||
std::mem::swap(&mut self.cos_features, &mut self.cos_features_branch[branch_idx]);
|
||||
}
|
||||
|
||||
/// Compute CVaR-based position scaling from IQN quantiles.
|
||||
///
|
||||
/// Runs the IQN forward-only kernel on `h_s2` (trunk activation),
|
||||
|
||||
@@ -2011,15 +2011,25 @@ impl FusedTrainingCtx {
|
||||
// ── IQN prep: decode actions + DtoD copies (main stream, before fork) ──
|
||||
// Must complete before execute_training_pipeline runs on iqn_stream.
|
||||
if let Some(ref mut iqn) = self.gpu_iqn {
|
||||
// SP5 Pearl 5: refresh τ schedule from ISV[IQN_TAU_BASE=250..270).
|
||||
// Read 4 branches × 5 quantiles, average per quantile, apply cold-start
|
||||
// floor from FIXED_TAUS, re-upload online_taus/target_taus/cos_features.
|
||||
// Must run before prepare_buffers so the kernel sees updated tau values.
|
||||
// SP6 Pearl 5: refresh per-branch τ schedules from ISV[250..270).
|
||||
// ISV layout: branch b, quantile q → IQN_TAU_BASE + b*5 + q.
|
||||
// Each branch gets its own [B,N] tau slab; the main (averaged) buffer
|
||||
// is also kept current via refresh_taus_from_isv for CVaR and any
|
||||
// consumer that has not migrated to the per-branch path.
|
||||
{
|
||||
let mut isv_taus = [0.0_f32; 20];
|
||||
for i in 0..20 {
|
||||
isv_taus[i] = self.trainer.read_isv_signal_at(IQN_TAU_BASE + i);
|
||||
}
|
||||
// Populate per-branch slabs first.
|
||||
for branch_idx in 0..4_usize {
|
||||
let mut tau5 = [0.0_f32; 5];
|
||||
for q in 0..5_usize {
|
||||
tau5[q] = isv_taus[branch_idx * 5 + q];
|
||||
}
|
||||
iqn.refresh_taus_for_branch(branch_idx, &tau5);
|
||||
}
|
||||
// Keep the averaged main buffer current for CVaR and backward-compat consumers.
|
||||
iqn.refresh_taus_from_isv(&isv_taus);
|
||||
}
|
||||
iqn.set_cached_target_h_s2(self.trainer.tg_h_s2_ptr());
|
||||
@@ -2042,6 +2052,13 @@ impl FusedTrainingCtx {
|
||||
let parallel = self.iqn_stream.is_some();
|
||||
let mut iqn_ok = false;
|
||||
|
||||
// SP6 Pearl 5 / Task 2.0: snapshot grad_buf branch 0+1 slice BEFORE any
|
||||
// IQN trunk gradient fires. The parallel and sequential paths both call
|
||||
// apply_iqn_trunk_gradient inline (4 passes × budget/4), so the snapshot
|
||||
// must precede the fork to correctly isolate the IQN contribution delta.
|
||||
self.trainer.grad_decomp_snapshot_iqn()
|
||||
.map_err(|e| anyhow::anyhow!("Task 2.0 grad_decomp_snapshot_iqn: {e}"))?;
|
||||
|
||||
if parallel {
|
||||
// Record fork-point event on main stream.
|
||||
let iql_ev = self.iql_done_event.as_ref()
|
||||
@@ -2050,6 +2067,10 @@ impl FusedTrainingCtx {
|
||||
.map_err(|e| anyhow::anyhow!("iql_done_event record: {e}"))?;
|
||||
|
||||
// ── Fork IQN to iqn_stream ──
|
||||
// SP6 Pearl 5: run 4 sequential IQN forward passes on iqn_stream, each
|
||||
// with its per-branch τ schedule active. After each pass, synchronize to
|
||||
// the main stream and apply iqn_budget/4 trunk gradient contribution so
|
||||
// all 4 branches' d_h_s2 values are folded into grad_buf equally.
|
||||
if has_iqn {
|
||||
let iqn_s = self.iqn_stream.as_ref().unwrap();
|
||||
iqn_s.wait(iql_ev)
|
||||
@@ -2059,10 +2080,6 @@ impl FusedTrainingCtx {
|
||||
let ws_override = Some((iqn_ws.raw_ptr(), 32 * 1024 * 1024_usize));
|
||||
let iqn_stream_ref = Arc::clone(iqn_s);
|
||||
|
||||
let iqn = self.gpu_iqn.as_mut().unwrap();
|
||||
let online_h_s2 = self.trainer.save_h_s2();
|
||||
let next_states_buf = self.trainer.next_states_buf();
|
||||
|
||||
// SP4 Layer B (Mech 9): ISV-driven post-Adam |p| clamp for
|
||||
// `iqn_adam_kernel`. IQN params live in group 3
|
||||
// (`ParamGroup::Iqn`); `read_group_adam_bounds` returns
|
||||
@@ -2077,24 +2094,62 @@ impl FusedTrainingCtx {
|
||||
let iqn_beta2 = self.trainer.read_isv_signal_at(ADAM_BETA2_BASE + iqn_g).clamp(0.99_f32, 0.9995_f32);
|
||||
let iqn_epsilon = self.trainer.read_isv_signal_at(ADAM_EPS_BASE + iqn_g).clamp(1e-10_f32, 1e-6_f32);
|
||||
|
||||
match iqn.execute_training_pipeline(
|
||||
online_h_s2, next_states_buf, &self.target_dueling,
|
||||
Some(&iqn_stream_ref), ws_override,
|
||||
weight_clamp_max_abs,
|
||||
weight_decay_value,
|
||||
iqn_beta1,
|
||||
iqn_beta2,
|
||||
iqn_epsilon,
|
||||
) {
|
||||
Ok(_) => { iqn_ok = true; }
|
||||
Err(e) => {
|
||||
tracing::warn!("IQN parallel step failed (non-fatal): {e}");
|
||||
// ÷4 normalization: each branch pass contributes iqn_budget/4 to
|
||||
// grad_buf via apply_iqn_trunk_gradient; 4 passes × (budget/4) =
|
||||
// budget total, preserving SP5 Layer B gradient magnitude contract.
|
||||
let iqn_budget_per_branch = iqn_budget / 4.0_f32;
|
||||
let iqn_ev = self.iqn_done_event.as_ref().unwrap();
|
||||
|
||||
for branch_idx in 0..4_usize {
|
||||
let online_h_s2 = self.trainer.save_h_s2();
|
||||
let next_states_buf = self.trainer.next_states_buf();
|
||||
let iqn = self.gpu_iqn.as_mut().unwrap();
|
||||
|
||||
iqn.activate_branch_taus(branch_idx);
|
||||
let pass_ok = match iqn.execute_training_pipeline(
|
||||
online_h_s2, next_states_buf, &self.target_dueling,
|
||||
Some(&iqn_stream_ref), ws_override,
|
||||
weight_clamp_max_abs,
|
||||
weight_decay_value,
|
||||
iqn_beta1,
|
||||
iqn_beta2,
|
||||
iqn_epsilon,
|
||||
) {
|
||||
Ok(_) => true,
|
||||
Err(e) => {
|
||||
tracing::warn!("IQN parallel branch {branch_idx} step failed (non-fatal): {e}");
|
||||
false
|
||||
}
|
||||
};
|
||||
iqn.deactivate_branch_taus(branch_idx);
|
||||
|
||||
if pass_ok {
|
||||
// Synchronize iqn_stream → main stream so apply_iqn_trunk_gradient
|
||||
// (cuBLAS, main stream) sees the completed d_h_s2 from this pass.
|
||||
iqn_ev.record(&iqn_stream_ref)
|
||||
.map_err(|e| anyhow::anyhow!("iqn branch{branch_idx} event record: {e}"))?;
|
||||
self.stream.wait(iqn_ev)
|
||||
.map_err(|e| anyhow::anyhow!("main wait iqn branch{branch_idx}: {e}"))?;
|
||||
|
||||
let d_h_s2_ptr = iqn.d_h_s2_raw_ptr();
|
||||
self.trainer.apply_iqn_trunk_gradient(
|
||||
d_h_s2_ptr, &mut self.online_dueling, iqn_budget_per_branch,
|
||||
).map_err(|e| anyhow::anyhow!("IQN trunk gradient branch {branch_idx}: {e}"))?;
|
||||
|
||||
// Re-fork iqn_stream off main so the next branch pass starts
|
||||
// after main has consumed the gradient (prevents use-after-free
|
||||
// of d_h_s2_buf if iqn_stream were to get ahead of main).
|
||||
iql_ev.record(&self.stream)
|
||||
.map_err(|e| anyhow::anyhow!("re-fork iqn branch{branch_idx}: {e}"))?;
|
||||
iqn_stream_ref.wait(iql_ev)
|
||||
.map_err(|e| anyhow::anyhow!("iqn_stream re-fork wait {branch_idx}: {e}"))?;
|
||||
|
||||
iqn_ok = true;
|
||||
}
|
||||
}
|
||||
|
||||
// Record completion on iqn_stream.
|
||||
let iqn_ev = self.iqn_done_event.as_ref().unwrap();
|
||||
iqn_ev.record(self.iqn_stream.as_ref().unwrap())
|
||||
// Record final iqn_stream completion for the join below.
|
||||
iqn_ev.record(&iqn_stream_ref)
|
||||
.map_err(|e| anyhow::anyhow!("iqn_done_event record: {e}"))?;
|
||||
}
|
||||
|
||||
@@ -2216,13 +2271,11 @@ impl FusedTrainingCtx {
|
||||
.map_err(|e| anyhow::anyhow!("Attention Adam: {e}"))?;
|
||||
}
|
||||
|
||||
// IQN training pipeline (sequential on main stream).
|
||||
if let Some(ref mut iqn) = self.gpu_iqn {
|
||||
let online_h_s2 = self.trainer.save_h_s2();
|
||||
let next_states_buf = self.trainer.next_states_buf();
|
||||
|
||||
// SP4 Layer B (Mech 9): same ISV-driven bound as the
|
||||
// parallel arm above. Mirror the dqn_adam pattern.
|
||||
// SP6 Pearl 5: IQN training pipeline — 4 sequential passes on main stream,
|
||||
// each using a per-branch τ schedule. ÷4 budget normalization preserves
|
||||
// total IQN gradient magnitude at iqn_budget (same as SP5 Layer B contract).
|
||||
if has_iqn {
|
||||
// SP4 Layer B (Mech 9): same ISV-driven bound as the parallel arm.
|
||||
let (weight_clamp_max_abs, weight_decay_value) =
|
||||
self.trainer.read_group_adam_bounds(crate::cuda_pipeline::ParamGroup::Iqn);
|
||||
// SP5 Layer B (Pearl 4): per-group Adam β1/β2/ε for Iqn=3.
|
||||
@@ -2231,44 +2284,50 @@ impl FusedTrainingCtx {
|
||||
let iqn_beta2 = self.trainer.read_isv_signal_at(ADAM_BETA2_BASE + iqn_g).clamp(0.99_f32, 0.9995_f32);
|
||||
let iqn_epsilon = self.trainer.read_isv_signal_at(ADAM_EPS_BASE + iqn_g).clamp(1e-10_f32, 1e-6_f32);
|
||||
|
||||
match iqn.execute_training_pipeline(
|
||||
online_h_s2, next_states_buf, &self.target_dueling,
|
||||
None, None,
|
||||
weight_clamp_max_abs,
|
||||
weight_decay_value,
|
||||
iqn_beta1,
|
||||
iqn_beta2,
|
||||
iqn_epsilon,
|
||||
) {
|
||||
Ok(_) => { iqn_ok = true; }
|
||||
Err(e) => {
|
||||
tracing::warn!("IQN step failed (non-fatal): {e}");
|
||||
let iqn_budget_per_branch = iqn_budget / 4.0_f32;
|
||||
|
||||
for branch_idx in 0..4_usize {
|
||||
let online_h_s2 = self.trainer.save_h_s2();
|
||||
let next_states_buf = self.trainer.next_states_buf();
|
||||
let iqn = self.gpu_iqn.as_mut().unwrap();
|
||||
|
||||
iqn.activate_branch_taus(branch_idx);
|
||||
let pass_ok = match iqn.execute_training_pipeline(
|
||||
online_h_s2, next_states_buf, &self.target_dueling,
|
||||
None, None,
|
||||
weight_clamp_max_abs,
|
||||
weight_decay_value,
|
||||
iqn_beta1,
|
||||
iqn_beta2,
|
||||
iqn_epsilon,
|
||||
) {
|
||||
Ok(_) => true,
|
||||
Err(e) => {
|
||||
tracing::warn!("IQN sequential branch {branch_idx} step failed (non-fatal): {e}");
|
||||
false
|
||||
}
|
||||
};
|
||||
iqn.deactivate_branch_taus(branch_idx);
|
||||
|
||||
if pass_ok {
|
||||
// Apply this branch's gradient contribution immediately while
|
||||
// d_h_s2_buf still holds this pass's result.
|
||||
let d_h_s2_ptr = iqn.d_h_s2_raw_ptr();
|
||||
self.trainer.apply_iqn_trunk_gradient(
|
||||
d_h_s2_ptr, &mut self.online_dueling, iqn_budget_per_branch,
|
||||
).map_err(|e| anyhow::anyhow!("IQN trunk gradient branch {branch_idx}: {e}"))?;
|
||||
iqn_ok = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ── Post-join IQN processing (main stream) ──
|
||||
// trunk gradient, EMA, IQR run on self.stream (main) because they use
|
||||
// the trainer's cuBLAS handle which is bound to the main stream.
|
||||
//
|
||||
// Task 2.0 — snapshot `grad_buf` branch 0+1 slice BEFORE the IQN
|
||||
// trunk gradient fires, so the post-IQN reduction sees only the
|
||||
// additive delta. Snapshot runs unconditionally (even when
|
||||
// `iqn_ok=false` or `gpu_iqn` is None) so the reduction below can
|
||||
// also run unconditionally and keep the pinned result slot's IQN
|
||||
// field (slots [0..2]) populated every step (all-zero delta → 0.0
|
||||
// ratio, which is the honest signal for a step that skipped IQN).
|
||||
self.trainer.grad_decomp_snapshot_iqn()
|
||||
.map_err(|e| anyhow::anyhow!("Task 2.0 grad_decomp_snapshot_iqn: {e}"))?;
|
||||
|
||||
// SP6 Pearl 5: apply_iqn_trunk_gradient is now called inline per branch
|
||||
// (4 × budget/4) inside the parallel/sequential loops above. The snapshot
|
||||
// was moved to before the fork. Only EMA + IQR run here post-join.
|
||||
if iqn_ok {
|
||||
if let Some(ref mut iqn) = self.gpu_iqn {
|
||||
let d_h_s2_ptr = iqn.d_h_s2_raw_ptr();
|
||||
self.trainer.apply_iqn_trunk_gradient(
|
||||
d_h_s2_ptr, &mut self.online_dueling, iqn_budget,
|
||||
).map_err(|e| anyhow::anyhow!("IQN trunk gradient: {e}"))?;
|
||||
|
||||
let tau = self.trainer.read_isv_signal_at(TAU_EFF_INDEX);
|
||||
iqn.target_ema_update(tau)
|
||||
.map_err(|e| anyhow::anyhow!("IQN EMA update: {e}"))?;
|
||||
|
||||
@@ -159,3 +159,27 @@ Constants live in `crates/ml/src/cuda_pipeline/sp5_isv_slots.rs`.
|
||||
Kelly slots `[280..286)` are NOT in the fold-reset registry. All other SP5 slots
|
||||
are per-fold and reset at fold boundaries. Task A0 is reservation-only; producers
|
||||
and consumers land in subsequent SP5 tasks.
|
||||
|
||||
## SP6 Pearl 5: IQN τ per-branch consumer wiring
|
||||
|
||||
SP6 Pearl 5 (commit in `sp6-pearl-5` worktree) upgrades the consumer side of
|
||||
`ISV[250..270)` (`IQN_TAU_BASE`, 4 branches × 5 quantiles).
|
||||
|
||||
**Before SP6 (SP5 Layer B contract):** `fused_training.rs` read all 20 slots once
|
||||
per epoch, averaged 4 branches per quantile to a single `[5]` tau array, and
|
||||
uploaded it to the shared `online_taus/target_taus/cos_features` buffers.
|
||||
One IQN forward pass per training step used the averaged schedule.
|
||||
|
||||
**After SP6 Pearl 5:** `GpuIqnHead` holds 12 additional `CudaSlice<f32>` buffers:
|
||||
`online_taus_branch[4]`, `target_taus_branch[4]`, `cos_features_branch[4]`.
|
||||
`refresh_taus_for_branch(b, tau5)` populates one branch slab per call — no
|
||||
averaging. `activate_branch_taus(b)` / `deactivate_branch_taus(b)` swap the main
|
||||
buffers for a per-branch pass via `mem::swap` (no allocation).
|
||||
|
||||
Training step: 4 sequential IQN forward passes, each with one branch's τ active.
|
||||
Each pass calls `apply_iqn_trunk_gradient(iqn_budget / 4.0)` so total IQN
|
||||
gradient contribution = `iqn_budget` (same as SP5 Layer B). The averaged
|
||||
`refresh_taus_from_isv` is kept for CVaR and backward-compat consumers.
|
||||
|
||||
**Files changed:** `crates/ml/src/cuda_pipeline/gpu_iqn_head.rs`,
|
||||
`crates/ml/src/trainers/dqn/fused_training.rs`.
|
||||
|
||||
Reference in New Issue
Block a user