gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(sp5): Task A4 — Pearl 4 per-group Adam β1/β2/ε

Browse Source 
Per-param-group Adam hyperparameters from per-group gradient
direction-stability EMA + L2 norm. 8 SP4 param groups × 3 hparams = 24
ISV slots [226..250).

Two-kernel chain:
  grad_cosine_sim_update — per-group cosine_sim of curr vs prev grads
                          + L2 norm; writes back grad_curr → grad_prev
                          for next step's comparison.
  pearl_4_adam_hparams_update — β1/β2/ε from cosine + l2_norm.

Structural envelopes (Invariant 1 anchors per spec line 88):
  β1 ∈ [0.85, 0.95]; β2 ∈ [0.99, 0.9995]; ε ∈ [1e-10, 1e-6]

ALPHA_META migration (Option A — atomic shared-contract migration per
feedback_no_partial_refactor): apply_pearls_ad_kernel.cu and the
launch_apply_pearls Rust wrapper now take alpha_meta as a parameter.
All 45 existing call sites (SP4 + SP5 producers) pass the prior default
1.0e-3 explicitly via crate::cuda_pipeline::sp4_wiener_ema::ALPHA_META.
Pearl 4's apply_pearls calls pass 5.0e-4 (half the default per spec
line 89) to limit β change rate.

Theoretical caveat: adaptive β breaks Adam's constant-β convergence
proof. Mitigations: structural envelopes + halved ALPHA_META + Pearls
A+D smoothing. Fall-back path defined: revert + ε-only adaptive
variant if Layer C destabilization observed.

New mapped-pinned buffer: grad_prev_buf_per_group [TOTAL_PARAMS] for
cosine-sim previous-step direction storage. Mapped-pinned i32 mirror
of grad_buf layout.

producer_step_scratch_buf grew 131 → 171 (40 new outputs: 8 cosine_sim
+ 8 l2_norm + 24 β1/β2/ε). wiener_state_buf already at 543 (A1 sized
for entire SP5 block).

StateResetRegistry: 4 new FoldReset entries: sp5_adam_beta1,
sp5_adam_beta2, sp5_adam_eps, sp5_grad_prev_buf. All sentinel 0 →
bootstrap to envelope midpoint via Pearls A+D + cosine_sim=0 fall-back
on first step of new fold.

Adam-launcher consumer migration deferred to Layer B.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-01 22:18:10 +02:00
parent 71a0275f54
commit 67dd414cb3
11 changed files with 744 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files

244
crates/ml/tests/sp5_producer_unit_tests.rs
View File

@@ -33,6 +33,9 @@ const SP5_PEARL_3_SIGMA_CUBIN: &[u8] =
const SP5_PEARL_2_BUDGET_CUBIN: &[u8] =
include_bytes!(concat!(env!("OUT_DIR"), "/pearl_2_budget_kernel.cubin"));
const SP5_PEARL_4_ADAM_HPARAMS_CUBIN: &[u8] =
include_bytes!(concat!(env!("OUT_DIR"), "/pearl_4_adam_hparams_kernel.cubin"));
// ── Helpers ───────────────────────────────────────────────────────────────────
fn load_pearl_3_sigma_kernel(stream: &Arc<CudaStream>) -> CudaFunction {
@@ -55,6 +58,16 @@ fn load_pearl_2_budget_kernel(stream: &Arc<CudaStream>) -> CudaFunction {
.expect("load pearl_2_budget_update function")
}
fn load_pearl_4_adam_hparams_kernel(stream: &Arc<CudaStream>) -> CudaFunction {
let module = stream
.context()
.load_cubin(SP5_PEARL_4_ADAM_HPARAMS_CUBIN.to_vec())
.expect("load pearl_4_adam_hparams_kernel cubin");
module
.load_function("pearl_4_adam_hparams_update")
.expect("load pearl_4_adam_hparams_update function")
}
fn make_test_stream() -> Arc<CudaStream> {
let ctx = CudaContext::new(0).expect("CUDA context — is a GPU available?");
ctx.default_stream()
@@ -861,3 +874,234 @@ fn pearl_2_budget_sharp_regime_iqn_dominates() {
);
}
}
// ── Tests 7 + 8: pearl_4_adam_hparams_update ──────────────────────────────────
/// SP5 Task A4 unit test: `pearl_4_adam_hparams_update` with high stability input.
///
/// Synthetic input: cosine_sim=[1.0; 8] (maximum stability), grad_norm=[1.0; 8].
///
/// Analytical expected:
/// stability = fmaxf(0, fminf(1, cosine_sim)) = 1.0
/// β1 = 0.85 + 0.10 × 1.0 = 0.95 (long-memory ceiling)
/// β2 = 0.99 + 0.0095 × 1.0 = 0.9995 (long-memory ceiling)
/// ε = clamp(1.0 × 1e-7, 1e-10, 1e-6) = 1e-7 → clamped to 1e-7 (within [1e-10, 1e-6])
///
/// Structural envelopes (Invariant 1 anchors): β1∈[0.85,0.95], β2∈[0.99,0.9995].
/// No CPU reference oracle per `feedback_no_cpu_test_fallbacks.md`.
#[test]
#[ignore = "requires GPU"]
fn pearl_4_adam_hparams_high_stability_yields_long_memory() {
// Scratch layout (Pearl 4 outputs):
// [0..8) = β1[8] (beta1_idx_base=0)
// [8..16) = β2[8] (beta2_idx_base=8)
// [16..24) = ε[8] (eps_idx_base=16)
const BETA1_BASE: usize = 0;
const BETA2_BASE: usize = 8;
const EPS_BASE: usize = 16;
const SCRATCH_SIZE: usize = 24;
const N_GROUPS: usize = 8;
let stream = make_test_stream();
let kernel = load_pearl_4_adam_hparams_kernel(&stream);
// cosine_sim = 1.0 for all 8 groups → maximum stability.
let cosine_buf = unsafe { MappedF32Buffer::new(N_GROUPS) }
.expect("alloc cosine_buf");
cosine_buf.write_from_slice(&[1.0_f32; N_GROUPS]);
// grad_norm = 1.0 for all 8 groups → ε = 1e-7 (within envelope).
let l2_norm_buf = unsafe { MappedF32Buffer::new(N_GROUPS) }
.expect("alloc l2_norm_buf");
l2_norm_buf.write_from_slice(&[1.0_f32; N_GROUPS]);
let scratch_buf = unsafe { MappedF32Buffer::new(SCRATCH_SIZE) }
.expect("alloc scratch_buf");
let cosine_dev = cosine_buf.dev_ptr;
let l2_dev = l2_norm_buf.dev_ptr;
let scratch_dev = scratch_buf.dev_ptr;
let beta1_base_i32 = BETA1_BASE as i32;
let beta2_base_i32 = BETA2_BASE as i32;
let eps_base_i32 = EPS_BASE as i32;
unsafe {
stream
.launch_builder(&kernel)
.arg(&cosine_dev)
.arg(&l2_dev)
.arg(&scratch_dev)
.arg(&beta1_base_i32)
.arg(&beta2_base_i32)
.arg(&eps_base_i32)
.launch(LaunchConfig {
grid_dim: (1, 1, 1),
block_dim: (8, 1, 1),
shared_mem_bytes: 0,
})
.expect("launch pearl_4_adam_hparams_update (high stability)");
}
stream.synchronize().expect("sync after pearl_4_adam_hparams_update (high stability)");
let scratch = scratch_buf.read_all();
// Analytical values for stability = 1.0:
let expected_beta1: f32 = 0.85 + 0.10 * 1.0; // = 0.95
let expected_beta2: f32 = 0.99 + 0.0095 * 1.0; // = 0.9995
// ε = clamp(1.0 × 1e-7, 1e-10, 1e-6) = 1e-7
let expected_eps: f32 = 1.0_f32 * 1e-7_f32; // within [1e-10, 1e-6]
for g in 0..N_GROUPS {
let beta1 = scratch[BETA1_BASE + g];
let beta2 = scratch[BETA2_BASE + g];
let eps = scratch[EPS_BASE + g];
println!("group={g} beta1={beta1:.6} beta2={beta6:.7} eps={eps:.2e}",
beta6 = beta2);
// β1 ceiling: 0.95 ± 1 ULP.
let rel_b1 = (beta1 - expected_beta1).abs() / expected_beta1.max(1e-9);
assert!(
rel_b1 < 1e-5,
"group={g}: beta1={beta1:.6} expected={expected_beta1:.6} rel={rel_b1:.2e}"
);
// β2 ceiling: 0.9995 ± 1 ULP.
let rel_b2 = (beta2 - expected_beta2).abs() / expected_beta2.max(1e-9);
assert!(
rel_b2 < 1e-5,
"group={g}: beta2={beta2:.7} expected={expected_beta2:.7} rel={rel_b2:.2e}"
);
// ε must be within structural envelope.
assert!(
eps >= 1e-10_f32 && eps <= 1e-6_f32,
"group={g}: eps={eps:.2e} outside structural envelope [1e-10, 1e-6]"
);
let rel_eps = (eps - expected_eps).abs() / expected_eps.max(1e-15);
assert!(
rel_eps < 1e-4,
"group={g}: eps={eps:.2e} expected={expected_eps:.2e} rel={rel_eps:.2e}"
);
// β1 must be at structural ceiling for maximum stability.
assert!(
(beta1 - 0.95_f32).abs() < 1e-5,
"group={g}: beta1={beta1:.6} should be at ceiling 0.95 for stability=1.0"
);
// β2 must be at structural ceiling for maximum stability.
assert!(
(beta2 - 0.9995_f32).abs() < 1e-5,
"group={g}: beta2={beta2:.7} should be at ceiling 0.9995 for stability=1.0"
);
}
}
/// SP5 Task A4 unit test: `pearl_4_adam_hparams_update` with low stability input.
///
/// Synthetic input: cosine_sim=[0.0; 8] (minimum stability), grad_norm=[0.01; 8].
///
/// Analytical expected:
/// stability = fmaxf(0, fminf(1, 0.0)) = 0.0
/// β1 = 0.85 + 0.10 × 0.0 = 0.85 (low-stability floor)
/// β2 = 0.99 + 0.0095 × 0.0 = 0.99 (low-stability floor)
/// ε = clamp(0.01 × 1e-7, 1e-10, 1e-6) = 1e-9 → clamped to 1e-10
///
/// No CPU reference oracle per `feedback_no_cpu_test_fallbacks.md`.
#[test]
#[ignore = "requires GPU"]
fn pearl_4_adam_hparams_low_stability_yields_short_memory() {
const BETA1_BASE: usize = 0;
const BETA2_BASE: usize = 8;
const EPS_BASE: usize = 16;
const SCRATCH_SIZE: usize = 24;
const N_GROUPS: usize = 8;
let stream = make_test_stream();
let kernel = load_pearl_4_adam_hparams_kernel(&stream);
// cosine_sim = 0.0 for all 8 groups → minimum stability.
let cosine_buf = unsafe { MappedF32Buffer::new(N_GROUPS) }
.expect("alloc cosine_buf");
cosine_buf.write_from_slice(&[0.0_f32; N_GROUPS]);
// grad_norm = 0.01 → ε = 0.01 × 1e-7 = 1e-9 → clamped to 1e-10.
let l2_norm_buf = unsafe { MappedF32Buffer::new(N_GROUPS) }
.expect("alloc l2_norm_buf");
l2_norm_buf.write_from_slice(&[0.01_f32; N_GROUPS]);
let scratch_buf = unsafe { MappedF32Buffer::new(SCRATCH_SIZE) }
.expect("alloc scratch_buf");
let cosine_dev = cosine_buf.dev_ptr;
let l2_dev = l2_norm_buf.dev_ptr;
let scratch_dev = scratch_buf.dev_ptr;
let beta1_base_i32 = BETA1_BASE as i32;
let beta2_base_i32 = BETA2_BASE as i32;
let eps_base_i32 = EPS_BASE as i32;
unsafe {
stream
.launch_builder(&kernel)
.arg(&cosine_dev)
.arg(&l2_dev)
.arg(&scratch_dev)
.arg(&beta1_base_i32)
.arg(&beta2_base_i32)
.arg(&eps_base_i32)
.launch(LaunchConfig {
grid_dim: (1, 1, 1),
block_dim: (8, 1, 1),
shared_mem_bytes: 0,
})
.expect("launch pearl_4_adam_hparams_update (low stability)");
}
stream.synchronize().expect("sync after pearl_4_adam_hparams_update (low stability)");
let scratch = scratch_buf.read_all();
// Analytical values for stability = 0.0:
let expected_beta1: f32 = 0.85; // low-stability floor
let expected_beta2: f32 = 0.99; // low-stability floor
// ε = clamp(0.01 × 1e-7, 1e-10, 1e-6) = clamp(1e-9, 1e-10, 1e-6) = 1e-9
// Actually 1e-9 is within [1e-10, 1e-6], so no clamping needed.
let expected_eps: f32 = (0.01_f32 * 1e-7_f32).clamp(1e-10, 1e-6); // = 1e-9
for g in 0..N_GROUPS {
let beta1 = scratch[BETA1_BASE + g];
let beta2 = scratch[BETA2_BASE + g];
let eps = scratch[EPS_BASE + g];
println!("group={g} beta1={beta1:.6} beta2={beta2:.7} eps={eps:.2e}");
// β1 floor: 0.85 ± 1 ULP.
let rel_b1 = (beta1 - expected_beta1).abs() / expected_beta1.max(1e-9);
assert!(
rel_b1 < 1e-5,
"group={g}: beta1={beta1:.6} expected={expected_beta1:.6} rel={rel_b1:.2e}"
);
// β2 floor: 0.99 ± 1 ULP.
let rel_b2 = (beta2 - expected_beta2).abs() / expected_beta2.max(1e-9);
assert!(
rel_b2 < 1e-5,
"group={g}: beta2={beta2:.7} expected={expected_beta2:.7} rel={rel_b2:.2e}"
);
// ε must be within structural envelope.
assert!(
eps >= 1e-10_f32 && eps <= 1e-6_f32,
"group={g}: eps={eps:.2e} outside structural envelope [1e-10, 1e-6]"
);
let rel_eps = (eps - expected_eps).abs() / expected_eps.max(1e-15);
assert!(
rel_eps < 1e-4,
"group={g}: eps={eps:.2e} expected={expected_eps:.2e} rel={rel_eps:.2e}"
);
// β1 must be at structural floor for zero stability.
assert!(
(beta1 - 0.85_f32).abs() < 1e-5,
"group={g}: beta1={beta1:.6} should be at floor 0.85 for stability=0.0"
);
// β2 must be at structural floor for zero stability.
assert!(
(beta2 - 0.99_f32).abs() < 1e-5,
"group={g}: beta2={beta2:.7} should be at floor 0.99 for stability=0.0"
);
}
}