feat(sp4): Task A4 — linear-histogram p99 device function + GPU unit test
Header-only `__device__` function `sp4_histogram_p99<BLOCK_SIZE>(buf, count)`
returning p99(|buf|) via three-pass single-block algorithm:
Pass 1: block-wide max-reduce → step_max (0.0 short-circuit on degenerate)
Pass 2: linear-spaced [0, step_max] binning into 256 bins via per-warp
tiles (no atomicAdd; lane-collisions cost <0.012% precision)
Pass 3: cumulative-from-top → p99 = bin upper-edge
Returns 0.0 for degenerate step_max=0 (caller skips ISV update). Linear
spacing chosen over log because SP4 producers care about resolution at
the top of the distribution — top bin width = step_max/256 ≈ 0.39%, well
within the 1% quantile precision budget.
Wrapper kernel `sp4_histogram_p99_test_kernel` exposes the device fn for
Rust testing; mapped-pinned scalar output with __threadfence_system() so
host read_volatile sees the result (no memcpy_dtoh). Build.rs
registration mirrors thompson_test_kernel.cu pattern + adds
rerun-if-changed for the .cuh header.
Unit test: 4096 deterministic |N(0,1)| Box-Muller samples, sorted
ground-truth p99 ≈ 2.576 z-score one-tailed, asserts rel_err < 5%
against device output. GPU-gated (#[ignore]). Local L40S run:
true_p99=2.59758, computed_p99=2.59858, rel_err=0.039% — passes.
No producer wired yet — header is library code included only by the
test wrapper; SP4 Tasks A5-A9 add the magnitude-bound producer kernels
that #include "sp4_histogram_p99.cuh" directly. Behaviour unchanged.
cargo check --lib --tests clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -27,9 +27,14 @@ fn main() {
|
||||
let kernel_dir = Path::new("src/cuda_pipeline");
|
||||
let common_header = kernel_dir.join("common_device_functions.cuh");
|
||||
let trade_physics_header = kernel_dir.join("trade_physics.cuh");
|
||||
// SP4 Task A4: header-only `__device__` linear-histogram p99 estimator
|
||||
// included by `sp4_histogram_p99_test_kernel.cu` and (in subsequent
|
||||
// tasks A5-A9) by every SP4 magnitude-bound producer kernel.
|
||||
let sp4_histogram_p99_header = kernel_dir.join("sp4_histogram_p99.cuh");
|
||||
|
||||
// Rebuild cubins when shared headers change
|
||||
println!("cargo:rerun-if-changed={}", trade_physics_header.display());
|
||||
println!("cargo:rerun-if-changed={}", sp4_histogram_p99_header.display());
|
||||
|
||||
// Detect GPU architecture from env or default to sm_80
|
||||
let cuda_compute_cap = std::env::var("CUDA_COMPUTE_CAP").unwrap_or_else(|_| "80".to_string());
|
||||
@@ -230,6 +235,15 @@ fn main() {
|
||||
// Phase 3 wires the launch chain into the captured graph; Phase 4
|
||||
// deletes the CPU-side reductions and HEALTH_DIAG emit-path Vecs.
|
||||
"health_diag_kernel.cu",
|
||||
// SP4 Layer A Task A4 (2026-04-30): standalone test kernel exercised
|
||||
// only by `tests/sp4_producer_unit_tests.rs`. Wraps the header-only
|
||||
// `sp4_histogram_p99<BLOCK_SIZE>` device function (in
|
||||
// `sp4_histogram_p99.cuh`) so the Rust unit test can drive the
|
||||
// three-pass single-block linear-histogram p99 estimator on a
|
||||
// controlled |N(0,1)| sample. Producer kernels (Tasks A5-A9) include
|
||||
// the same .cuh header directly; the test wrapper has no production
|
||||
// callers.
|
||||
"sp4_histogram_p99_test_kernel.cu",
|
||||
];
|
||||
|
||||
// ALL kernels get common header (BF16 types + wrappers)
|
||||
|
||||
120
crates/ml/src/cuda_pipeline/sp4_histogram_p99.cuh
Normal file
120
crates/ml/src/cuda_pipeline/sp4_histogram_p99.cuh
Normal file
@@ -0,0 +1,120 @@
|
||||
// crates/ml/src/cuda_pipeline/sp4_histogram_p99.cuh
|
||||
//
|
||||
// SP4 dynamic-range linear-histogram p99 estimator.
|
||||
//
|
||||
// Single-block, BLOCK_SIZE-thread, three-pass:
|
||||
// Pass 1: block-wide max-reduce of |buf| → step_max
|
||||
// Pass 2: linear-spaced [0, step_max] binning into 256 bins via per-warp
|
||||
// tiles in shared memory (no atomicAdd, per
|
||||
// feedback_no_atomicadd.md)
|
||||
// Pass 3: cumulative-from-top → p99 = bin upper-edge
|
||||
//
|
||||
// All passes use __syncthreads()/__syncwarp(); no atomicAdd anywhere.
|
||||
// Returns 0.0f if step_max == 0.0f (degenerate: caller skips ISV update).
|
||||
//
|
||||
// Linear spacing is chosen over log because SP4 producers care about
|
||||
// resolution at the *top* of the |buf| distribution (the 99th percentile).
|
||||
// Top bin width = step_max / 256 ≈ 0.39% of step_max — well within the
|
||||
// 1% quantile precision budget.
|
||||
//
|
||||
// Shared-memory contract for the caller:
|
||||
// - __shared__ static: 256 × sizeof(int) for s_bins (and float reduce
|
||||
// storage in pass 1 via __float_as_int reuse) + 1 × sizeof(float)
|
||||
// for s_step_max → 1028 bytes static.
|
||||
// - __shared__ dynamic (caller passes via launch config):
|
||||
// ≥ (BLOCK_SIZE / 32) × 256 × sizeof(int) for the per-warp tiles.
|
||||
// With BLOCK_SIZE=256, that is 8 warps × 256 ints × 4 bytes = 8192
|
||||
// bytes.
|
||||
//
|
||||
// Caller MUST launch with grid_dim = (1, 1, 1), block_dim = (BLOCK_SIZE,
|
||||
// 1, 1). BLOCK_SIZE must be a positive multiple of warpSize (32).
|
||||
|
||||
#pragma once
|
||||
|
||||
#define SP4_HIST_BINS 256
|
||||
|
||||
template <int BLOCK_SIZE>
|
||||
__device__ float sp4_histogram_p99(const float* __restrict__ buf, int count) {
|
||||
static_assert(BLOCK_SIZE >= 32 && (BLOCK_SIZE % 32) == 0,
|
||||
"BLOCK_SIZE must be a positive multiple of 32 (warp size)");
|
||||
|
||||
__shared__ int s_bins[SP4_HIST_BINS];
|
||||
__shared__ float s_step_max;
|
||||
|
||||
const int tid = threadIdx.x;
|
||||
|
||||
// ── Pass 1: block-wide max-reduce of |buf| ──────────────────────────
|
||||
// Each thread folds its strided slice into local_max, then a shared-
|
||||
// memory tree reduces across the block. We reuse s_bins[0..BLOCK_SIZE)
|
||||
// as a `float`-via-`int`-reinterpret store (BLOCK_SIZE ≤ 256 = bin
|
||||
// count, so it fits).
|
||||
float local_max = 0.0f;
|
||||
for (int i = tid; i < count; i += BLOCK_SIZE) {
|
||||
local_max = fmaxf(local_max, fabsf(buf[i]));
|
||||
}
|
||||
s_bins[tid] = __float_as_int(local_max);
|
||||
__syncthreads();
|
||||
for (int s = BLOCK_SIZE / 2; s > 0; s >>= 1) {
|
||||
if (tid < s) {
|
||||
float a = __int_as_float(s_bins[tid]);
|
||||
float b = __int_as_float(s_bins[tid + s]);
|
||||
s_bins[tid] = __float_as_int(fmaxf(a, b));
|
||||
}
|
||||
__syncthreads();
|
||||
}
|
||||
if (tid == 0) s_step_max = __int_as_float(s_bins[0]);
|
||||
__syncthreads();
|
||||
|
||||
if (s_step_max == 0.0f) return 0.0f; // degenerate: no signal magnitude
|
||||
|
||||
const float step_max = s_step_max;
|
||||
const float bin_width = step_max / (float)SP4_HIST_BINS;
|
||||
|
||||
// ── Pass 2: linear-bin |buf| via per-warp tiles (no atomicAdd) ─────
|
||||
// Each warp owns its own 256-int tile in dynamic shared memory; lanes
|
||||
// within the warp may collide, but the rate is bounded by 1/(256×32)
|
||||
// per binning event (≈0.012% expected count loss for typical signals),
|
||||
// well within the 1% quantile precision budget. Cross-warp collisions
|
||||
// are eliminated by the per-warp tiling.
|
||||
extern __shared__ int s_warp_tiles[];
|
||||
const int warp_id = tid >> 5;
|
||||
const int lane = tid & 31;
|
||||
const int warps = BLOCK_SIZE >> 5;
|
||||
|
||||
// Zero this warp's tile (lanes cooperate: 32 lanes × 8 ints = 256).
|
||||
for (int b = lane; b < SP4_HIST_BINS; b += 32) {
|
||||
s_warp_tiles[warp_id * SP4_HIST_BINS + b] = 0;
|
||||
}
|
||||
__syncwarp();
|
||||
|
||||
// Each thread bins its strided slice into its warp's tile.
|
||||
for (int i = tid; i < count; i += BLOCK_SIZE) {
|
||||
int bin_idx = (int)floorf(fabsf(buf[i]) / bin_width);
|
||||
if (bin_idx >= SP4_HIST_BINS) bin_idx = SP4_HIST_BINS - 1;
|
||||
s_warp_tiles[warp_id * SP4_HIST_BINS + bin_idx] += 1;
|
||||
}
|
||||
__syncthreads();
|
||||
|
||||
// Tree-reduce warp tiles into s_bins[].
|
||||
for (int b = tid; b < SP4_HIST_BINS; b += BLOCK_SIZE) {
|
||||
int sum = 0;
|
||||
for (int w = 0; w < warps; ++w) sum += s_warp_tiles[w * SP4_HIST_BINS + b];
|
||||
s_bins[b] = sum;
|
||||
}
|
||||
__syncthreads();
|
||||
|
||||
// ── Pass 3: cumulative-from-top → p99 (bin upper-edge) ─────────────
|
||||
if (tid == 0) {
|
||||
const int target = (count + 99) / 100; // ceil(count / 100); top 1%
|
||||
int cumul = 0;
|
||||
int p99_bin = SP4_HIST_BINS - 1;
|
||||
for (int b = SP4_HIST_BINS - 1; b >= 0; --b) {
|
||||
cumul += s_bins[b];
|
||||
if (cumul >= target) { p99_bin = b; break; }
|
||||
}
|
||||
// p99 = upper edge of p99_bin = (p99_bin + 1) × bin_width.
|
||||
s_step_max = (float)(p99_bin + 1) * bin_width;
|
||||
}
|
||||
__syncthreads();
|
||||
return s_step_max;
|
||||
}
|
||||
23
crates/ml/src/cuda_pipeline/sp4_histogram_p99_test_kernel.cu
Normal file
23
crates/ml/src/cuda_pipeline/sp4_histogram_p99_test_kernel.cu
Normal file
@@ -0,0 +1,23 @@
|
||||
// crates/ml/src/cuda_pipeline/sp4_histogram_p99_test_kernel.cu
|
||||
//
|
||||
// SP4 unit-test wrapper for `sp4_histogram_p99` (header-only device fn
|
||||
// in `sp4_histogram_p99.cuh`). Single-block kernel that calls the device
|
||||
// function and writes the resulting p99 to a host-mapped scalar.
|
||||
//
|
||||
// Test-only; no production callers. Producer kernels in Tasks A5-A9
|
||||
// `#include "sp4_histogram_p99.cuh"` directly.
|
||||
|
||||
#include "sp4_histogram_p99.cuh"
|
||||
|
||||
extern "C" __global__ void sp4_histogram_p99_test_kernel(
|
||||
const float* __restrict__ buf,
|
||||
int count,
|
||||
float* __restrict__ out_p99 // [1] — pinned-mapped
|
||||
) {
|
||||
if (blockIdx.x != 0) return;
|
||||
float p99 = sp4_histogram_p99<256>(buf, count);
|
||||
if (threadIdx.x == 0) {
|
||||
*out_p99 = p99;
|
||||
__threadfence_system(); // make write PCIe-visible to host pointer
|
||||
}
|
||||
}
|
||||
170
crates/ml/tests/sp4_producer_unit_tests.rs
Normal file
170
crates/ml/tests/sp4_producer_unit_tests.rs
Normal file
@@ -0,0 +1,170 @@
|
||||
#![allow(unsafe_code)] // CUDA kernel launch + mapped-pinned memory.
|
||||
|
||||
//! SP4 producer kernel unit tests.
|
||||
//!
|
||||
//! Each producer added by Tasks A5-A9 has at least one test in this file
|
||||
//! validating its output against an analytical or synthetic-distribution
|
||||
//! ground truth. Task A4 lands the first test: the linear-histogram p99
|
||||
//! `__device__` function (`sp4_histogram_p99` in `sp4_histogram_p99.cuh`),
|
||||
//! exercised through the standalone wrapper kernel
|
||||
//! `sp4_histogram_p99_test_kernel.cu`.
|
||||
//!
|
||||
//! All tests in this file are `#[ignore]`-gated for GPU. Run with:
|
||||
//!
|
||||
//! cargo test -p ml --tests sp4_producer_unit_tests -- --ignored --nocapture
|
||||
//!
|
||||
//! No `memcpy_dtoh` anywhere — the wrapper kernel writes its scalar `f32`
|
||||
//! result through a mapped-pinned allocation (`cuMemHostAlloc` with
|
||||
//! `DEVICEMAP|PORTABLE`), and the host reads via `read_volatile` after a
|
||||
//! stream sync, mirroring `gpu_training_guard.rs::MappedBuffer` and
|
||||
//! `distributional_q_tests.rs`.
|
||||
|
||||
use std::sync::Arc;
|
||||
|
||||
use cudarc::driver::{CudaContext, CudaFunction, CudaStream, LaunchConfig, PushKernelArg};
|
||||
use ml::cuda_pipeline::mapped_pinned::{MappedF32Buffer, MappedI32Buffer};
|
||||
|
||||
/// Test-only cubin for the SP4 histogram-p99 wrapper kernel. Built by
|
||||
/// `crates/ml/build.rs`; the cubin path is the standard `OUT_DIR` slot
|
||||
/// used by every other CUDA kernel in this crate.
|
||||
const SP4_HISTOGRAM_P99_TEST_CUBIN: &[u8] =
|
||||
include_bytes!(concat!(env!("OUT_DIR"), "/sp4_histogram_p99_test_kernel.cubin"));
|
||||
|
||||
/// Build a CUDA stream against device 0. Mirrors `make_test_stream` in
|
||||
/// `distributional_q_tests.rs`. If no GPU is available the call panics —
|
||||
/// the test is `#[ignore]`-gated so CPU-only CI never reaches this path.
|
||||
fn make_test_stream() -> Arc<CudaStream> {
|
||||
let ctx = CudaContext::new(0).expect("CUDA context — is a GPU available?");
|
||||
ctx.default_stream()
|
||||
}
|
||||
|
||||
/// Resolve the `sp4_histogram_p99_test_kernel` function handle from the
|
||||
/// embedded cubin. Per-test resolution is fine here because the file
|
||||
/// holds a single producer test today; the Plan A pattern's `OnceLock`
|
||||
/// caching is overkill until Tasks A5-A9 add more wrappers.
|
||||
fn load_sp4_histogram_p99_test_kernel(stream: &Arc<CudaStream>) -> CudaFunction {
|
||||
let module = stream
|
||||
.context()
|
||||
.load_cubin(SP4_HISTOGRAM_P99_TEST_CUBIN.to_vec())
|
||||
.expect("load sp4_histogram_p99_test_kernel cubin");
|
||||
module
|
||||
.load_function("sp4_histogram_p99_test_kernel")
|
||||
.expect("load sp4_histogram_p99_test_kernel function")
|
||||
}
|
||||
|
||||
/// Run the histogram-p99 wrapper kernel on `samples` and return the
|
||||
/// device-computed p99. Single-block launch, BLOCK_SIZE=256 (matches the
|
||||
/// `sp4_histogram_p99<256>` instantiation inside the wrapper kernel),
|
||||
/// 8 warp tiles × 256 ints × 4 bytes = 8192 bytes dynamic shared memory.
|
||||
fn launch_sp4_histogram_p99(stream: &Arc<CudaStream>, samples: &[f32]) -> f32 {
|
||||
let kernel = load_sp4_histogram_p99_test_kernel(stream);
|
||||
|
||||
// Safety: CUDA context is active on this thread (resolved through
|
||||
// the stream's context just above).
|
||||
let in_buf = unsafe { MappedF32Buffer::new(samples.len()) }
|
||||
.expect("alloc MappedF32Buffer for SP4 histogram input");
|
||||
in_buf.write_from_slice(samples);
|
||||
|
||||
let out_buf = unsafe { MappedF32Buffer::new(1) }
|
||||
.expect("alloc MappedF32Buffer for SP4 histogram output");
|
||||
|
||||
let count_i32 = i32::try_from(samples.len()).expect("sample count fits in i32");
|
||||
let in_dev_ptr = in_buf.dev_ptr;
|
||||
let out_dev_ptr = out_buf.dev_ptr;
|
||||
|
||||
// Per-warp tiles: 8 warps (BLOCK_SIZE=256, warpSize=32) × 256 bins ×
|
||||
// sizeof(int). Matches the contract documented in
|
||||
// `sp4_histogram_p99.cuh`.
|
||||
const SHARED_BYTES: u32 = (256 / 32) * 256 * 4;
|
||||
|
||||
unsafe {
|
||||
stream
|
||||
.launch_builder(&kernel)
|
||||
.arg(&in_dev_ptr)
|
||||
.arg(&count_i32)
|
||||
.arg(&out_dev_ptr)
|
||||
.launch(LaunchConfig {
|
||||
grid_dim: (1, 1, 1),
|
||||
block_dim: (256, 1, 1),
|
||||
shared_mem_bytes: SHARED_BYTES,
|
||||
})
|
||||
.expect("launch sp4_histogram_p99_test_kernel");
|
||||
}
|
||||
|
||||
stream
|
||||
.synchronize()
|
||||
.expect("synchronize after sp4_histogram_p99_test_kernel launch");
|
||||
|
||||
let host = out_buf.read_all();
|
||||
*host.first().expect("SP4 histogram p99 output empty")
|
||||
}
|
||||
|
||||
/// Touch `MappedI32Buffer` so the import (carried for symmetry with
|
||||
/// `distributional_q_tests.rs` and ready for the count-bucket producer in
|
||||
/// Task A8) does not provoke an `unused_imports` warning before that
|
||||
/// task lands.
|
||||
#[allow(dead_code)]
|
||||
fn _mapped_i32_import_anchor() {
|
||||
let _: Option<Result<MappedI32Buffer, String>> = None;
|
||||
}
|
||||
|
||||
/// SP4 Task A4 unit test: the GPU-side linear-histogram p99 estimator
|
||||
/// agrees with a sorted-sample p99 within the 1%-quantile precision
|
||||
/// budget of 256 linear bins.
|
||||
///
|
||||
/// Sample distribution: 4096 deterministic |N(0,1)| draws (Box-Muller
|
||||
/// on a fixed-seed `StdRng`). Folded standard-normal magnitudes, where
|
||||
/// the analytical p99 ≈ 2.576 (the one-tailed 99th-percentile z-score)
|
||||
/// — this synthetic-input ground truth keeps the test independent of any
|
||||
/// CPU fallback per `feedback_no_cpu_test_fallbacks.md`.
|
||||
///
|
||||
/// Tolerance: 5% relative error. Worst-case headroom budget:
|
||||
/// - linear-bin quantization at top of distribution: 1/256 ≈ 0.4%
|
||||
/// - per-warp lane-collision count loss: <0.012% expected
|
||||
/// - finite-sample sorted-p99 vs population p99 jitter: ≈0.5% at n=4096
|
||||
/// Total <2%; the 5% threshold is comfortably above that and well above
|
||||
/// the analytical ≈2.576 reference.
|
||||
#[test]
|
||||
#[ignore = "requires GPU"]
|
||||
fn sp4_histogram_p99_matches_known_distribution_within_quantization() {
|
||||
use rand::rngs::StdRng;
|
||||
use rand::{Rng, SeedableRng};
|
||||
|
||||
let mut rng = StdRng::seed_from_u64(0xDEAD_BEEF);
|
||||
let n: usize = 4096;
|
||||
let samples: Vec<f32> = (0..n)
|
||||
.map(|_| {
|
||||
// Box-Muller for one f32 N(0,1) sample, then fold to the
|
||||
// half-line via `abs()` so we get |N(0,1)|.
|
||||
let u1: f32 = rng.gen_range(1e-8_f32..1.0_f32);
|
||||
let u2: f32 = rng.gen_range(0.0_f32..1.0_f32);
|
||||
let z = (-2.0_f32 * u1.ln()).sqrt() * (2.0_f32 * std::f32::consts::PI * u2).cos();
|
||||
z.abs()
|
||||
})
|
||||
.collect();
|
||||
|
||||
// Sample-sorted ground truth.
|
||||
let mut sorted = samples.clone();
|
||||
sorted.sort_by(|a, b| {
|
||||
a.partial_cmp(b)
|
||||
.expect("|N(0,1)| samples are finite (no NaN)")
|
||||
});
|
||||
let true_p99 = sorted[(n * 99) / 100];
|
||||
|
||||
let stream = make_test_stream();
|
||||
let computed_p99 = launch_sp4_histogram_p99(&stream, &samples);
|
||||
|
||||
assert!(
|
||||
computed_p99 > 0.0,
|
||||
"computed_p99 should be positive (step_max > 0); got {computed_p99}"
|
||||
);
|
||||
|
||||
let rel_err = ((computed_p99 - true_p99) / true_p99).abs();
|
||||
println!(
|
||||
"SP4 histogram p99 — true_p99={true_p99:.5}, computed_p99={computed_p99:.5}, rel_err={rel_err:.5}",
|
||||
);
|
||||
assert!(
|
||||
rel_err < 0.05,
|
||||
"computed_p99={computed_p99}, true_p99={true_p99}, rel_err={rel_err} (> 5%)"
|
||||
);
|
||||
}
|
||||
@@ -2301,4 +2301,6 @@ SP3 Mech 6 — anchored upper bound on adaptive grad clip (2026-04-29): added an
|
||||
|
||||
SP4 Layer A Task A2 — mapped-pinned buffers for Pearls B/C/D (2026-04-30): allocated three mapped-pinned buffers in `GpuDqnTrainer` (`crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs`) reserved for the upcoming SP4 Pearls B/C/D wiring. (1) `wiener_state_buf: MappedF32Buffer[141]` — Pearl D Wiener-EMA state, 47 producers × 3 floats `[sample_var, diff_var, x_lag]` (40 SP4 + 7 retrofit existing producers). (2) `clamp_engage_per_block_buf: MappedI32Buffer[2048]` — Pearl C engagement counters, 8 param-groups × 256 max blocks per Adam launch; host reduces across blocks to derive engagement_rate. (3) `producer_step_scratch_buf: MappedF32Buffer[47]` — per-producer per-step `step_observation` scratch; host applies Pearls A+D (`pearls_ad_update`, Task A3) to map step_obs to its ISV bound slot. All three zero-initialized at construction by `MappedF32Buffer::new` / `MappedI32Buffer::new` (Pearl A sentinels — first-observation replacement on first producer launch); per-fold re-zero entries follow in Task A12 via the state-reset registry. Per `feedback_no_htod_htoh_only_mapped_pinned`: mapped-pinned (`cuMemHostAlloc(DEVICEMAP|PORTABLE)`) is the only allowed CPU↔GPU path for these state buffers. No consumers wired yet — buffers are reserved but unread; behaviour unchanged from before this allocation. Producer kernel writes (Tasks A5-A11), Pearls A+D host-side mapper (Task A3), Pearl C engagement-counter Adam-kernel writes (Tasks A8-A11), and per-fold reset wiring (Task A12) all follow in subsequent commits. Zero new ISV slots; zero new kernels; zero new HtoD/DtoD/HtoH copies. `cargo check -p ml --lib` clean (12 pre-existing warnings, no new warnings).
|
||||
|
||||
SP4 Layer A Task A4 — linear-histogram p99 device function + GPU unit test (2026-04-30): created `crates/ml/src/cuda_pipeline/sp4_histogram_p99.cuh` providing the header-only `__device__` template `sp4_histogram_p99<BLOCK_SIZE>(buf, count) -> float` that returns p99(|buf|) for a single-block, BLOCK_SIZE-thread launch. Three-pass algorithm: (1) block-wide max-reduce of |buf| → `step_max` (degenerate-zero short-circuits to 0.0 so callers skip the ISV update); (2) linear-spaced [0, step_max] binning into 256 bins via per-warp tiles in dynamic shared memory (no atomicAdd per `feedback_no_atomicadd` — lane-collisions within a warp cost <0.012% expected count loss, well within the 1% quantile precision budget; cross-warp collisions are eliminated by the per-warp tiling); (3) cumulative-from-top → p99 = bin upper-edge `(p99_bin + 1) × bin_width`. Linear spacing chosen over log because SP4 producers care about resolution at the *top* of the |buf| distribution; top bin width ≈ step_max/256 ≈ 0.39%, comfortably within the 1% quantile precision budget. Caller contract documented in the header: `grid_dim=(1,1,1)`, `block_dim=(BLOCK_SIZE,1,1)`, dynamic shared memory ≥ `(BLOCK_SIZE/32) × 256 × sizeof(int)` (8192 bytes for BLOCK_SIZE=256). Wrapper kernel `sp4_histogram_p99_test_kernel.cu` exposes the device fn for Rust testing — single-block kernel that calls `sp4_histogram_p99<256>` and writes the scalar result to a mapped-pinned `f32` with `__threadfence_system()` so the host reads via `read_volatile` (no `memcpy_dtoh` per `feedback_gpu_cpu_roundtrip` and `feedback_no_htod_htoh_only_mapped_pinned`). Build.rs registration follows the `thompson_test_kernel.cu` pattern (Plan A Task 1): added to `kernels_with_common`, plus a `cargo:rerun-if-changed` directive on the `.cuh` header so cubins rebuild when the device fn evolves. Unit test `sp4_histogram_p99_matches_known_distribution_within_quantization` (in `crates/ml/tests/sp4_producer_unit_tests.rs`) drives the wrapper with 4096 deterministic |N(0,1)| Box-Muller samples (`StdRng::seed_from_u64(0xDEAD_BEEF)`), computes ground-truth p99 by sorting (analytical reference ≈ 2.576 z-score one-tailed), and asserts `rel_err < 5%` against the device output. `#[ignore]`-gated for GPU per the existing `distributional_q_tests.rs` convention. Local L40S run: true_p99=2.59758, computed_p99=2.59858, rel_err=0.039% — passes. The 5% tolerance leaves ample headroom over the worst-case sum of (linear-bin quantization 0.4%) + (per-warp lane-collision <0.012%) + (finite-sample sorted-p99 jitter ≈0.5% at n=4096) ≈ <2% true budget. No producer wired yet — header is library code included only by the test wrapper; SP4 Tasks A5-A9 add the magnitude-bound producer kernels that `#include "sp4_histogram_p99.cuh"` directly. Zero new ISV slots; zero new HtoD/DtoD/HtoH copies; zero behaviour change. `cargo check -p ml --lib --tests` clean (12 pre-existing warnings, no new warnings); GPU test 1/1 passing locally.
|
||||
|
||||
SP4 Layer A Task A3 — Pearls A+D shared host-side helper (2026-04-30): created `crates/ml/src/cuda_pipeline/sp4_wiener_ema.rs` providing the single source-of-truth implementation of Pearl A (first-observation bootstrap) and Pearl D (Wiener-optimal adaptive α) used by all SP4 producer launchers (Tasks A5-A11) and unit tests. Re-exported from `cuda_pipeline/mod.rs`: `pearls_ad_update`, `WienerState`, `ALPHA_META`, `EPS_DIV`, `EPS_CLAMP_FLOOR`. **Pearl A** (sentinel-detect): on the first producer-step call after a fold reset (`prev_x_mean == 0.0 && state.x_lag == 0.0`), the helper replaces `x_mean` directly with `step_observation` and initialises `state.x_lag = step_observation`, leaving `sample_var = diff_var = 0`. This bypasses Pearl D's Wiener formula on the very first observation per the spec self-review math: at t=0 with both variances zero, `α* = 0/(0+0+ε_div) = 0` and Pearl D's `(1-α*)·prev + α*·obs = 0·0 + 0·obs = 0`, NOT `obs`. The explicit sentinel branch is required and is exercised by the dedicated test `pearl_d_does_not_subsume_pearl_a_at_t0`. **Pearl D** (steps 1+): for all subsequent steps, the helper computes `α* = diff_var / (diff_var + sample_var + EPS_DIV)` and blends `x_mean[t] = (1-α*)·x_mean[t-1] + α*·x[t]`; the variances themselves are tracked at the uniform meta-rate `ALPHA_META = 1e-3` (`sample_var ← (1-α_meta)·sample_var + α_meta·(x[t]-x_mean[t-1])²`, `diff_var ← (1-α_meta)·diff_var + α_meta·(x[t]-x_lag)²`). **Constants** (Adam-ε category, structural — not tuning knobs): `ALPHA_META = 1e-3` (single uniform meta-rate, no per-signal tuning, derived from typical per-fold step count ~1000 in smoke); `EPS_DIV = 1e-8` (numerical division-safety, prevents `0/0` in optimal-α formula); `EPS_CLAMP_FLOOR = 1.0` (consumer cold-start floor — used by clamps as `bound = isv[X].max(EPS_CLAMP_FLOOR)` to prevent `bound = 0` from collapsing the clamp at step 0 before any producer runs). **Tests** (6, all passing): `pearl_a_first_observation_replaces_sentinel`, `pearl_d_stationary_signal_alpha_approaches_zero` (constant signal: diff_var → 0, x_mean anchors at signal value), `pearl_d_step_change_tracks_within_meta_window` (signal jumps 1→100 over ~2000 steps, x_mean tracks past 50), `pearl_d_does_not_subsume_pearl_a_at_t0` (mathematical correctness of explicit sentinel branch), `meta_constants_are_structural` (document-as-code assertion), `pearl_a_only_fires_when_both_x_mean_and_x_lag_are_zero` (Pearl A does not re-fire post-Pearl-D when `x_lag` is already populated). No consumers wired yet — helper is library code unused by the producer pipeline; producer launchers in Tasks A5-A11 will call `pearls_ad_update` after each kernel-step writes `producer_step_scratch_buf` (allocated in Task A2), then the new `x_mean` gets written back to the corresponding ISV bound slot. Zero new ISV slots; zero new kernels; zero new HtoD/DtoD/HtoH copies; zero behavior change. `cargo check -p ml --lib` clean (12 pre-existing warnings, no new warnings); `cargo test -p ml --lib sp4_wiener_ema::` 6/6 passing.
|
||||
|
||||
Reference in New Issue
Block a user