629ebd667c
Two same-seed runs now produce bit-equal eval_summary.json, alpha_rl_train_summary.json,
and diag.jsonl (modulo wall-clock elapsed_s). The 5-phase falsification chain landed:
Phase 2 PER tree-rebuild: __threadfence is NOT a grid-wide barrier; multiple blocks
raced across sum-tree levels. Fix: Grid=(1) Block=(1024) + __syncthreads
in rl_per_tree_rebuild.cu.
Phase 2.3 cuBLAS GEMM_DFALT + TF32 default-math allowed split-K non-deterministic
accumulation at 3 sites. New crates/ml-alpha/src/cublas_determinism.rs
applies CUBLAS_PEDANTIC_MATH via FOXHUNT_DETERMINISTIC env toggle
(0=TF32 prod, 1=PEDANTIC dev default, 2=DEFAULT_MATH control).
Phase 2.6 Two bugs surfaced sequentially in the backward kernel chain:
(1) rl_iqn_tau_cos_features had a multi-block r/w race on prng_state[batch]
— all N_TAU=32 blocks read seed; only tau_idx==0 wrote back; no
inter-block barrier. Fix: split into READ-ONLY rl_iqn_tau_cos_features
+ new sibling rl_iqn_advance_prng_state launched on same stream
(kernel-launch ordering = grid-wide barrier).
(2) OutcomeHead::new called near_zero_xavier without scoped_init_seed,
falling back to time+thread-id RNG. Stayed dormant until first done
event activated non-sentinel labels and divergent weights flowed via
grad_h_t_outcome into encoder gradient. Fix: add seed param + install
scoped_init_seed(dqn_seed.wrapping_add(0x0CE0)) guard.
Validation (./scripts/determinism-check.sh --quick, RTX 3050, b=128, 200+50 steps):
- All 200 rows of checksums.* leaves match (rel-tol 1e-5, abs-tol 1e-7)
- eval_summary.json, alpha_rl_train_summary.json byte-equal between runs
- diag.jsonl byte-equal modulo elapsed_s
- Eval pnl identical run-A vs run-B at seed 42
Pre-fix baseline (Phase 2.5 measurement): same-seed eval pnl spread $450k
($187k vs -$261k). Post-fix: $0 spread.
Speed cost: ~1.5ms/step amortised; ~10-15% slower than TF32 production
(PEDANTIC tax — acceptable in dev, toggle to FOXHUNT_DETERMINISTIC=0 for prod).
Mapped-pinned discipline: all 11 NEW memcpy_dtoh sites in diagnostic dump methods
+ per-step checksum readback use a new pub(crate) helper
read_slice_d_into<T: Copy>(stream, src, dst) — MappedRecordBuffer + raw
memcpy_dtod_async + raw_stream_sync + volatile read. Generic over T (f32, f64,
i32, u32, u8). Satisfies feedback_no_htod_htoh_only_mapped_pinned + hook guard.
Bundled Tier 1.5 fast-dev-cycle infrastructure (spec
docs/superpowers/specs/2026-06-02-fast-dev-cycle.md):
- scripts/local-mid-smoke.sh b=128, 2000+500, ~10min on RTX 3050
- scripts/determinism-check.sh runs mid-smoke twice, diffs checksums
- scripts/tier1_5_verdict.py behavioral kill verdict
- AdamW checkpoint save/load (crates/ml-alpha/src/trainer/optim.rs)
- IntegratedTrainer checkpoint save/load (resume from checkpoint)
- 15 Phase 1 checksum leaves in build_diag_value
- Env-gated dump methods (FOXHUNT_DETERMINISM_DEBUG_PER/MAMBA2/RL/BACKWARD)
for future divergence-chasing — never run in production
Documentation:
- docs/superpowers/specs/2026-06-02-determinism-foundation.md
- docs/superpowers/specs/2026-06-02-fast-dev-cycle.md
- docs/superpowers/plans/2026-06-02-determinism-foundation-implementation.md
- docs/superpowers/notes/2026-06-02-determinism-phase{1,2,2.2,2.5,2.6}-*.md
- Adjacent specs/plans/notes from the analytical chain that surfaced determinism
as the load-bearing blocker (eval-summary, eval-boundary, regime-observer,
multi-head policy, regime-invariance, Phase 3 IQN-complement post-mortem)
Unlocks: every controller / architecture / reward-shaping A/B from this commit
onward attributes outcome differences to the change, not random-init kernel-race
drift cascading through training x eval LOB-sim trajectories. The eval-collapse
investigation (pearl_reward_signal_anti_aligned_with_pnl, multi-head spec,
regime-invariance spec) is now testable with trustworthy verdicts.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
78 lines
3.6 KiB
Rust
78 lines
3.6 KiB
Rust
//! cuBLAS math-mode helper for the determinism foundation (spec §2.B + §4).
|
|
//!
|
|
//! Per `docs/superpowers/notes/2026-06-02-determinism-phase2.2-mamba2-investigation.md`:
|
|
//! the cuBLAS GEMM default algorithm (`CUBLAS_GEMM_DFALT`) permits split-K
|
|
//! accumulation with non-deterministic ordering on Ampere+ when TF32 mode
|
|
//! is on. The fix is to call `cublasSetMathMode(handle, CUBLAS_PEDANTIC_MATH)`
|
|
//! at every handle construction site — that filters out the non-deterministic
|
|
//! algorithm variants.
|
|
//!
|
|
//! Spec §4 dev/prod toggle: `FOXHUNT_DETERMINISTIC` env var (default "1" in
|
|
//! dev, where verdict trust matters; "0" in production, where the 10-15%
|
|
//! TF32 speed gain matters more than reproducibility).
|
|
//!
|
|
//! Call this immediately after `CudaBlas::new(...)` at every handle site
|
|
//! in `crates/ml-alpha/src/`. The 3 known sites (2026-06-02) are:
|
|
//! - `mamba2_block.rs` (Mamba2Block::new, line ~535)
|
|
//! - `rl/dqn.rs` (DqnHead::new, line ~364)
|
|
//! - `rl/iqn.rs` (IqnHead::new, line ~261)
|
|
//!
|
|
//! Per `feedback_isv_for_adaptive_bounds.md`: env-var read is one-shot at
|
|
//! handle construction; the mode is fixed for the lifetime of the handle.
|
|
//! No per-step ISV adjustment is needed (or possible, since the mode is
|
|
//! handle-level state).
|
|
|
|
use anyhow::{anyhow, Result};
|
|
use cudarc::cublas::CudaBlas;
|
|
|
|
/// Apply the dev/prod-toggleable cuBLAS math mode to a freshly-constructed
|
|
/// `CudaBlas` handle.
|
|
///
|
|
/// `FOXHUNT_DETERMINISTIC=1` (default) → `CUBLAS_PEDANTIC_MATH`. Deterministic
|
|
/// across runs; ~10-15% slower than TF32 on Ampere+ GEMMs.
|
|
///
|
|
/// `FOXHUNT_DETERMINISTIC=0` → `CUBLAS_TF32_TENSOR_OP_MATH`. Faster, but
|
|
/// `CUBLAS_GEMM_DFALT` may pick non-deterministic split-K variants — same-seed
|
|
/// runs can diverge by ~1e-6 after a handful of GEMMs, cascading via the
|
|
/// optimizer into ~10% trajectory drift over 200 steps.
|
|
///
|
|
/// Returns the input handle wrapped in `Ok` on success (passthrough for
|
|
/// ergonomic call-site chaining), or `Err` if the cuBLAS call fails.
|
|
pub fn apply_deterministic_math_mode(cublas: CudaBlas) -> Result<CudaBlas> {
|
|
// Per `pearl_build_rs_rerun_if_env_changed.md`: this env-var read is a
|
|
// runtime decision (not a build-time one), so no `cargo:rerun-if-env-changed`
|
|
// is needed. The env var is checked once per handle construction.
|
|
//
|
|
// Phase 2.4 control-mode addition (2026-06-02): added `2` as control
|
|
// mode (CUBLAS_DEFAULT_MATH — no TF32, no PEDANTIC, pure FP32) to
|
|
// disambiguate whether PEDANTIC fully eliminates split-K or just
|
|
// some variants. Per dispatch ask.
|
|
let env_value = std::env::var("FOXHUNT_DETERMINISTIC")
|
|
.unwrap_or_else(|_| "1".to_string());
|
|
|
|
let mode = match env_value.as_str() {
|
|
"0" => cudarc::cublas::sys::cublasMath_t::CUBLAS_TF32_TENSOR_OP_MATH,
|
|
"2" => cudarc::cublas::sys::cublasMath_t::CUBLAS_DEFAULT_MATH,
|
|
_ => cudarc::cublas::sys::cublasMath_t::CUBLAS_PEDANTIC_MATH,
|
|
};
|
|
|
|
unsafe {
|
|
cudarc::cublas::sys::cublasSetMathMode(*cublas.handle(), mode)
|
|
.result()
|
|
.map_err(|e| anyhow!("cublasSetMathMode {mode:?}: {e:?}"))?;
|
|
}
|
|
|
|
// Print once per handle so callers can verify the env var propagated.
|
|
// Phase 2.4: this is essential diagnostics — the Phase 2.3 outcome
|
|
// note claimed PEDANTIC was active but later determinism-check runs
|
|
// showed attn_q still diverging, so we need explicit confirmation.
|
|
static MODE_PRINTED: std::sync::OnceLock<()> = std::sync::OnceLock::new();
|
|
MODE_PRINTED.get_or_init(|| {
|
|
eprintln!(
|
|
"[cublas_determinism] FOXHUNT_DETERMINISTIC={env_value} → mode={mode:?}"
|
|
);
|
|
});
|
|
|
|
Ok(cublas)
|
|
}
|