gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(sp20): c51_loss launcher aux_conf arg + Phase 5 gate tests

Browse Source 
Threads `self.aux_conf_at_state_buf` into the `c51_loss_batched` launch
in `GpuDqnTrainer::launch_c51_loss`. Position matches the kernel's
appended trailing arg from the previous commit.

Tests added in `crates/ml-dqn/src/gpu_replay_buffer.rs::tests`:

  - `aux_gate_high_confidence_passes_full_target` (CPU pure-math):
    gate(aux_conf=0.5, threshold=0.10, temp=0.05) > 0.99 proves
    high-confidence reward pass-through.
  - `aux_gate_low_confidence_attenuates_reward` (CPU pure-math):
    gate(aux_conf=0.02, threshold=0.10, temp=0.05) < 0.20 proves
    the uncertain-state neutralizer semantic.
  - `aux_gate_temp_floor_keeps_gate_finite` (CPU pure-math):
    sweeps {temp, aux_conf, threshold} and asserts finite gate ∈ [0,1]
    across the ISV-controllable parameter range — proves the
    fmaxf(temp, 1e-3) floor keeps the kernel numerically safe.
  - `aux_conf_direct_to_trainer_gather_populates_destination` (GPU
    behavioral): wires a fresh CudaSlice<f32> as the trainer
    destination, inserts 8 transitions with strictly-positive distinct
    aux_conf values, samples 1, asserts the trainer destination
    buffer post-sample holds a value from the inserted set (NOT the
    alloc_zeros sentinel) — proves the direct-gather wiring actually
    populates the trainer buffer with non-trivial data.

All 3 CPU math tests + 1 GPU integration test pass on RTX 3050.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-10 14:50:11 +02:00
parent 96b76d9298
commit ab4a7db33c
3 changed files with 240 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

184
crates/ml-dqn/src/gpu_replay_buffer.rs
View File

@@ -1871,4 +1871,188 @@ mod tests {
assert_eq!(buf.trainer_aux_conf_ptr(), 0,
"trainer_aux_conf_ptr should re-zero on set(0)");
}
/// SP20 Phase 5 (2026-05-10) — aux→Q reward-gate sigmoid formula
/// "high-confidence" pure-math test.
///
/// Mirrors the in-kernel computation:
/// `gate = sigmoid((aux_conf - threshold) / max(temp, 1e-3))`
///
/// At aux_conf=0.5 (max-confidence K=2 peak), threshold=0.10
/// (typical mid-range value from `clamp(aux_dir_acc - 0.50, 0.01,
/// 0.20)`), and temp=0.05, the gate must exceed 0.99 — i.e., the
/// reward passes through essentially unchanged. This proves that
/// when the aux head is highly confident, the reward gate has
/// near-identity effect on the Bellman target, so the model
/// trains on the full reward signal.
///
/// Per `pearl_tests_must_prove_not_lock_observations`: the
/// assertion is the structural "high confidence ⇒ near-identity"
/// invariant, not a locked numerical value.
#[test]
fn aux_gate_high_confidence_passes_full_target() {
let aux_conf: f32 = 0.5;
let threshold: f32 = 0.10;
let temp: f32 = 0.05_f32.max(1e-3);
let gate = 1.0 / (1.0 + ((-((aux_conf - threshold) / temp)) as f32).exp());
assert!(gate > 0.99,
"Phase 5 gate must pass full reward at high aux_conf: \
gate({aux_conf}, threshold={threshold}, temp={temp}) = {gate}, \
expected > 0.99. The Bellman target should receive ~unaltered \
reward at maximum aux confidence.");
}
/// SP20 Phase 5 (2026-05-10) — aux→Q reward-gate sigmoid formula
/// "low-confidence" pure-math test.
///
/// At aux_conf=0.02 (effectively no signal — well below the lower
/// threshold clamp of 0.01), threshold=0.10, and temp=0.05, the
/// gate must drop below 0.20 — i.e., > 80% of the reward is
/// attenuated. This proves the gate's "uncertain-state
/// neutralizer" semantic: when aux can't predict the next bar,
/// the Bellman target collapses toward `gamma * Q(s', a')` (no
/// reward feedback at this state).
#[test]
fn aux_gate_low_confidence_attenuates_reward() {
let aux_conf: f32 = 0.02;
let threshold: f32 = 0.10;
let temp: f32 = 0.05_f32.max(1e-3);
let gate = 1.0 / (1.0 + ((-((aux_conf - threshold) / temp)) as f32).exp());
assert!(gate < 0.20,
"Phase 5 gate must attenuate reward at low aux_conf: \
gate({aux_conf}, threshold={threshold}, temp={temp}) = {gate}, \
expected < 0.20. At low aux confidence the Bellman target \
must collapse toward gamma * Q(s', a') (uncertain-state \
neutralizer per Phase 3 Task 3.4 audit doc spec §4.4).");
}
/// SP20 Phase 5 (2026-05-10) — aux→Q reward-gate temperature
/// floor invariant.
///
/// The kernel uses `fmaxf(isv_signals[519], 1e-3f)` to floor temp,
/// preventing division-by-zero / numerical explosion when the
/// aux_conf_std EMA collapses to zero (e.g., constant-prediction
/// regime). With temp floored at 1e-3 and any finite aux_conf in
/// [0, 0.5], the gate produces a finite value in [0, 1] —
/// numerical safety contract.
#[test]
fn aux_gate_temp_floor_keeps_gate_finite() {
for &raw_temp in &[0.0_f32, 1e-9, 1e-6, 1e-3, 0.05, 0.5] {
let temp = raw_temp.max(1e-3);
for &aux_conf in &[0.0_f32, 0.05, 0.10, 0.25, 0.5] {
for &threshold in &[0.01_f32, 0.10, 0.20] {
let gate = 1.0 / (1.0 + ((-((aux_conf - threshold) / temp)) as f32).exp());
assert!(gate.is_finite(),
"gate must be finite for all (aux_conf={aux_conf}, \
threshold={threshold}, raw_temp={raw_temp}, floored \
temp={temp}); got {gate}");
assert!((0.0..=1.0).contains(&gate),
"gate must be in [0, 1] for all inputs; got {gate} \
at (aux_conf={aux_conf}, threshold={threshold}, \
temp={temp})");
}
}
}
}
/// SP20 Phase 5 (2026-05-10) — direct-to-trainer gather behavioral
/// round-trip.
///
/// Wires a fresh `CudaSlice<f32>` as the trainer destination via
/// `set_trainer_aux_conf_ptr(dst.raw_ptr())`, inserts N
/// transitions with distinct in-range aux_conf values, samples 1,
/// and asserts the destination buffer contains a value from the
/// inserted set (i.e., NOT the alloc_zeros 0.0 sentinel — the
/// gather actually fired). This proves the Phase 5 wiring (PER's
/// `gather_f32_scalar` direct-path) actually populates the
/// trainer-side buffer with non-trivial data.
///
/// Per `pearl_tests_must_prove_not_lock_observations`: invariant is
/// "destination contains an inserted value" (round-trip integrity),
/// not a locked specific value.
#[test]
fn aux_conf_direct_to_trainer_gather_populates_destination() {
let stream = make_stream();
let cfg = GpuReplayBufferConfig {
capacity: 64, alpha: 0.6, beta_start: 0.4, beta_max: 1.0,
beta_annealing_steps: 100, epsilon: 1e-6,
max_memory_bytes: 4 << 30, max_batch_size: 16,
};
let mut buf = GpuReplayBuffer::new(cfg, &stream).expect("buf");
let sd = ml_core::state_layout::STATE_DIM;
let n = 8;
let bs = 1;
// Pre-allocate a fresh "trainer destination" buffer (mirrors
// the production `aux_conf_at_state_buf` allocated by
// `GpuDqnTrainer::new`).
let dst = stream.alloc_zeros::<f32>(bs).expect("alloc dst");
// Wire it as the trainer destination so the direct-path
// gather fires.
buf.set_trainer_aux_conf_ptr(dst.raw_ptr());
// Pre-state assertion: the dst buffer is sentinel (alloc_zeros).
let mut dst_pre = vec![999.0_f32; bs];
stream.memcpy_dtoh(&dst, &mut dst_pre).expect("dtoh pre");
assert_eq!(dst_pre[0], 0.0,
"Pre-sample, trainer dst buffer should hold alloc_zeros sentinel");
// Build inserts with NON-ZERO aux_conf values so the post-sample
// destination is provably distinct from the sentinel.
let s = stream.alloc_zeros::<f32>(n * sd).unwrap();
let ns = stream.alloc_zeros::<f32>(n * sd).unwrap();
let a = stream.alloc_zeros::<u32>(n).unwrap();
let mut r = stream.alloc_zeros::<f32>(n).unwrap();
let d = stream.alloc_zeros::<f32>(n).unwrap();
let aux_sign = stream.alloc_zeros::<i32>(n).unwrap();
// Non-trivial rewards drive non-zero priorities (else PER samples
// randomly via uniform fallback).
let rewards_host: Vec<f32> = (0..n).map(|i| (i as f32 + 1.0) * 0.1).collect();
stream.memcpy_htod(&rewards_host, &mut r).unwrap();
// All distinct, all in range (0, 0.5]. By using values strictly
// > 0.0, the post-sample destination value being any of these
// proves the gather fired (NOT just the alloc_zeros sentinel).
let aux_conf_host: Vec<f32> = (0..n)
.map(|i| 0.05 + (i as f32) * 0.05) // 0.05, 0.10, ..., 0.40
.collect();
let mut aux_conf = stream.alloc_zeros::<f32>(n).unwrap();
stream.memcpy_htod(&aux_conf_host, &mut aux_conf).unwrap();
buf.insert_batch(&s, &ns, &a, &r, &d, &aux_sign, &aux_conf, n)
.expect("insert_batch with aux_conf");
assert_eq!(buf.len(), n, "post-insert size = n");
let ptrs = buf.sample_proportional(bs).expect("sample 1");
// Schema assertion: with `set_trainer_aux_conf_ptr` wired, the
// returned ptr equals the trainer destination (NOT the
// intermediate `sample_aux_conf` buffer).
assert_eq!(ptrs.aux_conf_ptr, dst.raw_ptr(),
"GpuBatchPtrs.aux_conf_ptr must equal the wired trainer ptr \
when set_trainer_aux_conf_ptr was called");
// Behavioral assertion: the trainer destination buffer was
// populated by the direct-path `gather_f32_scalar` — its value
// must be one of the inserted aux_conf_host values.
unsafe {
cudarc::driver::sys::cuStreamSynchronize(stream.cu_stream());
}
let mut dst_post = vec![999.0_f32; bs];
stream.memcpy_dtoh(&dst, &mut dst_post).expect("dtoh post");
let gathered = dst_post[0];
let in_set = aux_conf_host.iter().any(|&v| (v - gathered).abs() < 1e-6);
assert!(in_set,
"Direct-to-trainer gather populated dst[0] = {gathered}, but \
this is not in the inserted set {aux_conf_host:?} — either \
the gather did not fire (dst still at sentinel 0.0 — but our \
inserted set excludes 0.0) or the wiring is misrouted.");
assert!(gathered > 0.0,
"Direct-to-trainer gather must have fired (dst > 0.0); \
got dst = {gathered}");
}
}

7
crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs
View File

@@ -30691,6 +30691,13 @@ impl GpuDqnTrainer {
.arg(&cvar_alpha_buf_ptr)
// ── B3/G4: ISV signals for health-scaled Expected SARSA temperature ──
.arg(&self.isv_signals_dev_ptr)
// ── SP20 Phase 5: per-sample aux_conf at sampled state ──
// PER's `gather_f32_scalar` writes here every step (when
// `set_trainer_aux_conf_ptr` is wired at trainer init);
// c51 reads it for the reward gate at the Bellman target.
// NULL-tolerant in-kernel: gate=1.0 → identity if either
// aux_conf or isv_signals is NULL.
.arg(&self.aux_conf_at_state_buf)
.launch(LaunchConfig {
grid_dim: (b as u32, 1, 1),
block_dim: (256, 1, 1),

49
docs/dqn-wire-up-audit.md
View File

@@ -2,6 +2,55 @@
**Status:** Populated during Plan 1 Task 6 (A.5 orphan audit). Updated on every commit per Invariant 7.
## 2026-05-10 — SP20 Phase 5 (commit 3/4): launcher arg + tests
### Launcher
`GpuDqnTrainer::launch_c51_loss` (in `gpu_dqn_trainer.rs`) appends one
`.arg(&self.aux_conf_at_state_buf)` after the existing
`isv_signals_dev_ptr` argument. Position matches the kernel's appended
trailing arg from commit 2/4.
### Tests added (`crates/ml-dqn/src/gpu_replay_buffer.rs::tests`)
- **`aux_gate_high_confidence_passes_full_target`** (CPU pure-math).
Verifies `gate(aux_conf=0.5, threshold=0.10, temp=0.05) > 0.99` —
proves the "high confidence ⇒ near-identity reward" structural
invariant.
- **`aux_gate_low_confidence_attenuates_reward`** (CPU pure-math).
Verifies `gate(aux_conf=0.02, threshold=0.10, temp=0.05) < 0.20` —
proves the "uncertain-state neutralizer" semantic from Phase 3
Task 3.4 audit doc spec §4.4.
- **`aux_gate_temp_floor_keeps_gate_finite`** (CPU pure-math).
Sweeps {temp ∈ [0, 0.5]} × {aux_conf ∈ [0, 0.5]} × {threshold ∈
[0.01, 0.20]} and asserts `gate.is_finite()` ∧ `gate ∈ [0, 1]` for
all inputs — proves the `fmaxf(temp, 1e-3)` floor keeps the kernel
numerically stable across the full ISV-controllable parameter
range.
- **`aux_conf_direct_to_trainer_gather_populates_destination`** (GPU
behavioral). Wires a fresh `CudaSlice<f32>` as the trainer
destination via `set_trainer_aux_conf_ptr(dst.raw_ptr())`, inserts
8 transitions with strictly-positive distinct aux_conf values,
samples 1, asserts the destination buffer post-sample contains a
value from the inserted set (NOT the alloc_zeros sentinel of 0.0).
Proves the Phase 5 wiring (PER's `gather_f32_scalar` direct-path)
actually populates the trainer-side buffer with non-trivial data.
### Files modified
| File | Status | Purpose |
|------|--------|---------|
| `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` | +1 launcher arg | Thread aux_conf_at_state_buf into c51_loss kernel |
| `crates/ml-dqn/src/gpu_replay_buffer.rs` | +4 tests | Gate-formula + direct-gather verification |
### Verification
```
SQLX_OFFLINE=true cargo check --workspace --examples # green
SQLX_OFFLINE=true cargo test -p ml-dqn --lib aux_gate # 3/3 pass (CPU)
SQLX_OFFLINE=true cargo test -p ml-dqn --lib aux_conf # 3/3 pass (CPU + GPU integration)
```
## 2026-05-10 — SP20 Phase 5 (commit 2/4): c51_loss_kernel reward gate
Adds the consumer kernel-side gate. New kernel arg `const float*