From 13ce783853355f79fba29e17d7b0b7d431410fcc Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Sun, 10 May 2026 02:22:15 +0200 Subject: [PATCH] =?UTF-8?q?test(sp20):=20post-fix=20runtime=20diagnostic?= =?UTF-8?q?=20=E2=80=94=20read-back=20accessors=20+=20producer=20test=20sc?= =?UTF-8?q?affold?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds infrastructure for verifying the SP20 Phase 2 Task 2.2-fix (`is_win_per_env`) producer side at runtime. The consumer-side oracle test passes; the producer-side wire-up has not been independently verified end-to-end. Production HEALTH_DIAG observed `wr_ema=0.0000` across all training epochs even after `64bbbe418` landed, with `alpha_ema` evolving normally — strongly suggesting either (a) all profitable trade closes had `hold_time == 0` so the producer write didn't fire, or (b) the producer write fired but `(segment_return > 0.0f)` evaluated to false in the production data. Code-inspection audit (this commit, summarized in audit-doc): - Verified 81 kernel args (launcher) match 81 kernel signature args for `experience_env_step`. `is_win_per_env` at position 81; `alpha_per_env` at position 78. cudarc `PushKernelArg<&mut CudaSlice>` impl pushes `&arg.cu_device_ptr` (stable address for the lifetime of the chained statement). - Verified buffer allocation at construction (alloc_zeros), pub(crate) field, FoldReset entry + dispatch arm, kernel-arg pass at both experience_env_step and sp20_aggregate_inputs launch sites. - Verified producer site — both alpha + is_win writes inside the SAME `if (segment_complete && segment_hold_time > 0.0f)` block at experience_kernels.cu:3196, both NULL-guarded, race-free. - Verified consumer site — aggregate kernel reads `is_win_per_env[env]` + `alpha_per_env[env]` side-by-side inside `if (tc != 0)`. - Verified stream ordering — both kernels run on `self.stream` in the same `for t in 0..timesteps` loop; experience kernel at line 6069, aggregate at 6577. Stream-implicit producer→consumer ordering. **No code-level bug surfaced by inspection.** The `is_win_per_env` write is structurally identical to the `alpha_per_env` write 14 lines earlier in the same gating block. Files modified: - `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs` — `read_is_win_per_env_for_test()` and `read_alpha_per_env_for_test()` pub fns. Mirror `read_epoch_dsr_state()` pattern: dtoh memcpy via the collector's stream, returns Vec / Vec. Cost-free in production (only called from tests) and benign for invariants. - `crates/ml/tests/sp20_is_win_producer_test.rs` — production-path producer test scaffold. Builds 6-float-per-bar synthetic OHLCV matching `experience_env_step`'s tgt[0..6] contract, runs `collect_experiences_gpu`, reads back both buffers, asserts the three structural invariants: 1. alpha_per_env has at least one non-zero slot (precondition — the synthetic upward-trend data must produce ≥ 1 trade close). 2. is_win_per_env has at least one slot == 1 (regression-pin for the Task 2.2-fix producer wire-up). 3. Co-occurrence: every env where alpha > 0 must have is_win == 1 (R_event > 0 ⇒ segment_return > 0 ⇒ is_win = 1; structural guarantee from the kernel's two writes at the same gate). Currently `#[ignore]` because the collector's cuBLAS forward chain requires a real `trainer_params_ptr` (NUM_WEIGHT_TENSORS=163 weights). Activation requires either trainer-params plumbing in the test scaffold OR moving the test inline as `#[cfg(test)]` in gpu_experience_collector.rs. - `docs/dqn-wire-up-audit.md` — audit-doc entry per Invariant 7. Documents the inspection scope, the unverified producer-side pre-conditions, and the open-issue framing for the wr_ema=0.0000 discrepancy. Verification: - SQLX_OFFLINE=true cargo check -p ml --tests passes - 9 state_reset_registry tests pass (incl. sp20_is_win_per_env_registered_fold_reset) - The new test file compiles cleanly under `--features cuda` Co-Authored-By: Claude Opus 4.7 (1M context) --- .../cuda_pipeline/gpu_experience_collector.rs | 35 +++ crates/ml/tests/sp20_is_win_producer_test.rs | 289 ++++++++++++++++++ docs/dqn-wire-up-audit.md | 97 ++++++ 3 files changed, 421 insertions(+) create mode 100644 crates/ml/tests/sp20_is_win_producer_test.rs diff --git a/crates/ml/src/cuda_pipeline/gpu_experience_collector.rs b/crates/ml/src/cuda_pipeline/gpu_experience_collector.rs index b3fbbe9c7..2aa145dc7 100644 --- a/crates/ml/src/cuda_pipeline/gpu_experience_collector.rs +++ b/crates/ml/src/cuda_pipeline/gpu_experience_collector.rs @@ -6817,6 +6817,41 @@ impl GpuExperienceCollector { /// Number of timesteps per episode in the pre-allocated output buffers. pub fn alloc_timesteps(&self) -> usize { self.alloc_timesteps } + /// SP20 Phase 2 Task 2.2-fix (2026-05-10) — read-back of the per-env + /// `is_win_per_env` `[alloc_episodes]` i32 trade-close win flag scratch. + /// Producer site is the segment_complete branch in `experience_env_step` + /// (writes `(segment_return > 0.0f) ? 1 : 0` per close); consumer is + /// `sp20_aggregate_inputs_kernel` for the WR_EMA wins_count reduction. + /// This getter exists for the production-path diagnostic test that + /// validates the producer side end-to-end (the existing kernel oracle + /// test only validates the consumer side). + pub fn read_is_win_per_env_for_test(&self) -> Result, MLError> { + let mut host = vec![0_i32; self.alloc_episodes]; + self.stream + .memcpy_dtoh(&self.is_win_per_env, &mut host) + .map_err(|e| MLError::ModelError(format!( + "read_is_win_per_env_for_test memcpy_dtoh: {e}" + )))?; + Ok(host) + } + + /// SP20 Phase 2 Task 2.2-fix (2026-05-10) — read-back of the per-env + /// `alpha_per_env` `[alloc_episodes]` f32 trade-close alpha scratch. + /// Companion to `read_is_win_per_env_for_test` — both are written at + /// the same segment_complete site so the test asserts the producer + /// fires by checking that BOTH buffers received writes (alpha as f32 + /// non-zero, is_win as i32 with at least one slot == 1 if any trade + /// closed profitably). + pub fn read_alpha_per_env_for_test(&self) -> Result, MLError> { + let mut host = vec![0_f32; self.alloc_episodes]; + self.stream + .memcpy_dtoh(&self.alpha_per_env, &mut host) + .map_err(|e| MLError::ModelError(format!( + "read_alpha_per_env_for_test memcpy_dtoh: {e}" + )))?; + Ok(host) + } + /// Raw device pointer to the portfolio_state buffer [alloc_episodes * PS_STRIDE]. /// Used by the kelly_cap_update kernel (Plan 1 Task 11) to aggregate Kelly /// win/loss stats across environments at epoch boundaries without a GPU→CPU roundtrip. diff --git a/crates/ml/tests/sp20_is_win_producer_test.rs b/crates/ml/tests/sp20_is_win_producer_test.rs new file mode 100644 index 000000000..b5416a5fe --- /dev/null +++ b/crates/ml/tests/sp20_is_win_producer_test.rs @@ -0,0 +1,289 @@ +//! SP20 Phase 2 Task 2.2-fix (2026-05-10) — production-path producer +//! diagnostic for the `is_win_per_env` segment-level win-flag buffer. +//! +//! The aggregate-kernel oracle test +//! (`wr_ema_uses_segment_pnl_via_is_win_per_env_predicate` in +//! `sp20_aggregate_inputs_test`) validates the **consumer** side: when the +//! buffer is pre-populated with synthetic 1s, the kernel correctly +//! aggregates them into `wins_count`. That test passes. +//! +//! This test validates the **producer** side: when `experience_env_step` +//! runs the production launch path with synthetic OHLCV that forces +//! profitable trade closes, the kernel MUST write `1` into +//! `is_win_per_env[i]` at the segment_complete branch. +//! +//! Bug class targeted: production HEALTH_DIAG observed `wr_ema=0.0000` +//! across all training epochs even AFTER the kernel-body fix landed in +//! commit `64bbbe418`. Two runs (pre-fix vs post-fix) produced near +//! bit-identical EMA values for `alpha_ema` (which is written at the +//! SAME site), strongly suggesting the `is_win_per_env` write site is +//! NOT firing in production despite passing the consumer-side oracle +//! test. Companion test pins a parallel invariant: the `alpha_per_env` +//! producer DOES fire (alpha_ema evolves with non-zero values in +//! production), so any test where alpha_per_env got writes but +//! is_win_per_env did not would be a smoking gun for the bug. +//! +//! Run on a GPU host: +//! +//! SQLX_OFFLINE=true CUDA_COMPUTE_CAP=86 \ +//! cargo test -p ml --test sp20_is_win_producer_test --features cuda \ +//! -- --ignored --nocapture + +#![allow(clippy::tests_outside_test_module)] + +#[cfg(feature = "cuda")] +#[allow(unsafe_code)] // CUDA kernel launch + mapped-pinned memory. +mod gpu { + use std::sync::Arc; + + use ml::cuda_pipeline::gpu_experience_collector::{ + ExperienceCollectorConfig, GpuExperienceCollector, + }; + use ml::cuda_pipeline::mapped_pinned::MappedF32Buffer; + use ml::cuda_pipeline::shared_cublas_handle::PerStreamCublasHandles; + use ml_core::device::MlDevice; + + type CudaStream = cudarc::driver::CudaStream; + + fn try_cuda() -> Option> { + match MlDevice::cuda(0) { + Ok(dev) => Some(dev.cuda_stream().expect("CUDA stream required").clone()), + Err(_) => None, + } + } + + /// Build a 6-float-per-bar target buffer matching + /// `experience_kernels.cu::experience_env_step`'s contract: + /// `[preproc_close, preproc_next, raw_close, raw_next, raw_open, mid_open]` + /// + /// `prices[i]` is used as the raw close at bar i; raw_open is set to + /// `prices[i]` (intra-bar move = 0 — fine for synthetic test). + /// preproc_* are set to 0.0 (the Q-network reads them but our test + /// doesn't depend on Q-values). + fn build_targets(prices: &[f32]) -> MappedF32Buffer { + let n = prices.len(); + let mut data = vec![0.0_f32; n * 6]; + for i in 0..n { + let p = prices[i]; + let p_next = if i + 1 < n { prices[i + 1] } else { p }; + data[i * 6] = 0.0; // preproc_close + data[i * 6 + 1] = 0.0; // preproc_next + data[i * 6 + 2] = p; // raw_close + data[i * 6 + 3] = p_next; // raw_next + data[i * 6 + 4] = p; // raw_open + data[i * 6 + 5] = p; // mid_open + } + let buf = unsafe { MappedF32Buffer::new(n * 6) }.expect("alloc targets"); + buf.write_from_slice(&data); + buf + } + + /// Build a feature buffer where every bar has features that yield a + /// strong "long" signal (positive market values across all 51 dims). + /// Combined with `epsilon=1.0` random action selection, this gives + /// enough action diversity to trigger trade openings + closings. + fn build_market_features(n_bars: usize) -> MappedF32Buffer { + const MARKET_DIM: usize = 51; + let mut data = vec![0.0_f32; n_bars * MARKET_DIM]; + for i in 0..n_bars { + for j in 0..MARKET_DIM { + // Slightly-positive features so the network's untrained + // Q-values lean toward longs/shorts roughly equally with + // epsilon=1 random override. + data[i * MARKET_DIM + j] = (i as f32 * 0.001).sin() * 0.5; + } + } + let buf = unsafe { MappedF32Buffer::new(n_bars * MARKET_DIM) } + .expect("alloc market"); + buf.write_from_slice(&data); + buf + } + + /// SP20 Phase 2 Task 2.2-fix — production-path producer diagnostic. + /// + /// Constructs a synthetic monotonic upward price series so that any + /// long position with positive hold time closes profitably + /// (`segment_return > 0`). With `epsilon=1.0` (full random action), + /// the experience collector explores enough action space to open AND + /// close trades within `timesteps_per_episode=200` bars. + /// + /// Asserts: + /// 1. `alpha_per_env` has at least one non-zero slot — the producer + /// site at the segment_complete branch fires for at least one env + /// (the existing reference behavior — alpha_ema evolves in + /// production). + /// 2. `is_win_per_env` has at least one slot == 1 — the SP20 Phase 2 + /// Task 2.2-fix write site (`is_win_per_env[i] = (segment_return > + /// 0.0f) ? 1 : 0`) fires for at least one profitable trade close. + /// + /// **Without the fix**: assertion 2 FAILS even though assertion 1 + /// passes. With the fix wired correctly: BOTH pass. + /// + /// **The two assertions are at the SAME site in the kernel** (both + /// inside `if (segment_complete && segment_hold_time > 0.0f) {...}`), + /// so the alpha-passes-but-is-win-fails outcome is the structural + /// signature of the kernel-arg-threading or buffer-wireup bug + /// described in the audit-doc 2026-05-10 entry. + #[test] + #[ignore = "requires GPU"] + fn is_win_per_env_producer_fires_at_segment_complete() { + let Some(stream) = try_cuda() else { + eprintln!("SKIP: CUDA not available"); + return; + }; + + // Strong upward trend — every long position with hold > 0 closes + // profitably (segment_return > 0). This forces is_win_per_env=1 + // for at least one env if the producer fires. + let n_bars = 500_usize; + let mut prices = Vec::with_capacity(n_bars); + for i in 0..n_bars { + // Linear ramp from 100 to 200 — robust >0 segment_return on + // any long position closed after at least one bar. + prices.push(100.0 + (i as f32) * 0.2); + } + + let market_buf = build_market_features(n_bars); + let target_buf = build_targets(&prices); + + let shared_cublas = Arc::new( + PerStreamCublasHandles::new(&stream).expect("cublas handles"), + ); + // Run multiple episodes to maximize chance of trade close + // happening in at least one env. alloc_episodes >= n_episodes is + // required by the collector contract. Match the existing smoke + // test's (4, 50) allocation pattern that's known to work without + // explicit weight wiring. + let n_episodes = 4_usize; + let timesteps = 50_i32; + let mut collector = GpuExperienceCollector::new( + stream.clone(), + shared_cublas, + 0, // bottleneck_dim + 51, // market_dim + 100_000.0, + 0.01, // tx_cost_multiplier + 0.05, // max_position + (64, 64, 32, 32), + (51, 51), + n_episodes, // alloc_episodes + timesteps as usize, // alloc_timesteps + 0.0, // curiosity_weight + ) + .expect("create collector"); + let episode_starts: Vec = (0..n_episodes) + .map(|i| (i * (n_bars / n_episodes)) as i32) + .collect(); + + let mut exp_config = ExperienceCollectorConfig::default(); + exp_config.n_episodes = n_episodes as i32; + exp_config.timesteps_per_episode = timesteps; + exp_config.total_bars = n_bars as i32; + exp_config.episode_length = timesteps; + exp_config.epsilon = 1.0; // full random — maximizes trade diversity + exp_config.gamma = 0.99; + + let _batch = collector + .collect_experiences_gpu(&market_buf, &target_buf, &episode_starts, &exp_config) + .expect("collect_experiences_gpu"); + + // ── Read back per-env producer buffers ────────────────────────── + let alpha = collector + .read_alpha_per_env_for_test() + .expect("read alpha_per_env"); + let is_win = collector + .read_is_win_per_env_for_test() + .expect("read is_win_per_env"); + + // Assertion 1 — alpha producer fires (reference behavior). + let alpha_nonzero = alpha.iter().any(|&v| v != 0.0); + let alpha_writes = alpha.iter().filter(|&&v| v != 0.0).count(); + assert!( + alpha_nonzero, + "PRECONDITION FAILED — alpha_per_env has zero non-zero slots \ + out of {} after {} episodes × {} steps. The synthetic \ + upward-trend data must produce at least one trade close \ + with hold_time > 0 (alpha=R_event != 0). If this assertion \ + fails, the test scaffold is broken (no trade closes \ + happened); fix the test, not the production code.", + alpha.len(), + n_episodes, + timesteps, + ); + + eprintln!( + "alpha_per_env: {}/{} slots non-zero, sum={:.4}, max={:.4}", + alpha_writes, + alpha.len(), + alpha.iter().sum::(), + alpha.iter().cloned().fold(f32::NEG_INFINITY, f32::max), + ); + + // Assertion 2 — is_win producer fires (SP20 Phase 2 Task 2.2-fix). + let is_win_count = is_win.iter().filter(|&&v| v == 1).count(); + let is_win_max: i32 = is_win.iter().copied().max().unwrap_or(0); + + eprintln!( + "is_win_per_env: {} slots == 1 (out of {}), max={}", + is_win_count, + is_win.len(), + is_win_max, + ); + + assert!( + is_win_max > 0, + "SP20 Phase 2 Task 2.2-fix REGRESSION — is_win_per_env has \ + ZERO slots set to 1 across {} alloc_episodes, but \ + alpha_per_env has {} non-zero slots. Both buffers are \ + written at the SAME segment_complete site in \ + experience_env_step (line 3322 alpha, line 3337 is_win). \ + If alpha was written and is_win was not, the kernel-arg \ + threading for is_win_per_env is broken — likely the \ + pointer is being passed as NULL despite the launcher \ + alloc'ing the buffer. This is the production WR_EMA=0.0000 \ + pinning bug — the fix in commit `64bbbe418` did NOT \ + actually fire at runtime.", + is_win.len(), + alpha_writes, + ); + + // Tighter contract: every slot where alpha_per_env has a + // **positive** value (R_event > 0 ⇒ segment_return > 0) MUST + // have is_win_per_env == 1 at the same env index. This is the + // structural co-occurrence the kernel guarantees: + // + // alpha[i] = R_event ∈ {-loss_cap, -0.5, 0, +0.5, +1.0} + // is_win[i] = (segment_return > 0) ? 1 : 0 + // + // R_event > 0 implies segment_return > 0 implies is_win[i] = 1. + // + // Skip slot 0 (first-write may not yet be set if env didn't + // close), check ANY env where alpha > 0 has is_win = 1. + let positive_alpha_slots: Vec = alpha + .iter() + .enumerate() + .filter_map(|(i, &v)| if v > 0.0 { Some(i) } else { None }) + .collect(); + + if !positive_alpha_slots.is_empty() { + let mismatched: Vec = positive_alpha_slots + .iter() + .copied() + .filter(|&i| is_win[i] == 0) + .collect(); + assert!( + mismatched.is_empty(), + "SP20 Phase 2 Task 2.2-fix CO-OCCURRENCE VIOLATION — \ + envs with alpha_per_env > 0 (R_event positive ⇒ \ + segment_return > 0) MUST have is_win_per_env == 1. \ + Found {} envs where alpha > 0 but is_win == 0: {:?}. \ + This is the structural smoking gun for the kernel-arg \ + wiring bug (the producer site writes both fields in \ + succession at the same gating; if one fires the other \ + must too).", + mismatched.len(), + mismatched, + ); + } + } +} diff --git a/docs/dqn-wire-up-audit.md b/docs/dqn-wire-up-audit.md index 5b665eeb1..6e824b43c 100644 --- a/docs/dqn-wire-up-audit.md +++ b/docs/dqn-wire-up-audit.md @@ -2,6 +2,103 @@ **Status:** Populated during Plan 1 Task 6 (A.5 orphan audit). Updated on every commit per Invariant 7. +## 2026-05-10 — SP20 Phase 2 Task 2.2-fix: post-fix runtime diagnostic infrastructure (read-back + producer test scaffold) + +Adds infrastructure for verifying the SP20 Phase 2 Task 2.2-fix +(`is_win_per_env`) producer side at runtime. The consumer-side +`sp20_aggregate_inputs_kernel` oracle test +(`wr_ema_uses_segment_pnl_via_is_win_per_env_predicate`) passes — when +the buffer is pre-populated with synthetic 1s, the kernel correctly +aggregates them into `wins_count`. **The producer-side wire-up has +not been independently verified** — runtime evidence +(production HEALTH_DIAG observation that `wr_ema=0.0000` across all +training epochs even after `64bbbe418` landed, with `alpha_ema` +evolving normally) suggests the producer write at +`experience_kernels.cu::3337` may not be firing in production despite +all visible code paths appearing correctly wired. + +### Code-inspection audit (this commit) + +Verified: + +1. Kernel signature for `experience_env_step` — 81 args, matching the + Rust launcher's 81 `.arg(...)` calls. `is_win_per_env` at position + 81; `alpha_per_env` at position 78; both pass through cudarc's + `PushKernelArg<&mut CudaSlice>` impl which pushes + `&arg.cu_device_ptr` (a stable address for the lifetime of the + chained statement). +2. Buffer allocation — `alpha_per_env: CudaSlice [alloc_episodes]` + and `is_win_per_env: CudaSlice [alloc_episodes]` — both + `alloc_zeros`, both pub(crate) on `GpuExperienceCollector`. +3. Producer site — both writes inside the SAME + `if (segment_complete && segment_hold_time > 0.0f)` block at + `experience_kernels.cu:3196`, both NULL-guarded, race-free + (per-thread `[i]` writes), 14 lines apart. +4. Consumer site — aggregate kernel reads `is_win_per_env[env]` and + `alpha_per_env[env]` side-by-side inside the SAME + `if (tc != 0)` block. +5. State reset — `is_win_per_env` registered FoldReset, dispatch arm + present in `training_loop.rs::9572`. `every_fold_and_soft_reset_ + entry_has_dispatch_arm` test passes. +6. Stream ordering — both kernels run on `self.stream` in the same + `for t in 0..timesteps` loop; experience kernel launches at line + 6069, aggregate kernel at line 6577. Stream-implicit + producer→consumer ordering. + +**No code-level bug surfaced by inspection.** The `is_win_per_env` +write site is structurally identical to the working `alpha_per_env` +write site. + +### Files modified + +| File | Status | Purpose | +|------|--------|---------| +| `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs` | +2 pub fns | `read_is_win_per_env_for_test` + `read_alpha_per_env_for_test` device→host read-back accessors. dtoh memcpy gated behind public methods so future tests + production diagnostics can verify per-env producer firing without leaking field internals. | +| `crates/ml/tests/sp20_is_win_producer_test.rs` | +scaffold | Production-path producer test scaffold: builds 6-float-per-bar synthetic OHLCV, runs `collect_experiences_gpu`, reads back both per-env buffers. **Currently `#[ignore]`** because the collector's cuBLAS forward chain requires a real `trainer_params_ptr` (NUM_WEIGHT_TENSORS=163 weights) which the standalone test doesn't yet wire. The scaffold structure documents the intended invariants (alpha non-zero ⇒ is_win has at least one `1`; per-env alpha > 0 ⇒ same env's is_win == 1) for the future end-to-end verification. | +| `docs/dqn-wire-up-audit.md` | +entry | This entry | + +### Behavioural pre-conditions (post-fix) + +If the SP20 Phase 2 Task 2.2-fix is firing correctly, the following +invariants hold (they are currently **NOT** verified end-to-end — +only consumer-side and code-inspection): + +1. After any rollout step where at least one env had a profitable + trade close (segment_pnl > 0) AND held the position for ≥ 1 bar: + - `alpha_per_env[i]` for that env is in {+0.5, +1.0} (R_event from + `sp20_compute_event_reward` 4-quadrant table). + - `is_win_per_env[i]` for that env is `1`. +2. The aggregate kernel's `wins_count` reduction over closed envs is + strictly equal to `sum(is_win_per_env[env] != 0 ; trade_close_ + per_env[env*L + t] != 0)`. +3. The EMAs kernel's `WR_EMA` slot evolves with non-zero values when + `EmaInputs.is_close == 1` AND `2 * wins_count >= closed_count` for + the rollout step. + +### Outstanding verification + +The bit-similar-but-not-identical evidence (production runs: +alpha_ema=0.0555 → 0.0195 → 0.0143 (BUGGY) vs 0.0555 → 0.0194 → +0.0143 (FIX) — diverging at the 4th decimal at epoch 1 — implies the +kernel-arg threading is running differently in the FIX branch (extra +work / register pressure shifts float ordering). However, `wr_ema= +0.0000` in BOTH runs implies that EITHER: + +a) The producer write at `experience_kernels.cu:3337` is firing but + `(segment_return > 0.0f)` evaluates to false EVERY time it + evaluates (no profitable trade closes happen with `hold_time > 0` + in the production data — possible if the policy mostly executes + immediate-reversal trades or the policy collapses to + pre-segment-close exits). +b) The producer write is somehow not firing despite the apparently- + correct code path. + +The producer test scaffold landed in this commit will verify +hypothesis (a) vs (b) once the trainer-params plumbing is wired (or +moved to inline `#[cfg(test)]` in `gpu_experience_collector.rs` +with full visibility). Until that lands, the discrepancy remains an +**open issue** flagged via this audit-doc entry. + ## 2026-05-10 — SP20 Phase 2 Task 2.2-fix: WR_EMA pinning bug — fix commit (kernel body + producer + buffer + reset) Closes the fix gap left by the failing-test commit. The aggregation