gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

plan(sp18 v2): Phase 0 Task 0.1 — D-leg per-action reward decomposition diagnostic

Browse Source 
Adds the SP18 v2 Phase 0 D-leg observability scaffold per the plan's
"Phase 0 — diagnostic emit (NO functional change)" task:

- New kernel `reward_decomp_diag_kernel.cu`: block tree-reduce
  (4 blocks × 256 threads, one block per direction-axis bin) reading
  `reward_components_per_sample [N×6]` + `actions_out [N]` and emitting
  5 per-bin stats (mean r_micro / mean r_opp_cost / mean r_popart /
  mean |reward| / fire_rate) into a 20-float row-major output. Bin
  order: Hold(0)→Long(1)→Short(2)→Flat(3); col order: micro→opp→
  popart→abs→fire. Empty-bin guard emits 0.0 (NOT NaN) per the
  consumer-side KILL CRITERION arithmetic contract.

- Cubin manifest entry in `crates/ml/build.rs` + `REWARD_DECOMP_DIAG_
  CUBIN` re-export in `gpu_dqn_trainer.rs`.

- 20-float `MappedF32Buffer sp18_reward_decomp_diag_buf` field on
  `GpuDqnTrainer` + accessor pair (`sp18_reward_decomp_diag_dev_ptr`
  for the writer-side launcher; `read_sp18_reward_decomp_diag` for the
  HEALTH_DIAG reader). Buffer is constructor-zeroed so cold-start
  HEALTH_DIAG emits a deterministic zero block.

- `sp18_reward_decomp_diag_kernel` field on `GpuExperienceCollector` +
  cubin load on the collector's stream + `launch_sp18_reward_decomp_
  diag(n, b1, b2, b3, out_dev_ptr)` launcher. Wired in
  `training_loop.rs` at the per-step boundary, BEFORE
  `launch_reward_component_ema_inplace` (which `memset_zeros` the
  source buffer after consuming it) per `pearl_canary_input_freshness_
  launch_order`.

- New per-epoch HEALTH_DIAG line emit at the existing per-epoch
  boundary (after the SP17 dueling line):

    HEALTH_DIAG[N]: reward_decomp [hold(micro=X opp=Y popart=Z abs=W
                    fire=F) long(...) short(...) flat(...)]

  Reads the mapped-pinned 20-float diag buffer directly via the
  collector→trainer host_ptr — no DtoH copy.

- New `crates/ml/tests/sp18_hold_reward_oracle_tests.rs`:
  * `reward_decomp_per_action_cpu_oracle` (CPU oracle pinning the
    per-bin reduction math against a 4-sample synthetic batch).
  * `reward_decomp_per_action_gpu_oracle` (GPU oracle, ignored unless
    `--ignored`; asserts kernel matches CPU oracle bit-for-bit within
    1e-6 f32 budget).
  * `reward_decomp_empty_bin_emits_zero_not_nan` (empty-bin contract
    guard).

Pure observability — no production-path consumer in this commit. No
reward changes, no Bellman target changes, no kernel modifications to
the action-selection or training paths. Per
`feedback_no_partial_refactor` the kernel + cubin manifest + buffer +
launcher + production wire-up + HEALTH_DIAG emit + GPU oracle test all
land atomically.

Verification:
  SQLX_OFFLINE=true CUDA_COMPUTE_CAP=86 cargo check --workspace clean.
  CPU oracle test passes; GPU oracle + empty-bin guard both pass on
  RTX 3050 Ti (2.13s).

Plan: docs/superpowers/plans/2026-05-08-sp18-reward-shape-hold-attractor.md
      § Phase 0 Task 0.1.
Audit: docs/dqn-wire-up-audit.md § "SP18 v2 Phase 0 Task 0.1".

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-09 01:06:48 +02:00
parent 52100ae7a1
commit 15b50ac38f
7 changed files with 921 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

96
crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs
View File

@@ -114,6 +114,15 @@ static ATOMS_UPDATE_CUBIN: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/ato
static Q_QUANTILE_CUBIN: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/q_quantile_kernel.cubin"));
/// Plan 3 Task 1 C.2: per-component |reward| EMA into ISV[63..69).
pub(crate) static REWARD_COMPONENT_EMA_CUBIN: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/reward_component_ema_kernel.cubin"));
/// SP18 v2 Phase 0 Task 0.1 (2026-05-08): D-leg per-action reward
/// decomposition diagnostic. Reads `reward_components_per_sample`
/// + `actions_out` from the collector, bins per-direction
/// (DIR_HOLD / DIR_LONG / DIR_SHORT / DIR_FLAT), emits 5 stats per bin
/// (mean r_micro / r_opp_cost / r_popart / |reward| / fire_rate) into
/// a 20-float mapped-pinned `reward_decomp_diag_buf`. Block tree-
/// reduce — 4 blocks × 256 threads. Pure observability — no
/// production-path consumer in this commit. Plan: SP18 Phase 0 Task 0.1.
pub(crate) static REWARD_DECOMP_DIAG_CUBIN: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/reward_decomp_diag_kernel.cubin"));
/// Plan 3 Task 3 B.2: Flat→Positioned transition rate EMA into ISV[71].
pub(crate) static TRADE_RATE_EMA_CUBIN: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/trade_rate_ema_kernel.cubin"));
/// Plan 3 Task 4 B.4: per-batch readiness EMA + derived plan_threshold.
@@ -6803,6 +6812,25 @@ pub struct GpuDqnTrainer {
/// host writes — kernel-owned). Lifetime matches the trainer; reset
/// once at construct, written-then-read inside the kernel each launch.
sp17_advantage_clip_scratch: super::mapped_pinned::MappedF32Buffer,
// ── SP18 v2 Phase 0 Task 0.1 (2026-05-08): D-leg per-action reward
// decomposition diagnostic buffer ──────────────────────────────────
/// 20-float mapped-pinned diagnostic buffer for the SP18 D-leg
/// per-action reward decomposition. Layout (row-major, 4×5):
///
/// row 0 = Hold, row 1 = Long, row 2 = Short, row 3 = Flat
/// col 0 = mean(r_micro), col 1 = mean(r_opp_cost),
/// col 2 = mean(r_popart), col 3 = mean(|reward|),
/// col 4 = fire_rate
///
/// Written by the collector's
/// `launch_sp18_reward_decomp_diag(...)` kernel each per-step
/// boundary, BEFORE `launch_reward_component_ema_inplace` zeros the
/// source buffer. Consumed exclusively by the per-epoch HEALTH_DIAG
/// emit at `training_loop.rs`. Pure observability — no production-
/// path consumer in this commit. Plan: SP18 Phase 0 Task 0.1.
pub(crate) sp18_reward_decomp_diag_buf: super::mapped_pinned::MappedF32Buffer,
// SP14 Layer C Phase C.1 (2026-05-08): α-machinery struct fields deleted
// atomically with the kernel files (alpha_grad_compute_kernel,
// gradient_hack_detect_kernel, sp14_scale_wire_col_kernel) per
@@ -9407,6 +9435,48 @@ impl GpuDqnTrainer {
Ok(self.read_isv_signal_at(ADVANTAGE_CLIP_BOUND_INDEX))
}
/// SP18 v2 Phase 0 Task 0.1 (2026-05-08) — D-leg per-action reward
/// decomposition diagnostic device pointer.
///
/// Returns the mapped-pinned device pointer of the 20-float
/// `sp18_reward_decomp_diag_buf` so the collector's
/// `launch_sp18_reward_decomp_diag` can write into it. The host-side
/// HEALTH_DIAG emit reads via `read_sp18_reward_decomp_diag()` after
/// stream sync. Pure observability — caller is expected to be the
/// per-step training loop wiring point.
pub fn sp18_reward_decomp_diag_dev_ptr(&self) -> u64 {
self.sp18_reward_decomp_diag_buf.dev_ptr
}
/// SP18 v2 Phase 0 Task 0.1 (2026-05-08) — D-leg per-action reward
/// decomposition diagnostic readback.
///
/// Returns the 20-float diagnostic block as a `[f32; 20]`, row-major
/// 4 bins × 5 stats:
///
/// [Hold_micro, Hold_opp, Hold_popart, Hold_abs, Hold_fire,
/// Long_micro, Long_opp, Long_popart, Long_abs, Long_fire,
/// Short_micro, Short_opp, Short_popart, Short_abs, Short_fire,
/// Flat_micro, Flat_opp, Flat_popart, Flat_abs, Flat_fire]
///
/// Reads the mapped-pinned host_ptr directly — caller MUST have
/// stream-synced after the collector's launch (e.g. via the natural
/// per-step boundary that already runs `cuStreamSynchronize` between
/// collection and training). No HtoD/DtoH copy.
///
/// On cold-start (before any kernel launch) returns the
/// constructor-initialised zero block.
pub fn read_sp18_reward_decomp_diag(&self) -> [f32; 20] {
let v = self.sp18_reward_decomp_diag_buf.read_all();
debug_assert_eq!(
v.len(), 20,
"sp18_reward_decomp_diag_buf size invariant: 4 bins × 5 stats"
);
let mut out = [0.0_f32; 20];
for (i, x) in v.iter().take(20).enumerate() { out[i] = *x; }
out
}
/// Update per-branch liquid tau modulation — GPU kernel (RK4/Euler adaptive ODE).
///
/// Reads per_branch_q_gaps (pinned device-mapped) and qlstm_context from device.
@@ -20568,6 +20638,27 @@ impl GpuDqnTrainer {
"sp17_advantage_clip_scratch alloc ({sp17_clip_scratch_len} f32): {e}"
)))?;
// SP18 v2 Phase 0 Task 0.1 (2026-05-08): D-leg per-action reward
// decomposition diagnostic buffer. 4 bins × 5 stats = 20 floats
// mapped-pinned. The collector's
// `launch_sp18_reward_decomp_diag` writes per-step; the host-side
// HEALTH_DIAG emit reads via the mapped host_ptr after stream
// sync. Pure observability — no production-path consumer.
const SP18_REWARD_DECOMP_DIAG_LEN: usize = 4 * 5;
let sp18_reward_decomp_diag_buf = unsafe {
super::mapped_pinned::MappedF32Buffer::new(SP18_REWARD_DECOMP_DIAG_LEN)
}
.map_err(|e| MLError::ModelError(format!(
"sp18_reward_decomp_diag_buf alloc ({SP18_REWARD_DECOMP_DIAG_LEN} f32): {e}"
)))?;
// Initialise to 0.0 so the first HEALTH_DIAG read on cold-start
// (before the collector has fired) is a deterministic zero block
// rather than indeterminate page contents. Pearl-A first-
// observation bootstrap fires inside the kernel itself when
// n_samples > 0; this just zero-fills before the kernel ever runs.
sp18_reward_decomp_diag_buf
.write_from_slice(&vec![0.0_f32; SP18_REWARD_DECOMP_DIAG_LEN]);
// SP14 Layer C Phase C.1 (2026-05-08): α-machinery cubin loads
// (alpha_grad_compute, gradient_hack_detect, sp14_scale_wire_col)
// deleted atomically with the kernel files. Coupling A
@@ -24810,6 +24901,11 @@ impl GpuDqnTrainer {
sp17_v_share_kernel,
sp17_advantage_clip_bound_kernel,
sp17_advantage_clip_scratch,
// SP18 v2 Phase 0 Task 0.1 (2026-05-08): D-leg per-action
// reward decomposition diagnostic buffer. Mapped-pinned 20
// floats (4 bins × 5 stats). Written by collector kernel,
// read host-side at HEALTH_DIAG cadence.
sp18_reward_decomp_diag_buf,
sp14_dir_concat_qaux_kernel,
sp14_dir_qaux_concat_scratch,
// SP14 backward dX scratch — column-SH2 wire is dropped at