feat(sp20): allocate trainer aux_conf_at_state_buf + wire PER direct-gather
Phase 5 plumbing — consumer-side wire-up. Allocates `aux_conf_at_state_buf: CudaSlice<f32>` ([batch_size]) on `GpuDqnTrainer`, exposes the raw_ptr via `aux_conf_at_state_buf_ptr()` and the `FusedTrainerCtx` delegating accessor `trainer_aux_conf_at_state_buf_ptr()`, and invokes `GpuReplayBuffer::set_trainer_aux_conf_ptr` at both fused_ctx init sites in training_loop.rs (init + re-init, atomic per `feedback_no_partial_refactor`). The PER `gather_f32_scalar` now writes the SAMPLED bar's per-batch aux_conf directly into the trainer's f32 buffer on every step — same direct-to-trainer pattern as the SP13 B1.1b `aux_nb_label_buf` (i32) wire-up immediately above the new call. The c51_loss_batched reward gate (lands in the next commit) reads this buffer to compute `gate = sigmoid((aux_conf - ISV[AUX_CONF_THRESHOLD]) / ISV[AUX_GATE_TEMP])` and applies `r_used = gate * reward` at the Bellman projection. `alloc_zeros` cold-start: 0.0 sentinel → at threshold ≈ 0.10 and temp ≈ 0.05 the gate is `sigmoid(-2) ≈ 0.12` → reward is mostly suppressed pre-population. This is the "graceful degradation" semantic from the Phase 3 Task 3.4 audit doc spec §4.4. Once PER's direct-gather populates from the producer ring on the first sample step, the per-bar aux_conf values drive the gate as designed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -6160,6 +6160,27 @@ pub struct GpuDqnTrainer {
|
||||
/// `[max_aux_param_len]` reusable per-tensor reduce target for
|
||||
/// `aux_param_grad_reduce`. Sized to the largest of the 8 aux tensors.
|
||||
aux_param_grad_final_buf: CudaSlice<f32>,
|
||||
/// SP20 Phase 5 (2026-05-10): per-sample aux confidence at the SAMPLED
|
||||
/// state — `[batch_size]` f32, range `[0, 0.5]` (K=2 peak-softmax-above-
|
||||
/// uniform; 0.0 = uniform / no information sentinel; 0.5 = peak softmax
|
||||
/// at 1.0). Populated by direct-to-trainer PER gather from the replay
|
||||
/// buffer's `aux_conf` ring (filled by `experience_env_step` per (i, t)
|
||||
/// in Phase 3 Task 3.4). Wired in at trainer init via
|
||||
/// `GpuReplayBuffer::set_trainer_aux_conf_ptr(self.aux_conf_at_state_buf
|
||||
/// .raw_ptr())` so PER's `gather_f32_scalar` writes the SAMPLED bar's
|
||||
/// per-batch aux_conf directly into here on every step (mirrors the
|
||||
/// SP13 B1.1b `aux_nb_label_buf` direct-to-trainer pattern). Consumed
|
||||
/// by `c51_loss_batched`'s reward gate at the Bellman projection:
|
||||
/// `gate = sigmoid((aux_conf - ISV[AUX_CONF_THRESHOLD]) / ISV[AUX_GATE_TEMP])`,
|
||||
/// `r_used = gate × reward`. NULL-tolerant (gate=1.0 = no-op) at the
|
||||
/// kernel level so test scaffolds without a wired aux head still work.
|
||||
/// `alloc_zeros` cold-start: 0.0 sentinel → at threshold ≈ 0.10 and
|
||||
/// temp ≈ 0.05 the gate is `sigmoid(-2) ≈ 0.12` → reward is mostly
|
||||
/// suppressed pre-population — the "graceful degradation" semantic
|
||||
/// from the Phase 3 Task 3.4 audit doc spec §4.4. Once the PER
|
||||
/// gather populates it on the first sample step, the per-bar values
|
||||
/// from the producer drive the gate as designed.
|
||||
aux_conf_at_state_buf: CudaSlice<f32>,
|
||||
/// Per-step aux-loss weight, baked into the SAXPY alpha at graph-capture
|
||||
/// time. ISV-driven via `set_aux_weight`:
|
||||
/// `aux_weight = clamp(0.1 × LEARNING_HEALTH × (1 - tanh(SHARPE_SCALE × sharpe_ema)), 0.05, 0.3)`.
|
||||
@@ -22564,6 +22585,12 @@ impl GpuDqnTrainer {
|
||||
.max(aux_kr)
|
||||
.max(aux_knb);
|
||||
let aux_param_grad_final_buf = alloc_f32(&stream, max_aux_tensor_len, "aux_param_grad_final_buf")?;
|
||||
// SP20 Phase 5: per-sample aux_conf at sampled state ([B] f32). PER's
|
||||
// direct-to-trainer `gather_f32_scalar` writes here on every sample
|
||||
// step once `set_trainer_aux_conf_ptr` is wired; until then, the
|
||||
// alloc_zeros 0.0 sentinel drives the c51_loss reward-gate to ~0
|
||||
// (graceful degradation per Phase 3 Task 3.4 audit doc spec §4.4).
|
||||
let aux_conf_at_state_buf = alloc_f32(&stream, aux_b, "aux_conf_at_state_buf")?;
|
||||
// Initial aux_weight at the lower numerical-stability clamp (0.05).
|
||||
// training_loop's per-step `set_aux_weight` recomputes from ISV every
|
||||
// step. The first graph capture bakes 0.05 — an honest cold-start
|
||||
@@ -24975,6 +25002,7 @@ impl GpuDqnTrainer {
|
||||
aux_partial_rg_w2,
|
||||
aux_partial_rg_b2,
|
||||
aux_param_grad_final_buf,
|
||||
aux_conf_at_state_buf,
|
||||
aux_weight,
|
||||
stochastic_depth_scale_buf,
|
||||
stochastic_depth_kernel,
|
||||
@@ -33800,6 +33828,17 @@ impl GpuDqnTrainer {
|
||||
/// in `gpu_experience_collector.rs`) into here on every step.
|
||||
pub(crate) fn aux_nb_label_buf_ptr(&self) -> u64 { self.aux_nb_label_buf.raw_ptr() }
|
||||
|
||||
/// SP20 Phase 5 (2026-05-10): raw pointer to `aux_conf_at_state_buf`
|
||||
/// for direct-to-trainer PER gather. Mirrors the SP13 B1.1b
|
||||
/// `aux_nb_label_buf_ptr` direct-to-trainer pattern. The buffer is
|
||||
/// `CudaSlice<f32>` `[batch_size]`; the c51 reward-gate consumer
|
||||
/// (`c51_loss_batched`) reads it as f32 to compute
|
||||
/// `gate = sigmoid((aux_conf - ISV[AUX_CONF_THRESHOLD]) / ISV[AUX_GATE_TEMP])`.
|
||||
/// `GpuReplayBuffer::set_trainer_aux_conf_ptr` is invoked once at
|
||||
/// trainer init in `training_loop.rs` so PER's `gather_f32_scalar`
|
||||
/// writes the SAMPLED bar's aux_conf into here on every step.
|
||||
pub(crate) fn aux_conf_at_state_buf_ptr(&self) -> u64 { self.aux_conf_at_state_buf.raw_ptr() }
|
||||
|
||||
/// Padded state dimension (128-byte aligned) for direct-to-trainer gather.
|
||||
pub(crate) fn state_dim_padded(&self) -> usize { ml_core::state_layout::STATE_DIM_PADDED }
|
||||
|
||||
|
||||
Reference in New Issue
Block a user