# GPU Log Ring — Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development. Each task is checkbox-tracked.

**Goal:** Unified high-performance in-kernel diagnostic logging for ml-alpha CUDA kernels via mapped-pinned ring buffer with deterministic slot reservation (no atomicAdd), header-last commit, and async host drain.

**Predecessor spec:** `docs/superpowers/specs/2026-05-21-gpu-log-ring-design.md`

**Architecture in one paragraph.** A 32768-slot × 64 B mapped-pinned ring buffer holds `LogRecord` entries. A 1-thread tick kernel runs first in each captured-graph replay, incrementing a device `step_counter`. Every logging-capable kernel takes two new device-pointer args (`LogRing*`, `const int* step_counter`) and emits records via a `log_record()` device helper that derives its slot from `(step * N_RPS + kid * N_RT + rt) & MASK`, single-writer per slot, header-last commit. CPU drainer is a tokio task that volatile-reads the ring at 500 ms cadence, dispatching to per-(kernel_id, record_type) decoders that emit structured `tracing::info!` events. All gated by a `cuda-diag-log` Cargo feature.

**Tech Stack:** Rust 1.85+, cudarc 0.19, CUDA 12.4 (sm_89 L40S, sm_80 RTX 3050 Ti local), `tokio`, `tracing`, pre-compiled cubins.

---

## File structure (locked-in decisions)

| File | Action |
|------|--------|
| `crates/ml-alpha/cuda/gpu_log_ids.h` | NEW — kernel_id + record_type macros (single source of truth) |
| `crates/ml-alpha/cuda/gpu_log_ring.cu` | NEW — `LogRecord`, `LogRing`, `log_record()` device helper, `gpu_log_tick` kernel |
| `crates/ml-alpha/build.rs` | Add `"gpu_log_ring"` to KERNELS; bump cache-bust v13 → v14 |
| `crates/ml-alpha/Cargo.toml` | Add feature flag `cuda-diag-log = []` |
| `crates/ml-alpha/src/pinned_mem.rs` | Add generic `MappedRecordBuffer<T>`; `MappedF32Buffer` becomes alias `MappedRecordBuffer<f32>` |
| `crates/ml-alpha/src/gpu_log.rs` | NEW — ring allocator, tokio drain task, decoder registry |
| `crates/ml-alpha/src/lib.rs` | `#[cfg(feature = "cuda-diag-log")] pub mod gpu_log;` |
| `crates/ml-alpha/tests/gpu_log_ring_invariants.rs` | NEW — 6 unit tests + 1 integration test |
| `crates/ml-alpha/cuda/smoothness_lambda_controller.cu` | Add `g_log_ring`, `g_step_counter` args; emit RT_INPUT/STATE/OUTPUT |
| `crates/ml-alpha/src/trainer/perception.rs` | Allocate ring + step counter at construction (feature-gated); wire pointers into smoothness controller launch |

---

## Task 1: Central IDs header

- [ ] **1.1** Create `crates/ml-alpha/cuda/gpu_log_ids.h`:

```c
// gpu_log_ids.h — single source of truth for log-ring kernel/record IDs.
//
// Both kernel-side (#include from .cu files) and Rust-side decoder reference
// these symbols. Drift between them is checked by `build.rs` (Task 2).
//
// kernel_id (uint8) — which kernel emitted the record.
// record_type (uint8) — sub-type within the kernel (input/state/output/debug).
// payload layout — defined per (kernel_id, record_type) in decoder.

#ifndef GPU_LOG_IDS_H
#define GPU_LOG_IDS_H

#define GPU_LOG_N_KERNELS         8
#define GPU_LOG_N_RT_PER_KERNEL   4
#define GPU_LOG_N_RECORDS_PER_STEP (GPU_LOG_N_KERNELS * GPU_LOG_N_RT_PER_KERNEL)
#define GPU_LOG_N_SLOTS           32768
#define GPU_LOG_RECORD_BYTES      64
#define GPU_LOG_PAYLOAD_F32       12
#define GPU_LOG_MAGIC             0xF0F0A710u

// Kernel IDs (uint8). Add new kernels at the bottom; never reorder.
#define KID_SMOOTHNESS_CONTROLLER  0
#define KID_OUTPUT_SMOOTHNESS      1
#define KID_BCE_MULTI_HORIZON      2
#define KID_HORIZON_LAMBDA         3
#define KID_HEADS_GRN_FWD          4
#define KID_HEADS_GRN_BWD          5
#define KID_CFC_STEP               6
#define KID_RESERVED_7             7

// Record types within a kernel (uint8). Reuse RT_INPUT/STATE/OUTPUT/DEBUG
// across kernels — payload layout differs per (kid, rt) pair but the four
// roles are conventional.
#define RT_INPUT   0
#define RT_STATE   1
#define RT_OUTPUT  2
#define RT_DEBUG   3

#endif // GPU_LOG_IDS_H
```

- [ ] **1.2** Commit:
```bash
cd /home/jgrusewski/Work/foxhunt
git add crates/ml-alpha/cuda/gpu_log_ids.h
git commit -m "feat(gpu-log): add gpu_log_ids.h — single source of truth for ring IDs"
```

---

## Task 2: Device-side ring + tick kernel

- [ ] **2.1** Create `crates/ml-alpha/cuda/gpu_log_ring.cu`:

```c
// gpu_log_ring.cu — unified GPU diagnostic log ring (device-side).
//
// Provides `log_record()` (a device __forceinline__ helper) and the
// `gpu_log_tick` kernel that increments the step counter once per
// captured-graph step.
//
// Constraints:
//   - No atomicAdd (slot reservation by deterministic step/kid/rt math).
//   - No host branches; thread-id gating only.
//   - Header-last commit (magic field written after payload + other header).
//   - Mapped-pinned ring; host reads via volatile, no DtoH memcpy.
//
// Per `feedback_nvidia_grade_perf_for_kernels`: single-thread writer for
// the slot's 64 B cacheline, one coalesced store.

#include "gpu_log_ids.h"

#include <cstdint>

struct LogHeader {
    uint32_t magic;
    uint32_t step;
    uint8_t  kernel_id;
    uint8_t  record_type;
    uint8_t  payload_words;
    uint8_t  reserved;
    uint32_t _padding;
};

struct LogRecord {
    LogHeader header;
    float     payload[GPU_LOG_PAYLOAD_F32];
};

struct LogRing {
    LogRecord records[GPU_LOG_N_SLOTS];
};

// Tick kernel — 1 thread, 1 block. Increments step counter; runs FIRST in
// every captured-graph replay so all subsequent kernels see the new step.
extern "C" __global__ void gpu_log_tick(int* step_counter) {
    if (threadIdx.x == 0 && blockIdx.x == 0) {
        step_counter[0] = step_counter[0] + 1;
    }
}

// log_record — emit a diagnostic record from a kernel.
//
// Single-writer per (step, kernel_id, record_type) slot. Thread (0,0) of
// block 0 writes; all other threads no-op. Header-last commit ensures
// torn-record reads on the host side are detectable via magic mismatch.
//
// Caller responsibility:
//   - g_log_ring == nullptr → no-op (logging disabled).
//   - g_step_counter is the device step counter (incremented by gpu_log_tick).
//   - payload_words ≤ GPU_LOG_PAYLOAD_F32 (12); kernel must respect this.
__device__ __forceinline__ void log_record(
    LogRing*       g_log_ring,
    const int*     g_step_counter,
    uint8_t        kernel_id,
    uint8_t        record_type,
    const float*   payload,
    int            payload_words
) {
    if (g_log_ring == nullptr) return;
    if (blockIdx.x != 0 || threadIdx.x != 0) return;

    const int step = g_step_counter[0];
    const int slot = (step * GPU_LOG_N_RECORDS_PER_STEP
                    + (int)kernel_id * GPU_LOG_N_RT_PER_KERNEL
                    + (int)record_type) & (GPU_LOG_N_SLOTS - 1);
    LogRecord* rec = &g_log_ring->records[slot];

    const int n = payload_words < GPU_LOG_PAYLOAD_F32 ? payload_words : GPU_LOG_PAYLOAD_F32;
    #pragma unroll
    for (int i = 0; i < GPU_LOG_PAYLOAD_F32; ++i) {
        rec->payload[i] = (i < n) ? payload[i] : 0.0f;
    }
    rec->header.step          = (uint32_t)step;
    rec->header.kernel_id     = kernel_id;
    rec->header.record_type   = record_type;
    rec->header.payload_words = (uint8_t)n;
    rec->header.reserved      = 0;
    rec->header._padding      = 0;

    __threadfence();                       // make payload + header fields visible …
    rec->header.magic = GPU_LOG_MAGIC;     // … BEFORE the commit sentinel.
}
```

- [ ] **2.2** Register the kernel in `crates/ml-alpha/build.rs`. After `"smoothness_lambda_controller"` in the `KERNELS` array, add:
```rust
    "smoothness_lambda_controller",  // CRT.train: ISV-driven λ controller anchored on h30 jitter
    "gpu_log_ring",                  // GPU diagnostic log ring — tick kernel + log_record helper
];
```
Also bump the cache-bust comment from v13 → v14:
```rust
// Cache bust v14 (2026-05-21): gpu_log_ring.cu added — unified in-kernel diagnostic logging.
```

- [ ] **2.3** Verify:
```bash
SQLX_OFFLINE=true CARGO_FEATURE_CUDA=1 cargo build -p ml-alpha 2>&1 | grep -E "gpu_log|^error" | head
```
Expected: `compiled cuda/gpu_log_ring.cu -> .../gpu_log_ring.cubin`. No errors.

- [ ] **2.4** Commit:
```bash
git add crates/ml-alpha/cuda/gpu_log_ring.cu crates/ml-alpha/build.rs
git commit -m "feat(gpu-log): add gpu_log_ring.cu device helpers + tick kernel"
```

---

## Task 3: Cargo feature flag

- [ ] **3.1** In `crates/ml-alpha/Cargo.toml`, locate the `[features]` section (or add one if missing — grep first). Add the feature:
```toml
[features]
default = []
cuda = []
cuda-diag-log = []
```
(Adjust to merge with existing features. If `cuda` already exists with content, preserve it; only ADD `cuda-diag-log`.)

- [ ] **3.2** Verify:
```bash
SQLX_OFFLINE=true cargo build -p ml-alpha --features cuda-diag-log 2>&1 | tail -5
```
Expected: builds cleanly (feature flag declared, not yet referenced anywhere).

- [ ] **3.3** Commit:
```bash
git add crates/ml-alpha/Cargo.toml
git commit -m "feat(gpu-log): add cuda-diag-log Cargo feature flag"
```

---

## Task 4: Generic `MappedRecordBuffer<T>` in pinned_mem.rs

- [ ] **4.1** In `crates/ml-alpha/src/pinned_mem.rs`, locate the existing `MappedF32Buffer` impl. Extract its inner logic into a generic `MappedRecordBuffer<T: Copy>` and keep `MappedF32Buffer` as a type alias for backward compat. Concrete steps:

  a. Add the generic struct + impl above the existing F32 type:
  ```rust
  /// Mapped-pinned host buffer of `T` records. Allocated via cudaHostAlloc
  /// with the Mapped flag; both host and device see the same physical
  /// memory. Device pointer obtained via cudaHostGetDevicePointer.
  /// Per `feedback_no_htod_htoh_only_mapped_pinned`: the ONLY legitimate
  /// channel for cross-direction transfer in the captured-graph hot path.
  pub struct MappedRecordBuffer<T: Copy> {
      pub host_ptr: *mut T,
      pub dev_ptr: u64,
      pub len: usize,
  }
  
  impl<T: Copy> MappedRecordBuffer<T> {
      pub unsafe fn new(n: usize) -> Result<Self, String> {
          // … existing allocation logic from MappedF32Buffer::new, generalised …
      }
      pub fn write_record(&self, idx: usize, val: T) {
          assert!(idx < self.len);
          unsafe { std::ptr::write_volatile(self.host_ptr.add(idx), val); }
      }
      pub fn read_record(&self, idx: usize) -> T {
          assert!(idx < self.len);
          unsafe { std::ptr::read_volatile(self.host_ptr.add(idx)) }
      }
      pub fn read_all(&self) -> Vec<T> {
          (0..self.len).map(|i| self.read_record(i)).collect()
      }
  }
  
  impl<T: Copy> Drop for MappedRecordBuffer<T> {
      fn drop(&mut self) {
          // … existing cudaFreeHost logic …
      }
  }
  ```

  b. Make `MappedF32Buffer` a type alias:
  ```rust
  pub type MappedF32Buffer = MappedRecordBuffer<f32>;
  ```

  c. Preserve any F32-specific helpers (`write_from_slice(&[f32])`, etc.) by adding an `impl MappedRecordBuffer<f32>` block.

- [ ] **4.2** Verify nothing else broke:
```bash
SQLX_OFFLINE=true cargo check -p ml-alpha --all-targets 2>&1 | grep -E "^error" | head
```
Expected: no errors. Existing callers of `MappedF32Buffer::new()` still work via the type alias.

- [ ] **4.3** Commit:
```bash
git add crates/ml-alpha/src/pinned_mem.rs
git commit -m "refactor(gpu-log): generic MappedRecordBuffer<T>; MappedF32Buffer as alias"
```

---

## Task 5: Rust-side log ring + drain task (`gpu_log.rs`)

- [ ] **5.1** Create `crates/ml-alpha/src/gpu_log.rs`:

```rust
//! GPU diagnostic log ring — host-side allocator, drainer, decoder registry.
//!
//! See `docs/superpowers/specs/2026-05-21-gpu-log-ring-design.md`.
//!
//! Behaviour: when the `cuda-diag-log` feature is enabled, the trainer
//! allocates a mapped-pinned ring buffer + a 1-element device step counter,
//! passes their pointers into logging-capable kernels, and starts a tokio
//! drain task that polls the ring at 500 ms cadence and emits structured
//! tracing events.

use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::Duration;
use anyhow::{Context, Result};
use cudarc::driver::{CudaSlice, CudaStream};
use tokio::task::JoinHandle;
use tracing::{info, warn, debug};

use crate::pinned_mem::MappedRecordBuffer;

pub const N_KERNELS:           usize = 8;
pub const N_RT_PER_KERNEL:     usize = 4;
pub const N_RECORDS_PER_STEP:  usize = N_KERNELS * N_RT_PER_KERNEL;
pub const N_SLOTS:             usize = 32768;
pub const PAYLOAD_F32:         usize = 12;
pub const MAGIC:               u32   = 0xF0F0_A710;

#[repr(C)]
#[derive(Clone, Copy, Debug)]
pub struct LogHeader {
    pub magic: u32,
    pub step: u32,
    pub kernel_id: u8,
    pub record_type: u8,
    pub payload_words: u8,
    pub reserved: u8,
    pub _padding: u32,
}

#[repr(C)]
#[derive(Clone, Copy)]
pub struct LogRecord {
    pub header: LogHeader,
    pub payload: [f32; PAYLOAD_F32],
}

impl Default for LogRecord {
    fn default() -> Self {
        Self {
            header: LogHeader {
                magic: 0, step: 0,
                kernel_id: 0, record_type: 0, payload_words: 0, reserved: 0,
                _padding: 0,
            },
            payload: [0.0; PAYLOAD_F32],
        }
    }
}

pub struct LogRing {
    pub buffer: MappedRecordBuffer<LogRecord>,
}

impl LogRing {
    pub fn alloc() -> Result<Self> {
        let buffer = unsafe { MappedRecordBuffer::<LogRecord>::new(N_SLOTS) }
            .map_err(|e| anyhow::anyhow!("log ring alloc: {e}"))?;
        // Zero-initialise so magic is 0 everywhere (no false-positive reads).
        let zero = LogRecord::default();
        for i in 0..N_SLOTS {
            buffer.write_record(i, zero);
        }
        Ok(Self { buffer })
    }

    /// Device pointer to the ring (for kernel arg).
    pub fn dev_ptr(&self) -> u64 {
        self.buffer.dev_ptr
    }
}

/// Allocate the step counter as a device buffer initialised to 0.
pub fn alloc_step_counter(stream: &Arc<CudaStream>) -> Result<CudaSlice<i32>> {
    stream.alloc_zeros::<i32>(1).context("step_counter alloc")
}

/// Type for a per-(kernel_id, record_type) decoder function.
pub type DecoderFn = fn(step: u32, payload: &[f32]);

/// Per-(kernel_id, record_type) decoder registry. Default is the
/// generic debug-print decoder. Specific decoders override entries.
fn default_decoder(step: u32, payload: &[f32]) {
    debug!(target: "gpu_log", step, ?payload, "unknown record");
}

fn smoothness_input(step: u32, payload: &[f32]) {
    if payload.len() >= 10 {
        info!(target: "gpu_log",
            step,
            raw_h30=payload[0], raw_h100=payload[1], raw_h300=payload[2],
            raw_h1000=payload[3], raw_h6000=payload[4],
            jitter_in_h30=payload[5], jitter_in_h100=payload[6],
            jitter_in_h300=payload[7], jitter_in_h1000=payload[8],
            jitter_in_h6000=payload[9],
            "smoothness_ctrl RT_INPUT");
    }
}

fn smoothness_state(step: u32, payload: &[f32]) {
    if payload.len() >= 10 {
        info!(target: "gpu_log",
            step,
            ema_h30=payload[0], ema_h100=payload[1], ema_h300=payload[2],
            ema_h1000=payload[3], ema_h6000=payload[4],
            target_h30=payload[5], target_h100=payload[6], target_h300=payload[7],
            target_h1000=payload[8], target_h6000=payload[9],
            "smoothness_ctrl RT_STATE");
    }
}

fn smoothness_output(step: u32, payload: &[f32]) {
    if payload.len() >= 10 {
        info!(target: "gpu_log",
            step,
            excess_h30=payload[0], excess_h100=payload[1], excess_h300=payload[2],
            excess_h1000=payload[3], excess_h6000=payload[4],
            lambda_h30=payload[5], lambda_h100=payload[6], lambda_h300=payload[7],
            lambda_h1000=payload[8], lambda_h6000=payload[9],
            "smoothness_ctrl RT_OUTPUT");
    }
}

/// Static dispatch table indexed by [kernel_id][record_type].
/// Add entries here as new kernels opt into the ring.
fn decode(kernel_id: u8, record_type: u8, step: u32, payload: &[f32]) {
    match (kernel_id, record_type) {
        (0, 0) => smoothness_input(step, payload),    // KID_SMOOTHNESS_CONTROLLER, RT_INPUT
        (0, 1) => smoothness_state(step, payload),
        (0, 2) => smoothness_output(step, payload),
        _      => default_decoder(step, payload),
    }
}

/// Spawn the drain task. Polls the ring at 500 ms cadence; for each new
/// step in [last_drained .. step_counter], walks N_RECORDS_PER_STEP slots,
/// validates magic + step match, dispatches to decoder.
/// Returns a JoinHandle + a stop signal.
pub fn spawn_drain_task(
    ring: Arc<LogRing>,
    step_counter_host_shadow: Arc<MappedRecordBuffer<i32>>,
    stop: Arc<AtomicBool>,
) -> JoinHandle<()> {
    tokio::spawn(async move {
        let mut last_drained_step: u32 = 0;
        let interval = Duration::from_millis(500);
        loop {
            if stop.load(Ordering::Relaxed) { break; }
            tokio::time::sleep(interval).await;

            // Read device step counter via mapped-pinned shadow.
            let current_step = step_counter_host_shadow.read_record(0) as u32;
            if current_step <= last_drained_step { continue; }

            let max_drain = (N_SLOTS / N_RECORDS_PER_STEP) as u32;   // step window
            let oldest_safe_step = current_step.saturating_sub(max_drain);
            if last_drained_step < oldest_safe_step {
                warn!(target: "gpu_log",
                    last_drained_step, current_step,
                    skipped=oldest_safe_step.saturating_sub(last_drained_step),
                    "drainer fell behind — records dropped");
                last_drained_step = oldest_safe_step;
            }

            for step in (last_drained_step + 1)..=current_step {
                for kid in 0..N_KERNELS as u8 {
                    for rt in 0..N_RT_PER_KERNEL as u8 {
                        let slot = ((step as usize) * N_RECORDS_PER_STEP
                                  + (kid as usize) * N_RT_PER_KERNEL
                                  + rt as usize) & (N_SLOTS - 1);
                        let rec = ring.buffer.read_record(slot);
                        if rec.header.magic != MAGIC { continue; }
                        if rec.header.step != step { continue; }
                        let n = rec.header.payload_words as usize;
                        let n = n.min(PAYLOAD_F32);
                        decode(rec.header.kernel_id, rec.header.record_type,
                               rec.header.step, &rec.payload[..n]);
                    }
                }
            }
            last_drained_step = current_step;
        }
    })
}
```

- [ ] **5.2** Expose the module in `crates/ml-alpha/src/lib.rs`. Add (preserving existing `pub mod` style):
```rust
#[cfg(feature = "cuda-diag-log")]
pub mod gpu_log;
```

- [ ] **5.3** Verify:
```bash
SQLX_OFFLINE=true cargo build -p ml-alpha --features cuda-diag-log 2>&1 | tail -10
SQLX_OFFLINE=true cargo check -p ml-alpha --all-targets 2>&1 | grep -E "^error" | head
```
Expected: both clean.

- [ ] **5.4** Commit:
```bash
git add crates/ml-alpha/src/gpu_log.rs crates/ml-alpha/src/lib.rs
git commit -m "feat(gpu-log): Rust-side LogRing, drain task, decoder registry"
```

---

## Task 6: Invariant tests

- [ ] **6.1** Create `crates/ml-alpha/tests/gpu_log_ring_invariants.rs`. The tests don't depend on a real kernel writing records — they exercise the Rust-side allocator and the host-visible round-trip by writing records FROM the host to the mapped-pinned buffer and verifying the layout matches what a kernel would write (`#[repr(C)]` ensures binary compatibility).

Six unit tests:

1. `ring_alloc_zeros_magic` — alloc ring, every slot's magic = 0.
2. `host_write_round_trip` — host writes a record at a slot, reads it back, all fields match (asserts `#[repr(C)]` layout).
3. `magic_validation_skips_torn` — write record with magic=0; drainer ignores. Write record with magic=MAGIC and step mismatch; drainer also ignores. Write record with magic=MAGIC and matching step; drainer reads.
4. `ring_wrap_modulo_arithmetic` — for step values N_SLOTS/N_RECORDS_PER_STEP and beyond, slot indices wrap to start; verify no out-of-bounds.
5. `step_gap_detection` — emit records at step 100 and step 200 with gap of 100 between; verify drainer detects the gap and emits warn.
6. `decoder_dispatch_unknown_falls_through` — write a record with kernel_id=99 (unregistered); verify no panic, default decoder hit.

Test 7 (integration, gated by `--features cuda-diag-log` AND CUDA present):

7. `kernel_writes_visible_to_host` — launch the gpu_log_tick kernel + a tiny test kernel that calls log_record() with a known payload; drain; verify host sees the record.

Use `tokio::test` for any test exercising the drain task. Use `serial_test` if multiple tests share the ring (otherwise allocate fresh per test).

- [ ] **6.2** Build + run:
```bash
SQLX_OFFLINE=true cargo test -p ml-alpha --features cuda-diag-log --test gpu_log_ring_invariants -- --nocapture 2>&1 | tail -30
```
Expected: 6 unit tests pass; test 7 passes if CUDA available, skips otherwise.

- [ ] **6.3** Commit:
```bash
git add crates/ml-alpha/tests/gpu_log_ring_invariants.rs
git commit -m "test(gpu-log): invariant tests for ring allocator + drainer"
```

---

## Task 7: Wire smoothness_lambda_controller as the first user

This is the ATOMIC refactor (per `feedback_no_partial_refactor`): the kernel signature changes; the trainer launch changes; the controller emits records. Single commit covers all three.

- [ ] **7.1** Modify `crates/ml-alpha/cuda/smoothness_lambda_controller.cu`:

  a. Add `#include "gpu_log_ids.h"` and a forward declaration of `log_record()` (or factor `log_record` into a separate `gpu_log_helpers.cuh` header and include it). Decide once: header-only inclusion is simpler — extract `log_record` and the `LogRecord`/`LogRing`/`LogHeader` structs into `cuda/gpu_log_helpers.cuh`, then `gpu_log_ring.cu` includes the header to define the tick kernel; smoothness_lambda_controller.cu ALSO includes the header to use `log_record`.

  b. Add two new args to the kernel signature:
  ```c
  extern "C" __global__ void smoothness_lambda_controller(
      const float* __restrict__ raw_per_h,
      float* __restrict__ jitter_ema,
      int*   __restrict__ first_obs,
      float  base_lambda,
      float* __restrict__ lambda_out,
      LogRing*   g_log_ring,             // NEW (nullable)
      const int* g_step_counter           // NEW
  )
  ```

  c. After the lambda update at end of kernel, emit three records (RT_INPUT, RT_STATE, RT_OUTPUT). Thread 0 of the single block reads the relevant arrays into stack variables for the payload, calls `log_record()` three times:
  ```c
      if (h == 0) {
          // RT_INPUT: raw_per_h + jitter_ema_in (10 floats)
          float in_payload[10] = {
              raw_per_h[0], raw_per_h[1], raw_per_h[2], raw_per_h[3], raw_per_h[4],
              /* jitter_ema_in was the prior value — but we've already overwritten it.
                 Capture BEFORE the EMA update via a temp register. */
              jitter_in_h30, jitter_in_h100, jitter_in_h300, jitter_in_h1000, jitter_in_h6000,
          };
          log_record(g_log_ring, g_step_counter, KID_SMOOTHNESS_CONTROLLER, RT_INPUT, in_payload, 10);

          // RT_STATE: jitter_ema_out + target (10 floats)
          float state_payload[10] = {
              s_jitter_after[0], s_jitter_after[1], s_jitter_after[2], s_jitter_after[3], s_jitter_after[4],
              target_0, target_1, target_2, target_3, target_4,
          };
          log_record(g_log_ring, g_step_counter, KID_SMOOTHNESS_CONTROLLER, RT_STATE, state_payload, 10);

          // RT_OUTPUT: excess_ratio + lambda_out (10 floats)
          float out_payload[10] = {
              excess_0, excess_1, excess_2, excess_3, excess_4,
              lambda_out[0], lambda_out[1], lambda_out[2], lambda_out[3], lambda_out[4],
          };
          log_record(g_log_ring, g_step_counter, KID_SMOOTHNESS_CONTROLLER, RT_OUTPUT, out_payload, 10);
      }
  ```
  d. The kernel's per-thread excess_ratio + target are computed per-h; thread (h=0) needs to gather all 5. Use a shared array `__shared__ float s_target[5]; __shared__ float s_excess[5];` populated by each thread (per-(h) writes its own slot), barrier, then thread 0 reads.

- [ ] **7.2** Modify `crates/ml-alpha/src/trainer/perception.rs`:

  a. Behind `#[cfg(feature = "cuda-diag-log")]`, add new fields to `PerceptionTrainer`:
  ```rust
      #[cfg(feature = "cuda-diag-log")]
      log_ring: std::sync::Arc<crate::gpu_log::LogRing>,
      #[cfg(feature = "cuda-diag-log")]
      log_step_counter_d: CudaSlice<i32>,
      #[cfg(feature = "cuda-diag-log")]
      log_step_counter_host_shadow: std::sync::Arc<crate::pinned_mem::MappedRecordBuffer<i32>>,
      #[cfg(feature = "cuda-diag-log")]
      gpu_log_tick_fn: CudaFunction,
      #[cfg(feature = "cuda-diag-log")]
      _gpu_log_module: std::sync::Arc<cudarc::driver::CudaModule>,
      #[cfg(feature = "cuda-diag-log")]
      log_drain_stop: std::sync::Arc<std::sync::atomic::AtomicBool>,
      #[cfg(feature = "cuda-diag-log")]
      log_drain_handle: Option<tokio::task::JoinHandle<()>>,
  ```

  b. In the constructor (behind `#[cfg(feature = "cuda-diag-log")]`): allocate the ring, step counter, host shadow; load the `gpu_log_tick` cubin and function; spawn drain task.

  c. In `step_batched`: BEFORE the existing kernel launches in the captured graph, launch `gpu_log_tick`:
  ```rust
  #[cfg(feature = "cuda-diag-log")]
  {
      let tick_cfg = LaunchConfig {
          grid_dim: (1, 1, 1),
          block_dim: (1, 1, 1),
          shared_mem_bytes: 0,
      };
      let mut launch = self.stream.launch_builder(&self.gpu_log_tick_fn);
      launch.arg(&mut self.log_step_counter_d);
      unsafe { launch.launch(tick_cfg).context("gpu_log_tick launch")?; }
  }
  ```

  d. In the smoothness_lambda_controller launch (the existing block from Task 4 of the predecessor plan), add two more args at the end:
  ```rust
  launch
      .arg(&self.smoothness_loss_per_horizon_d)
      .arg(&mut self.smoothness_jitter_ema_d)
      .arg(&mut self.smoothness_jitter_first_obs_d)
      .arg(&base_lambda)
      .arg(&mut self.smoothness_lambda_d)
      // NEW (cfg-gated):
      .arg(&log_ring_dev_ptr)         // u64 device pointer; null when feature off
      .arg(&log_step_counter_dev_ptr);
  ```
  When feature is off, pass null pointers (the kernel branches on `g_log_ring == nullptr`). When feature is on, pass `self.log_ring.dev_ptr()` and the step counter pointer.

  Since the launch args change between feature-on and feature-off builds, this is a `#[cfg]`-gated build of the launch block. The cleanest approach: keep the call site uniform, but conditionally compute the pointer values. Concrete:
  ```rust
  #[cfg(feature = "cuda-diag-log")]
  let log_ring_ptr: u64 = self.log_ring.dev_ptr();
  #[cfg(not(feature = "cuda-diag-log"))]
  let log_ring_ptr: u64 = 0;
  
  #[cfg(feature = "cuda-diag-log")]
  let log_step_ptr: u64 = {
      let (p, _g) = self.log_step_counter_d.device_ptr(&self.stream);
      p
  };
  #[cfg(not(feature = "cuda-diag-log"))]
  let log_step_ptr: u64 = 0;
  ```
  And the kernel signature ALWAYS has the two args (kernel always takes them; checks for null at top). So the kernel signature change is permanent; the values are conditional.

  e. In drop / shutdown: set `log_drain_stop` and await the drain handle.

- [ ] **7.3** Verify both builds:
```bash
SQLX_OFFLINE=true cargo build -p ml-alpha 2>&1 | tail -5                   # feature off
SQLX_OFFLINE=true cargo build -p ml-alpha --features cuda-diag-log 2>&1 | tail -5   # feature on
SQLX_OFFLINE=true cargo check -p ml-alpha --all-targets 2>&1 | grep -E "^error" | head
```
Expected: all three clean.

Also run the existing controller invariant test (since the kernel signature changed, the test must also pass the two new args — likely null pointers for feature-off):
```bash
SQLX_OFFLINE=true cargo test -p ml-alpha --test smoothness_lambda_controller_invariants -- --nocapture 2>&1 | tail
```
Expected: 4/4 pass. If the test breaks because the kernel signature changed, update the test in this same commit to pass null pointers (atomic refactor).

- [ ] **7.4** Commit (atomic):
```bash
git add crates/ml-alpha/cuda/smoothness_lambda_controller.cu \
        crates/ml-alpha/cuda/gpu_log_helpers.cuh \
        crates/ml-alpha/src/trainer/perception.rs \
        crates/ml-alpha/tests/smoothness_lambda_controller_invariants.rs
git commit -m "feat(gpu-log): wire smoothness_lambda_controller as first ring producer"
```

---

## Task 8: Push + smoke + retrain decision

- [ ] **8.1** Push:
```bash
git push origin ml-alpha-phase-a
```

- [ ] **8.2** Local CUDA smoke (RTX 3050 Ti):
```bash
SQLX_OFFLINE=true cargo test -p ml-alpha --features cuda-diag-log --test gpu_log_ring_invariants -- --nocapture 2>&1 | tail -30
FOXHUNT_TEST_DATA=test_data/futures-baseline cargo test -p ml-alpha --features cuda-diag-log --lib -- smoke_tests --ignored --nocapture 2>&1 | tail -30
```
Expected: ring tests pass; smoke runs without panic; tracing emits structured `gpu_log` records.

- [ ] **8.3** Decision gate based on CRT.diag (`lob-backtest-sweep-sqkst`) WIN-gate result:
  - **WIN passed at base_lambda=0.001**: the log ring is valuable infra but not blocking; leave the cuda-diag-log feature off by default in CI; revisit if a subsequent controller needs diagnostics.
  - **WIN failed at base_lambda=0.001**: enable `cuda-diag-log` for the next retrain at base_lambda=0.01-0.02 and use the per-step `lambda_h6000` trajectory to confirm mechanism #1 (amplitude underflow) vs mechanism #2 (shared trunk leakage).

  Update the project memory file `project_crt_diag_findings.md` with the gate outcome and the diagnostic plan.

- [ ] **8.4** (Conditional) Submit Argo retrain at base_lambda=0.01 with diagnostic logging:
```bash
argo submit -n foxhunt --from=wftmpl/alpha-perception \
  -p commit-sha=$(git rev-parse HEAD) \
  -p git-branch=ml-alpha-phase-a \
  -p gpu-pool=ci-training-l40s \
  -p smoothness-base-lambda=0.01 \
  -p n-train-seqs=8000 -p n-val-seqs=1000 -p epochs=5
# Note: the Argo template doesn't currently expose --features. To enable
# cuda-diag-log on the cluster build, add a workflow parameter
# `cargo-features` and thread it into `cargo build --release --features
# "$CARGO_FEATURES" -p ml-alpha --example alpha_train` in the
# ensure-binary step. Task 9 below covers that wiring if needed.
```

---

## Task 9: (Optional) Plumb feature flag through Argo template

Only execute if Task 8.3 decides we need diagnostics on the cluster (WIN failed branch).

- [ ] **9.1** In `infra/k8s/argo/alpha-perception-template.yaml`, add a workflow parameter:
```yaml
      - name: cargo-features
        value: ""
```

- [ ] **9.2** In the `ensure-binary` step's cargo build invocation:
```yaml
            cargo build --release \
              ${CARGO_FEATURES:+--features "$CARGO_FEATURES"} \
              -p ml-alpha --example alpha_train
```
Where `CARGO_FEATURES="{{workflow.parameters.cargo-features}}"` is set from the parameter.

- [ ] **9.3** Commit + push:
```bash
git add infra/k8s/argo/alpha-perception-template.yaml
git commit -m "feat(gpu-log): plumb --features through alpha-perception template"
git push origin ml-alpha-phase-a
```

- [ ] **9.4** Submit the retrain with diagnostics enabled:
```bash
argo submit -n foxhunt --from=wftmpl/alpha-perception \
  -p commit-sha=$(git rev-parse HEAD) \
  -p git-branch=ml-alpha-phase-a \
  -p gpu-pool=ci-training-l40s \
  -p smoothness-base-lambda=0.01 \
  -p cargo-features=cuda-diag-log \
  -p n-train-seqs=8000 -p n-val-seqs=1000 -p epochs=5
```

---

## Self-Review

1. **Spec coverage:**
   - §3.1 (record format) → Task 1 (header) + Task 2 (struct in .cu) ✓
   - §3.2 (slot reservation) → Task 2 (`log_record`) ✓
   - §3.3 (header-last commit) → Task 2 ✓
   - §3.4 (mapped-pinned host visibility + drain) → Task 4 (MappedRecordBuffer) + Task 5 (drain task) ✓
   - §3.5 (kernel-author API) → Task 1 (header) + Task 7 (first user) ✓
   - §3.6 (Rust decoder) → Task 5 ✓
   - §3.7 (wiring lifecycle) → Task 7 ✓
   - §4 (memory rule compliance) → enforced throughout; verified per-task ✓
   - §5 (validation gates) → Task 6 (6 unit tests + 1 integration) ✓
2. **No placeholders.** All snippets are complete code; no TODOs in shipping content. ✓
3. **Type consistency.** `LogRecord`, `LogHeader`, `LogRing` named identically in Tasks 1-5. `MappedRecordBuffer<T>` named identically in Tasks 4-5. ✓
4. **Atomic-refactor compliance.** Task 7 ships kernel sig change + trainer wiring + test update in ONE commit. ✓
5. **Memory rule citations.**
   - `feedback_no_atomicadd` → Task 2 (slot reservation by deterministic math) ✓
   - `feedback_no_htod_htoh_only_mapped_pinned` → Tasks 4, 5 (mapped-pinned ring + step counter shadow) ✓
   - `feedback_no_nvrtc` → Task 2 (build.rs registration) ✓
   - `pearl_no_host_branches_in_captured_graph` → Task 7 (only device pointer args change between captures) ✓
   - `pearl_cudarc_disable_event_tracking_for_graph_capture` → Tasks 5, 7 (no host malloc inside captured graph; ring + step counter pre-allocated at construction) ✓
   - `feedback_no_stubs`, `feedback_no_todo_fixme`, `feedback_no_hiding` → enforced ✓
   - `feedback_no_partial_refactor` → Task 7 single commit ✓
   - `feedback_single_source_of_truth_no_duplicates` → Task 1 single header ✓
   - `feedback_wire_everything_up` → Tasks 7-8 wire allocate → tick → record → drain → tracing ✓
   - `pearl_build_rs_rerun_if_env_changed` → no new env reads ✓

Ready to execute when CRT.diag's WIN-gate decision lands.
