# GPU Log Ring — Unified High-Performance In-Kernel Diagnostic Logging (Design)

**Date:** 2026-05-21
**Branch context:** `ml-alpha-phase-a` (HEAD `0f664cf36`; ISV-driven λ controller shipped but per-step trajectory is invisible — `final_smooth_loss_per_horizon` is the only smoothness telemetry, emitted once at end-of-run)
**Predecessor:** `2026-05-21-crt-train-isv-driven-lambda-controller.md` (the controller whose internal state we cannot inspect)
**Status:** Design — pending plan + execution

---

## 1. Motivation

ml-alpha kernels run inside a captured CUDA graph at ~370 steps/sec on L40S. To diagnose why a controller is or is not converging, we need per-step visibility into its inputs, state, and outputs. Today the only available telemetry channels are:

- `tracing::info!` from Rust, which can only see what the Rust side already computes (per-epoch validation outputs, the existing `isv snapshot` log line driven by `horizon_lambda.cu`). Anything that lives only inside a kernel — like `smoothness_lambda_controller`'s `jitter_ema`, `target`, `excess_ratio`, `lambda_out` — is invisible without a DtoH copy after every step.
- Per-step DtoH copy of every diagnostic buffer: forbidden by [feedback_no_htod_htoh_only_mapped_pinned](memory/feedback_no_htod_htoh_only_mapped_pinned.md), [feedback_cpu_is_read_only](memory/feedback_cpu_is_read_only.md), and adds a synchronisation barrier in the hot path.
- Per-step `printf` from the kernel: violates [pearl_no_host_branches_in_captured_graph](memory/pearl_no_host_branches_in_captured_graph.md) and stalls every block on a serialised CPU queue.

The empirical motivator: at base_lambda=0.001 the smoothness regularizer produced +6.5pt mean AUC but `final_smooth_loss_per_horizon = [0.082, 0.079, 0.075, 0.082, 0.076]` — no horizon differentiation. We cannot tell whether (a) the controller never amplified λ at all, (b) λ amplified but oscillated, (c) λ amplified steadily but the gradient was overpowered by BCE, or (d) something else. The right tool is per-step λ + jitter_ema + target trajectory. The right channel is a unified GPU log ring readable by host without DtoH copy.

## 2. Hypothesis

A mapped-pinned ring buffer with deterministic slot reservation (no `atomicAdd`), header-last commit ordering, and step-counter-derived addressing yields:
- One coalesced 64 B write per logging event from the kernel hot path
- Zero DtoH copies, zero kernel-host synchronisation
- Full per-step trajectory for every kernel that opts in
- Drop-detection on the host side (step-counter gaps in successive valid records)

This solves the diagnostic gap without violating the GPU/CPU contract memory rules.

## 3. Architectural Intervention

### 3.1 Record format (64 B, cacheline-aligned)

```c
struct LogHeader {           // 16 B
    uint32_t magic;          // 4 — 0xF0XHN710; commit sentinel, written LAST
    uint32_t step;           // 4 — monotonic step counter
    uint8_t  kernel_id;      // 1 — kernel emitting (see gpu_log_ids.h)
    uint8_t  record_type;    // 1 — sub-type within the kernel
    uint8_t  payload_words;  // 1 — # of f32 in payload (≤ 12)
    uint8_t  reserved;       // 1
    uint32_t _padding;       // 4
};

struct LogRecord {           // 64 B
    LogHeader header;        // 16 B
    float     payload[12];   // 48 B = 12 × f32
};
```

Total: 64 B = one cacheline; single coalesced store per emission.

The `magic` field is the **commit sentinel**: written last, after the payload and the other header fields. Host readers check `magic == LOG_MAGIC` and `step == expected_step_for_this_slot` to validate. A torn record (kernel mid-write while CPU reads) reads either the OLD magic (skip) or matches the new magic but not the expected step (skip).

### 3.2 Slot reservation without atomicAdd

A 1-thread tick kernel runs ONCE per training step (first in the captured graph, before any logging kernel) and increments a device counter:

```c
extern "C" __global__ void gpu_log_tick(int* step_counter) {
    if (threadIdx.x == 0 && blockIdx.x == 0) {
        step_counter[0] += 1;
    }
}
```

Single thread, single block, no atomics needed (sole writer per launch). Lives at the head of the captured graph; trainer never touches it from host.

Each logging kernel then derives its slot at emission time:
```
slot = (step * N_RECORDS_PER_STEP + kernel_id * N_RT_PER_KERNEL + record_type) & (N_SLOTS - 1)
```

with N_KERNELS=8, N_RT_PER_KERNEL=4, N_RECORDS_PER_STEP=32, N_SLOTS=32768 (all power-of-2). Modulo becomes a bitwise AND. Each (step, kernel_id, record_type) maps to exactly ONE slot — no contention, no atomics, no allocation.

Step-wrap window: 32768 / 32 = 1024 steps ≈ 2.8 s at 370 steps/sec. Drain cadence 500 ms keeps ring at < 20% utilisation.

### 3.3 Single-writer commit (header-last)

```c
__device__ __forceinline__ void log_record(
    LogRing*       ring,
    const int*     step_counter,
    uint8_t        kernel_id,
    uint8_t        record_type,
    const float*   payload,
    int            payload_words
) {
    // Thread (0,0) of block 0 only — single writer per (step, kernel_id, record_type).
    if (blockIdx.x | threadIdx.x) return;
    const int step = step_counter[0];
    const int slot = (step * N_RECORDS_PER_STEP
                    + ((int)kernel_id) * N_RT_PER_KERNEL
                    + (int)record_type) & (N_SLOTS - 1);
    LogRecord* rec = &ring->records[slot];

    // Payload + non-magic header fields first.
    #pragma unroll
    for (int i = 0; i < payload_words; ++i) rec->payload[i] = payload[i];
    rec->header.step          = (uint32_t)step;
    rec->header.kernel_id     = kernel_id;
    rec->header.record_type   = record_type;
    rec->header.payload_words = (uint8_t)payload_words;
    rec->header.reserved      = 0;
    rec->header._padding      = 0;

    __threadfence();                    // make all the above globally visible …
    rec->header.magic = LOG_MAGIC;      // … BEFORE the magic commit sentinel.
}
```

### 3.4 Mapped-pinned host visibility + drain

The ring is allocated via a `MappedF32Buffer`-style mapped-pinned host allocation (extended to support arbitrary record sizes; see §8). Device gets a device pointer to the same memory via `cudaHostGetDevicePointer`. Host sees writes via volatile reads — no DtoH memcpy, no synchronisation. Per [feedback_no_htod_htoh_only_mapped_pinned](memory/feedback_no_htod_htoh_only_mapped_pinned.md): only legitimate cross-direction channel.

CPU drainer is a tokio task in `alpha_train.rs` (or the trainer's `step_batched` driver, depending on lifetime). It:
1. Reads the device-side `step_counter[0]` via mapped-pinned (one volatile u32 read).
2. For each slot in `[last_drained_step .. step_counter] × N_RECORDS_PER_STEP`, reads `record.header.magic` — if not LOG_MAGIC, skip silently. If LOG_MAGIC and `header.step == expected_step_for_slot`, dispatch to the per-(kernel_id, record_type) decoder.
3. Decoder emits a structured `tracing::info!` line with the payload fields named per the record type.

Drop detection: if the gap between successive valid records' `step` fields exceeds N_SLOTS/N_RECORDS_PER_STEP, the drainer fell behind. Emit a `tracing::warn!` with the gap size; downstream analysis sees the discontinuity.

### 3.5 Kernel-author API

A single header `cuda/gpu_log_ids.h` defines the kernel_id and record_type enums; both the kernels and the Rust decoder include/reference it.

```c
// gpu_log_ids.h
#define KID_SMOOTHNESS_CONTROLLER  0
#define KID_OUTPUT_SMOOTHNESS      1
#define KID_BCE                    2
#define KID_HORIZON_LAMBDA         3
#define KID_HEADS_GRN_FWD          4
#define KID_HEADS_GRN_BWD          5
#define KID_CFC_STEP               6
#define KID_RESERVED_7             7

#define RT_INPUT   0
#define RT_STATE   1
#define RT_OUTPUT  2
#define RT_DEBUG   3
```

In a kernel:
```c
#include "gpu_log_ids.h"

extern "C" __global__ void smoothness_lambda_controller(
    /* … existing args … */,
    LogRing* g_log_ring,           // NEW: optional, NULL = no logging
    const int* g_step_counter       // NEW
) {
    /* … existing logic … */

    if (g_log_ring != nullptr) {
        float payload[10] = { jitter_ema[0], jitter_ema[1], jitter_ema[2], jitter_ema[3], jitter_ema[4],
                              lambda_out[0], lambda_out[1], lambda_out[2], lambda_out[3], lambda_out[4] };
        log_record(g_log_ring, g_step_counter, KID_SMOOTHNESS_CONTROLLER, RT_STATE, payload, 10);
    }
}
```

The two new args (`g_log_ring`, `g_step_counter`) are device pointers, stable across captures. When logging is disabled at construction time, the trainer passes NULL for `g_log_ring` and the kernel skips the emission — no diagnostic overhead.

### 3.6 Decoder side (Rust)

A central `gpu_log.rs` defines:
- `pub fn alloc_log_ring(n_slots: usize) -> Result<(MappedBuffer<LogRecord>, CudaSlice<i32>)>` — returns mapped-pinned ring + the device step_counter.
- `pub fn drain_loop(ring: &MappedBuffer<LogRecord>, step_counter: &CudaSlice<i32>, ...) -> JoinHandle<()>` — tokio task that runs until cancelled, periodically draining new records and dispatching to decoders.
- Decoders are registered via a const table indexed by `(kernel_id, record_type)`:
  ```rust
  type DecoderFn = fn(step: u32, payload: &[f32]);
  static DECODERS: [[DecoderFn; N_RT_PER_KERNEL]; N_KERNELS] = [...];
  ```
- The default decoder (no entry registered) emits a generic `tracing::debug!("unknown log record kernel_id={} record_type={}", ...)`.

### 3.7 Wiring lifecycle

- Trainer construction: alloc log ring + step counter; pass device pointers into every kernel handle that opts in.
- Trainer step: tick kernel runs first in the captured graph (incrementing step_counter); all logging-capable kernels then derive their slots from it.
- Trainer drop: drain task cancelled, ring buffer dropped (pinned memory unmapped).

Compile-time gating: a Rust feature flag `cuda-diag-log` enables ring allocation + drain task. When disabled, the trainer passes NULL pointers and the drain task never starts — zero cost in production training runs.

## 4. Memory-Rule Compliance

| Rule | Compliance |
|------|------------|
| `feedback_no_atomicadd` | Slot reservation by `step × N_RPS + kernel_id × N_RT + record_type` modulo — single-writer per slot. No atomics. |
| `feedback_no_htod_htoh_only_mapped_pinned` | Ring is mapped-pinned; host reads via volatile, no DtoH memcpy. |
| `feedback_no_nvrtc` | Tick kernel + log helpers pre-compiled via build.rs. |
| `pearl_no_host_branches_in_captured_graph` | Kernels' new args are device pointers, stable across captures. No host branching. |
| `pearl_cudarc_disable_event_tracking_for_graph_capture` | No host malloc, no scalar arg changes between captures, no pointer swaps. |
| `pearl_first_observation_bootstrap` | N/A (no EMA state in the ring) |
| `pearl_one_unbounded_signal_per_reward` | N/A (diagnostic, not loss) |
| `feedback_no_stubs`, `feedback_no_todo_fixme`, `feedback_no_hiding` | Implementation must complete every site or delete; no markers; no `_` hiding. |
| `feedback_no_partial_refactor` | When a kernel signature gains the two new pointer args, every consumer must migrate atomically. |
| `feedback_single_source_of_truth_no_duplicates` | One ring per training process. One header file (`gpu_log_ids.h`) is the source of truth for IDs. |
| `feedback_wire_everything_up` | Ring is allocated, tick fires, decoders dispatch, drain task runs — no orphans. |

## 5. Validation Gate

Unit tests (no training run needed):

1. **Round-trip**: emit a known record from a test kernel; drain; verify decoder reconstructs payload exactly. Asserts no off-by-one in slot arithmetic, no payload truncation, no endianness surprise.
2. **Magic validation**: write a deliberately torn record (payload only, no magic); verify drainer skips it without panic.
3. **Step monotonicity**: emit records at steps 0, 1, 2, …; verify drainer sees monotonic step counter without gaps.
4. **Ring wrap**: emit > N_SLOTS records; verify drain catches all (assuming fast enough) and reports wrap discontinuity correctly when slow.
5. **NULL ring**: launch kernel with NULL ring pointer; verify no crash, no record emitted.
6. **Multi-kernel coexistence**: emit from 3 different kernel_ids simultaneously in one step; verify all 3 records distinct and decodable.

Integration test:

7. **Live smoothness trajectory**: enable cuda-diag-log feature, run 100 trainer steps, drain, verify per-step records exist for smoothness_lambda_controller (RT_INPUT, RT_STATE, RT_OUTPUT) showing jitter_ema and lambda evolving monotonically toward equilibrium.

These are runnable as `cargo test -p ml-alpha --features cuda-diag-log --test gpu_log_ring_invariants`.

## 6. Risks

| Risk | Severity | Mitigation |
|------|----------|------------|
| Drainer falls behind, ring wraps, records lost | Low | Step-gap detection emits warn; bump N_SLOTS if observed in practice. |
| Memory fence cost slows hot path | Low | Only thread (0,0) writes; fence cost amortised over kernel's natural completion fence. Benchmark before/after smoke. |
| Decoder dispatch table drifts from kernel-side IDs | Medium | Single shared `gpu_log_ids.h` header included by both kernels and the Rust generated wrapper. Build.rs may verify counts match. |
| Adding two pointer args to kernels triggers partial-refactor compile errors | Medium | Atomic refactor commits (per `feedback_no_partial_refactor`); feature flag gates the new args at the source level. |
| Captured-graph re-record invalidates if pointers change | Low | Pointers are allocated once at trainer construction; never freed during training. |
| Magic collision (random memory matches LOG_MAGIC) | Negligible | 32-bit sentinel + step-count match doubles up the validation. |
| Performance regression on the captured-graph hot path | Low | The tick kernel is 1 thread, 1 block, 1 instruction. Logging kernels add 1 cacheline write. Microsecond-scale overhead. |

## 7. Open Questions

1. **Should the drainer run in `alpha_train` as a tokio task, or as a sidecar process attached to the same mapped-pinned buffer via shared-memory IPC?** Tokio task is simpler; sidecar would allow real-time monitoring tools to attach. Default: tokio task in-process; expose the buffer geometry so a sidecar can be added later without code changes.

2. **Variable-length payloads?** Current design fixes payload at ≤ 12 floats per record. Larger records (e.g., 256 B for matrix dumps) would require either a separate ring or a "continuation record" pattern. Defer until a use case appears.

3. **Drainer cadence — per-step vs per-second?** Per-second (500 ms tick) is sufficient for 2.8 s ring window. Per-step adds CPU contention for no diagnostic benefit. Default 500 ms.

4. **`gpu_log_ids.h` location.** `crates/ml-alpha/cuda/gpu_log_ids.h` is the kernel-side home. Rust side via `include_str!` + a small parser, OR a hand-maintained mirror in `gpu_log.rs`. Build.rs check would catch drift. Default: hand-mirror with build.rs verification.

5. **Should logging be on by default or opt-in?** Opt-in via cuda-diag-log feature flag. Production runs ship without it; investigation runs enable it.

6. **What happens if a logging-capable kernel is launched with `g_log_ring == NULL` but `g_step_counter != NULL` (or vice versa)?** Kernel branches on `g_log_ring != nullptr` only; step counter is unused without the ring. Safe under partial wiring during incremental rollout.

## 8. Implementation Surface

| File | Action |
|------|--------|
| `crates/ml-alpha/cuda/gpu_log_ring.cu` | NEW — defines `LogRecord`, `LogRing`, `log_record()`, the tick kernel |
| `crates/ml-alpha/cuda/gpu_log_ids.h` | NEW — kernel_id and record_type enum macros (single source of truth) |
| `crates/ml-alpha/build.rs` | Register `gpu_log_ring` in `KERNELS`; bump cache-bust v13 → v14 |
| `crates/ml-alpha/src/gpu_log.rs` | NEW — Rust-side ring allocator, drain task, decoder registry |
| `crates/ml-alpha/src/pinned_mem.rs` | Extend with a generic `MappedRecordBuffer<T>` (preserve `MappedF32Buffer` as type alias) |
| `crates/ml-alpha/src/lib.rs` | Expose `pub mod gpu_log` behind `#[cfg(feature = "cuda-diag-log")]` |
| `crates/ml-alpha/Cargo.toml` | Add `cuda-diag-log` feature |
| `crates/ml-alpha/src/trainer/perception.rs` | Add `log_ring_d`, `log_step_counter_d` fields; allocate at construction; pass to logging-capable kernels |
| `crates/ml-alpha/cuda/smoothness_lambda_controller.cu` | Add `g_log_ring`, `g_step_counter` args; emit RT_INPUT, RT_STATE, RT_OUTPUT records |
| `crates/ml-alpha/tests/gpu_log_ring_invariants.rs` | NEW — 6 unit tests + 1 integration test |

Plus the wire-up commits for each subsequent kernel as it opts in (BCE, output_smoothness, horizon_lambda, heads_grn_fwd, heads_grn_bwd, cfc_step). Initial spec scope wires only smoothness_lambda_controller; remaining kernels are follow-on tasks once the ring is proven.

Estimated total LOC (initial scope): ~550 lines new code + ~50 lines of perception.rs wiring + per-kernel wiring (5 lines × 1 kernel initially).

## 9. Out of Scope (deferred)

- Sidecar IPC drain (a separate process attaching to the mapped-pinned buffer)
- Variable-length payload records
- Compression / quantisation of recorded floats
- Auto-generation of the kernel_id/record_type enums via macros (manual maintenance + build.rs check)
- Wire-up of kernels other than `smoothness_lambda_controller` in the first plan; each subsequent kernel is its own atomic refactor in a follow-on commit.
- Visualisation tooling (plotly/grafana for the drained records) — out of scope; the tracing output is structured enough for offline analysis with jq/pandas.

## 10. Predecessor and Successor

**Predecessor:** `2026-05-21-crt-train-isv-driven-lambda-controller.md` — the controller whose per-step trajectory we cannot inspect.

**Successor (if WIN gate fails at base_lambda=0.001 and a second retrain is required):** The log ring enables per-step inspection of the controller's λ trajectory in the second retrain, telling us whether the issue is amplitude (mechanism #1 in the post-mortem) or trunk-sharing (mechanism #2 → intervention B).

**Successor (if WIN gate passes at base_lambda=0.001):** The ring is still valuable infrastructure for future controllers and kernel debugging, even if not immediately critical. Lower priority in that branch.

---

**End of GPU log ring design. Awaiting plan and execution.**
