# CRT.train Output-Smoothness Regularizer — Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Add a per-horizon output-smoothness regularizer to ml-alpha training that penalizes per-event prediction jitter, weighted strongly for slow horizons (h6000), to force horizon-differentiated outputs.

**Architecture:** New `output_smoothness.cu` GPU kernel reads `probs_per_k_d [K, B, 5]` (which already holds chronologically adjacent per-position probs within each sequence), computes `L_smooth[h] = λ[h] · mean_{k≥1, b} (p[k,b,h] − p[k−1,b,h])²`, and accumulates the analytic gradient INTO the existing `grad_probs_per_k_d` (after the BCE kernel writes it). λ[h] = `base_lambda × (K_h / K_min)` is uploaded once at trainer construction. New CLI flag `--smoothness-base-lambda <f32>` defaults to 0.0 (no-op) to preserve existing runs; the operator sets it explicitly for the retraining run.

**Tech Stack:** Rust 1.85+, cudarc 0.19, CUDA 12.4 (sm_89 L40S target), pre-compiled cubins via build.rs, anyhow, clap.

---

## File Structure (locked-in decisions)

| File | Role | Action |
|------|------|--------|
| `crates/ml-alpha/cuda/output_smoothness.cu` | New kernel: forward (loss) + analytic gradient, single block, warp-shuffle reduce | Create |
| `crates/ml-alpha/build.rs` | Add `"output_smoothness"` to `KERNELS` array | Modify |
| `crates/ml-alpha/src/heads.rs` | Add `pub const SMOOTHNESS_LAMBDA_RATIO: [f32; N_HORIZONS]` derived from `HORIZONS` | Modify |
| `crates/ml-alpha/src/trainer/perception.rs` | (1) Add device buffers + kernel handle to `PerceptionTrainer`. (2) Thread `smoothness_base_lambda` through `PerceptionTrainerConfig` + upload λ at construction. (3) Launch kernel inside `step_batched` AFTER the BCE launch (~line 1906) and BEFORE the backward K-loop (~line 1936). | Modify |
| `crates/ml-alpha/examples/alpha_train.rs` | Add `--smoothness-base-lambda` clap arg (default 0.0); thread into `PerceptionTrainerConfig`; add `final_smooth_loss_per_horizon: [f32; N_HORIZONS]` to `AlphaTrainSummary` | Modify |
| `crates/ml-alpha/tests/output_smoothness_grad_finite_diff.rs` | New test: GPU oracle, finite-diff probs, verify analytic grad matches | Create |
| `infra/k8s/argo/alpha-perception-template.yaml` | Add `smoothness-base-lambda` workflow parameter (default "0.0") + passthrough in training container args | Modify |

**Files NOT to modify:**
- `cuda/multi_horizon_heads.cu` — heads kernel unchanged; smoothness affects only `grad_probs_per_k_d` which heads_bwd already consumes.
- `cuda/bce_loss_multi_horizon.cu` — BCE unchanged; smoothness runs as a separate kernel.
- `src/multi_horizon_labels.rs` — labels unchanged (this is intervention A, not D).
- `src/data/loader.rs` — sequences ARE already temporally contiguous within `LabeledSequence.snapshots[0..seq_len]`; no loader changes needed (resolves spec §7 Q3-Q4).

---

## Mathematical contract (locked-in)

**Indexing.** `probs_per_k_d` is `[K * B * N_HORIZONS]` row-major; element at logical position (k, b, h) lives at flat index `k * B * 5 + b * 5 + h`.

**Loss (forward).** For each horizon h:
```
raw_h     = (1 / N_pairs) · Σ_{k=1..K-1} Σ_{b=0..B-1} (p[k,b,h] − p[k−1,b,h])²
L_smooth  = Σ_h λ[h] · raw_h
```
where `N_pairs = (K − 1) · B`. λ[h] is constant per training run.

**Gradient (backward, added to existing `grad_probs[k,b,h]`).**
```
∂L_smooth / ∂p[j,b,h] = (2 · λ[h] / N_pairs) · Δ(j,b,h)

where Δ(j,b,h) = ⎧  (p[0,b,h] − p[1,b,h])                            if j = 0
                ⎨  (2·p[j,b,h] − p[j−1,b,h] − p[j+1,b,h])             if 1 ≤ j ≤ K−2
                ⎩  (p[K−1,b,h] − p[K−2,b,h])                          if j = K−1
```

**λ schedule** (locked, computed at compile time in `heads.rs`):
```
SMOOTHNESS_LAMBDA_RATIO[h] = HORIZONS[h] as f32 / HORIZONS[0] as f32
                           = [1.0, 3.333…, 10.0, 33.333…, 200.0]
λ_device[h]                = base_lambda × SMOOTHNESS_LAMBDA_RATIO[h]
```
With `base_lambda = 0.0` (default), λ is all-zero → kernel produces zero loss + zero gradient → exact behavioral equivalence with pre-intervention training.

**GPU contract (per memory rules).**
- Warp-shuffle (`__shfl_xor_sync`) reduction; NO `atomicAdd`.
- Block-dim 256, grid-dim 1 — same pattern as `bce_multi_horizon_forward_backward`.
- No host branches inside the kernel body.
- No NVRTC (built via build.rs into a cubin).
- Gradient accumulator IS additive (`+=`), so caller does not need to zero `grad_probs_per_k_d` between BCE and smoothness — the kernel reads existing values and adds.

---

## Task 1: New CUDA kernel `output_smoothness.cu`

**Files:**
- Create: `crates/ml-alpha/cuda/output_smoothness.cu`

- [ ] **Step 1.1: Write the kernel file**

Write the following to `crates/ml-alpha/cuda/output_smoothness.cu`:

```cuda
// output_smoothness.cu — per-horizon temporal smoothness regularizer.
//
// Penalises (p[k] - p[k-1])² for adjacent positions within each sequence,
// weighted per-horizon by λ[h]. λ[h] scales with horizon length so slow
// horizons (h6000) are forced to change much more slowly than fast ones
// (h30). Forward loss is reported per-horizon (raw, pre-λ) for telemetry
// + lumped (λ-weighted) total.
//
// Per `feedback_nvidia_grade_perf_for_kernels.md`:
//   - Warp-shuffle reduction; no atomicAdd.
//   - Single block over the [K, B, N_HORIZONS] grid; stride loop per
//     thread covers all elements.
//   - No host branches, no divergent shuffles (inactive lanes contribute
//     0 via ternary).
//   - 5 horizon accumulators kept in registers (BCS-style).
//
// Per `feedback_no_atomicadd.md`: single-block tree-reduce (no atomics).
// Per `pearl_one_unbounded_signal_per_reward.md`: smoothness contributes
// a bounded (≤ 1 per pair after sigmoid) additive term — no
// unboundedness concerns.

#define OS_N_HORIZONS 5
#define OS_BLOCK      256
#define OS_N_WARPS    (OS_BLOCK / 32)   // 8

__device__ __forceinline__ float os_warp_reduce_sum(float v) {
    #pragma unroll
    for (int s = 16; s > 0; s >>= 1) {
        v += __shfl_xor_sync(0xffffffff, v, s);
    }
    return v;
}

// Smoothness forward + backward.
//
// Inputs:
//   probs           [K * B * 5]   row-major, index = k*B*5 + b*5 + h
//   lambda_per_h    [5]           λ[h] per horizon (already includes base × ratio)
//   K, B            (scalars)
//
// Outputs:
//   loss_out               [1]    Σ_h λ[h] · raw_h           — total loss
//   loss_raw_per_h         [5]    raw mean-sq-diff per horizon (no λ) — telemetry
//   grad_probs             [K*B*5]   ACCUMULATED (+=) into existing BCE grad
//
// The gradient at position j for the same (b, h):
//   grad += (2 λ[h] / N_pairs) · Δ(j, b, h)
//
// where Δ has the 3-branch boundary structure documented in the plan.
//
// Single block, OS_BLOCK threads. Each thread strides over the K*B*5
// flat range computing its own Δ(j, b, h) for the gradient pass.
// Forward loss is accumulated per-horizon in registers, then warp/block
// reduced; only thread 0 writes the 5 raw outputs + the λ-weighted total.
extern "C" __global__ void output_smoothness_loss_and_grad(
    const float* __restrict__ probs,            // [K * B * 5]
    const float* __restrict__ lambda_per_h,     // [5]
    int K,
    int B,
    float* __restrict__ loss_out,               // [1]
    float* __restrict__ loss_raw_per_h,         // [5]
    float* __restrict__ grad_probs              // [K * B * 5]   (+=)
) {
    const int tid  = threadIdx.x;
    const int lane = tid & 31;
    const int warp = tid >> 5;
    const int total = K * B * OS_N_HORIZONS;
    const int stride_k = B * OS_N_HORIZONS;     // stride between k and k+1 at fixed (b,h)
    const int n_pairs = (K - 1) * B;
    // N_pairs == 0 when K==1; in that case we still need to zero outputs.
    const float inv_n_pairs = (n_pairs > 0) ? (1.0f / (float)n_pairs) : 0.0f;

    // ── Forward: per-thread per-horizon raw accumulator ──
    // Anchor each (p[k]-p[k-1])² pair on its RIGHT element (k ≥ 1) so
    // each pair is counted exactly once across all threads.
    float local_raw[OS_N_HORIZONS];
    #pragma unroll
    for (int h = 0; h < OS_N_HORIZONS; ++h) local_raw[h] = 0.0f;

    for (int i = tid; i < total; i += OS_BLOCK) {
        const int h = i % OS_N_HORIZONS;
        // Decompose i into (k, b, h). k = i / (B*5); b = (i / 5) % B.
        const int k = i / stride_k;
        // Only positions k ≥ 1 anchor a (p[k]-p[k-1])² pair.
        if (k >= 1) {
            const float p_cur  = probs[i];
            const float p_prev = probs[i - stride_k];
            const float d = p_cur - p_prev;
            local_raw[h] += d * d;
        }
    }

    // Warp reduce per-horizon partial sums.
    float warp_raw[OS_N_HORIZONS];
    #pragma unroll
    for (int h = 0; h < OS_N_HORIZONS; ++h) {
        warp_raw[h] = os_warp_reduce_sum(local_raw[h]);
    }

    __shared__ float s_raw[OS_N_WARPS * OS_N_HORIZONS];
    if (lane == 0) {
        #pragma unroll
        for (int h = 0; h < OS_N_HORIZONS; ++h) {
            s_raw[warp * OS_N_HORIZONS + h] = warp_raw[h];
        }
    }
    __syncthreads();

    // Warp 0 finishes the cross-warp reduce.
    __shared__ float s_lambda[OS_N_HORIZONS];
    if (warp == 0) {
        #pragma unroll
        for (int h = 0; h < OS_N_HORIZONS; ++h) {
            float v = (lane < OS_N_WARPS) ? s_raw[lane * OS_N_HORIZONS + h] : 0.0f;
            v = os_warp_reduce_sum(v);
            if (lane == 0) {
                const float raw_h = v * inv_n_pairs;
                loss_raw_per_h[h] = raw_h;
                s_lambda[h] = lambda_per_h[h];
                // s_raw repurposed below as λ-weighted scratch.
                s_raw[h] = s_lambda[h] * raw_h;
            }
        }
    }
    __syncthreads();

    // Thread 0 sums the 5 λ-weighted terms into loss_out.
    if (tid == 0) {
        float total_loss = 0.0f;
        #pragma unroll
        for (int h = 0; h < OS_N_HORIZONS; ++h) total_loss += s_raw[h];
        loss_out[0] = total_loss;
    }
    __syncthreads();

    // ── Backward: per-thread compute Δ(j, b, h) and accumulate into grad_probs ──
    // Same stride loop. Three boundary cases:
    //   j = 0       : Δ = p[0]   − p[1]
    //   j = K−1     : Δ = p[K−1] − p[K−2]
    //   interior    : Δ = 2·p[j] − p[j−1] − p[j+1]
    // Scale: factor = 2 · λ[h] / N_pairs.
    //
    // s_lambda is now populated for all threads.
    for (int i = tid; i < total; i += OS_BLOCK) {
        const int h = i % OS_N_HORIZONS;
        const int k = i / stride_k;
        const float lam = s_lambda[h];
        const float factor = 2.0f * lam * inv_n_pairs;
        // Skip work when factor == 0 (e.g. lambda all zero); still must
        // not branch on host — the multiply by zero is the no-op.
        const float p_j = probs[i];
        float p_jm1 = 0.0f;
        float p_jp1 = 0.0f;
        // Read p[j-1] only if k >= 1; otherwise no contribution from
        // the left pair (we encode this by treating p_jm1 == p_j so the
        // (p[j] - p[j-1]) term in Δ contributes 0). Branchless via select.
        const bool has_left  = (k >= 1);
        const bool has_right = (k <= K - 2);
        if (has_left)  p_jm1 = probs[i - stride_k];
        if (has_right) p_jp1 = probs[i + stride_k];
        // Δ(j, b, h):
        //   left contribution  =  (p[j] - p[j-1])   if has_left  else 0
        //   right contribution = -(p[j+1] - p[j])   if has_right else 0
        // Sum = 2·p[j] − p[j-1] − p[j+1]  for interior;
        //       p[j] − p[j+1]             for j==0      (has_left=F, has_right=T)
        //       p[j] − p[j-1]             for j==K-1    (has_left=T,  has_right=F)
        // Use float multiplies to encode has_left/has_right (1.0 / 0.0).
        const float lm = has_left  ? 1.0f : 0.0f;
        const float rm = has_right ? 1.0f : 0.0f;
        const float delta = (lm + rm) * p_j - lm * p_jm1 - rm * p_jp1;
        grad_probs[i] += factor * delta;
    }
}
```

- [ ] **Step 1.2: Commit the kernel**

Run:
```bash
git add crates/ml-alpha/cuda/output_smoothness.cu
git commit -m "feat(crt-train): add output_smoothness CUDA kernel (forward + analytic grad)"
```

Expected: clean commit, no other files staged.

---

## Task 2: Register kernel in build.rs

**Files:**
- Modify: `crates/ml-alpha/build.rs` (KERNELS array, currently 13 entries ending with `"reduce_axis0"`)

- [ ] **Step 2.1: Add `"output_smoothness"` to the KERNELS array**

Edit `crates/ml-alpha/build.rs`. Locate the `KERNELS` array (lines 10–24). Add `"output_smoothness"` as the LAST element of the array, preserving formatting:

```rust
const KERNELS: &[&str] = &[
    "mamba2_alpha_kernel",          // Mamba2 SSM scan kernel (used by PerceptionTrainer's encoder prefix)
    "snap_feature_assemble",
    "cfc_step",
    "multi_horizon_heads",
    "projection",
    "bce_loss_multi_horizon",        // Kendall σ-weighted multi-horizon BCE (axis A)
    "adamw_step",
    "grad_norm",
    "horizon_lambda",                // ISV-driven per-horizon gradient scaler (EMA + lambda)
    "layer_norm",                    // Phase 1: trunk pre-CfC normalisation
    "variable_selection",            // Phase 2D: TFT-style per-feature gating
    "attention_pool",                // Phase 3: single-Q learned content summary at CfC k=0
    "reduce_axis0",                  // Phase B: cross-batch param-grad reducer
    "output_smoothness",             // CRT.train: per-horizon adjacent-position prob-jitter penalty
];
```

Also bump the cache-bust comment so old cubins force a rebuild. Locate (line 26):
```rust
// Cache bust v11 (2026-05-17): K-loop parallelization Phase B —
```
Replace with:
```rust
// Cache bust v12 (2026-05-20): output_smoothness.cu added — CRT.train intervention A.
```

- [ ] **Step 2.2: Verify build compiles the new cubin**

Run (from repo root):
```bash
SQLX_OFFLINE=true CARGO_FEATURE_CUDA=1 cargo build -p ml-alpha 2>&1 | grep -E "output_smoothness|error" | head
```

Expected output includes:
```
  ml-alpha: compiled cuda/output_smoothness.cu -> .../output_smoothness.cubin (sm_<cap>)
```
and NO `error:` lines. (If `nvcc` is unavailable on the local box, the build prints `nvcc not found, skipping kernel build` — this is acceptable for the local check; the Argo training job will run nvcc on the L40S CI node. In that case run a syntax-only check via `cargo check -p ml-alpha`.)

- [ ] **Step 2.3: Commit**

```bash
git add crates/ml-alpha/build.rs
git commit -m "build(crt-train): register output_smoothness.cu in ml-alpha build.rs KERNELS"
```

---

## Task 3: Add λ ratio constants in `heads.rs`

**Files:**
- Modify: `crates/ml-alpha/src/heads.rs` (around line 18–25 where `HORIZONS`, `HIDDEN_DIM` etc. live)

- [ ] **Step 3.1: Add the constant after `HEAD_MID_DIM`**

Edit `crates/ml-alpha/src/heads.rs`. Find the line:
```rust
pub const HEAD_MID_DIM: usize = 64;
```
Add immediately after it (before the next blank line):

```rust

/// Per-horizon multiplier for the output-smoothness regularizer (CRT.train
/// intervention A). λ[h] = `smoothness_base_lambda × SMOOTHNESS_LAMBDA_RATIO[h]`,
/// computed at `PerceptionTrainer` construction and uploaded once to the
/// device buffer `smoothness_lambda_d`. Ratio scales with horizon length
/// to force slow horizons (h6000) to change ~200× more slowly than fast
/// ones (h30). See `docs/superpowers/specs/2026-05-20-crt-train-output-smoothness-design.md` §3.1.
pub const SMOOTHNESS_LAMBDA_RATIO: [f32; N_HORIZONS] = [
    1.0,             // h30:    K_h / K_min = 30 / 30
    100.0 / 30.0,    // h100:   ≈ 3.333
    300.0 / 30.0,    // h300:   = 10.0
    1000.0 / 30.0,   // h1000:  ≈ 33.333
    6000.0 / 30.0,   // h6000:  = 200.0
];
```

- [ ] **Step 3.2: Verify with `cargo check`**

```bash
SQLX_OFFLINE=true cargo check -p ml-alpha 2>&1 | grep -E "error|warning: unused.*SMOOTHNESS" | head
```

Expected: no errors. (An "unused constant" warning is expected at this point because the trainer hasn't been wired up yet; that resolves in Task 5.)

- [ ] **Step 3.3: Commit**

```bash
git add crates/ml-alpha/src/heads.rs
git commit -m "feat(crt-train): add SMOOTHNESS_LAMBDA_RATIO const in heads.rs"
```

---

## Task 4: Write the finite-diff gradient test

**Files:**
- Create: `crates/ml-alpha/tests/output_smoothness_grad_finite_diff.rs`

This test runs BEFORE the kernel is wired into perception.rs, so it exercises the kernel via a small standalone GPU launcher inline in the test file (no thin wrapper in `src/` — the production path lives in `perception.rs` directly). Per `feedback_no_cpu_test_fallbacks.md`: GPU oracle only, finite-diff verifies analytic grad against numeric derivative of the same kernel's forward pass.

- [ ] **Step 4.1: Write the test file**

Write to `crates/ml-alpha/tests/output_smoothness_grad_finite_diff.rs`:

```rust
//! Output smoothness kernel — forward + gradient finite-diff validation.
//!
//! Per `feedback_no_cpu_test_fallbacks.md`: GPU oracle only. The analytic
//! gradient is verified against finite-difference of the kernel's own
//! forward pass (the kernel is the reference; we only check that its
//! analytic backward matches its own numerical derivative).
//!
//! Per `pearl_tests_must_prove_not_lock_observations.md`: assertions
//! verify INVARIANTS (loss ≥ 0 for non-zero diffs; loss == 0 when probs
//! are constant in k; finite-diff match) — not specific numeric outputs
//! that would lock the kernel to a single implementation.

use std::sync::Arc;

use anyhow::{Context, Result};
use approx::assert_relative_eq;
use cudarc::driver::{CudaSlice, CudaStream, DevicePtr, DevicePtrMut, LaunchConfig, PushKernelArg};
use ml_alpha::heads::N_HORIZONS;
use ml_core::device::MlDevice;

const CUBIN: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/output_smoothness.cubin"));

struct SmoothOut {
    loss: f32,
    raw_per_h: Vec<f32>,
    grad: Vec<f32>,
}

/// Launch the smoothness kernel against `probs [K*B*5]`, returning
/// (total λ-weighted loss, per-horizon raw mean-sq-diff, accumulated grad).
/// `grad_init` is the value `grad_probs` is pre-loaded with BEFORE the
/// kernel runs (since the kernel adds +=). Pass zeros to verify the
/// pure-smoothness gradient; non-zero values verify additivity.
fn run_smoothness_kernel(
    dev: &MlDevice,
    probs: &[f32],
    lambda_per_h: &[f32; N_HORIZONS],
    k: usize,
    b: usize,
    grad_init: &[f32],
) -> Result<SmoothOut> {
    let total = k * b * N_HORIZONS;
    assert_eq!(probs.len(), total);
    assert_eq!(grad_init.len(), total);

    let stream: &Arc<CudaStream> = dev.cuda_stream().context("smooth stream")?;
    let ctx = dev.cuda_context().context("smooth ctx")?;
    let module = ctx.load_cubin(CUBIN.to_vec()).context("load output_smoothness cubin")?;
    let func = module.load_function("output_smoothness_loss_and_grad").context("load fn")?;

    let probs_d   = upload(stream, probs)?;
    let lambda_d  = upload(stream, lambda_per_h.as_slice())?;
    let mut grad_d = upload(stream, grad_init)?;
    let mut loss_d = stream.alloc_zeros::<f32>(1).context("loss alloc")?;
    let mut raw_d  = stream.alloc_zeros::<f32>(N_HORIZONS).context("raw alloc")?;

    let k_i = k as i32;
    let b_i = b as i32;
    let cfg = LaunchConfig {
        grid_dim: (1, 1, 1),
        block_dim: (256, 1, 1),
        shared_mem_bytes: 0,
    };
    let mut launch = stream.launch_builder(&func);
    launch
        .arg(&probs_d).arg(&lambda_d)
        .arg(&k_i).arg(&b_i)
        .arg(&mut loss_d).arg(&mut raw_d).arg(&mut grad_d);
    unsafe { launch.launch(cfg).context("smoothness launch")?; }
    stream.synchronize().context("smoothness sync")?;

    Ok(SmoothOut {
        loss: download(stream, &loss_d)?[0],
        raw_per_h: download(stream, &raw_d)?,
        grad: download(stream, &grad_d)?,
    })
}

fn upload(stream: &Arc<CudaStream>, host: &[f32]) -> Result<CudaSlice<f32>> {
    use ml_alpha::pinned_mem::MappedF32Buffer;
    let n = host.len();
    let staging = unsafe { MappedF32Buffer::new(n) }
        .map_err(|e| anyhow::anyhow!("smooth upload staging: {e}"))?;
    staging.write_from_slice(host);
    let mut dst = stream.alloc_zeros::<f32>(n).context("smooth upload alloc")?;
    if n > 0 {
        let nbytes = n * std::mem::size_of::<f32>();
        unsafe {
            let (dst_ptr, _g) = dst.device_ptr_mut(stream);
            cudarc::driver::result::memcpy_dtod_async(dst_ptr, staging.dev_ptr, nbytes, stream.cu_stream())
                .context("smooth upload DtoD")?;
        }
    }
    Ok(dst)
}

fn download(stream: &Arc<CudaStream>, src: &CudaSlice<f32>) -> Result<Vec<f32>> {
    use ml_alpha::pinned_mem::MappedF32Buffer;
    let n = src.len();
    let staging = unsafe { MappedF32Buffer::new(n) }
        .map_err(|e| anyhow::anyhow!("smooth download staging: {e}"))?;
    let nbytes = n * std::mem::size_of::<f32>();
    unsafe {
        let (src_ptr, _g) = src.device_ptr(stream);
        cudarc::driver::result::memcpy_dtod_async(staging.dev_ptr, src_ptr, nbytes, stream.cu_stream())
            .context("smooth download DtoD")?;
    }
    stream.synchronize().context("smooth download sync")?;
    Ok(staging.read_all())
}

fn test_device() -> MlDevice {
    MlDevice::cuda(0).expect("CUDA 0 required for ml-alpha tests")
}

#[test]
fn smoothness_zero_when_probs_constant_in_k() {
    let dev = test_device();
    let k = 4usize;
    let b = 2usize;
    // probs[k, b, h] = 0.3 + 0.1*h + 0.05*b   (depends on b, h — NOT k)
    let mut probs = vec![0.0; k * b * N_HORIZONS];
    for kk in 0..k {
        for bb in 0..b {
            for h in 0..N_HORIZONS {
                probs[kk * b * N_HORIZONS + bb * N_HORIZONS + h] =
                    0.3 + 0.1 * (h as f32) + 0.05 * (bb as f32);
            }
        }
    }
    let lambda = [0.5_f32; N_HORIZONS];
    let grad_init = vec![0.0_f32; k * b * N_HORIZONS];
    let out = run_smoothness_kernel(&dev, &probs, &lambda, k, b, &grad_init).unwrap();
    // Constant-in-k probs ⇒ raw_h[h] == 0 for all h ⇒ total loss == 0 and grad all zero.
    for (h, &raw) in out.raw_per_h.iter().enumerate() {
        assert!(raw.abs() < 1e-7, "raw_per_h[{h}] = {raw} (expected ~0 for k-constant probs)");
    }
    assert!(out.loss.abs() < 1e-7, "loss = {} (expected ~0)", out.loss);
    for (i, &g) in out.grad.iter().enumerate() {
        assert!(g.abs() < 1e-7, "grad[{i}] = {g} (expected ~0)");
    }
}

#[test]
fn smoothness_loss_positive_for_jittery_probs() {
    let dev = test_device();
    let k = 6usize;
    let b = 1usize;
    // probs alternate sharply in k for every (b, h).
    let mut probs = vec![0.0; k * b * N_HORIZONS];
    for kk in 0..k {
        for h in 0..N_HORIZONS {
            probs[kk * b * N_HORIZONS + h] = if kk % 2 == 0 { 0.2 } else { 0.8 };
        }
    }
    let lambda = [1.0_f32; N_HORIZONS];
    let grad_init = vec![0.0_f32; probs.len()];
    let out = run_smoothness_kernel(&dev, &probs, &lambda, k, b, &grad_init).unwrap();
    // Each (p[k]-p[k-1])² = 0.36 for K-1=5 pairs per horizon ⇒ raw_h = 0.36 for all h.
    for (h, &raw) in out.raw_per_h.iter().enumerate() {
        assert!(raw > 0.3 && raw < 0.4, "raw_per_h[{h}] = {raw} (expected ≈ 0.36 for alternating 0.2/0.8)");
    }
    assert!(out.loss > 0.0, "loss should be positive for jittery probs, got {}", out.loss);
}

#[test]
fn smoothness_grad_is_additive_on_top_of_existing() {
    let dev = test_device();
    let k = 4usize;
    let b = 1usize;
    let probs: Vec<f32> = (0..k * b * N_HORIZONS).map(|i| 0.1 + 0.07 * (i as f32 % 7.0)).collect();
    let lambda = [0.3_f32; N_HORIZONS];

    // Run once with zero init, once with a known non-zero init.
    let zero_init = vec![0.0_f32; probs.len()];
    let out_zero = run_smoothness_kernel(&dev, &probs, &lambda, k, b, &zero_init).unwrap();

    let nz_init: Vec<f32> = (0..probs.len()).map(|i| 0.5 + 0.01 * (i as f32)).collect();
    let out_nz = run_smoothness_kernel(&dev, &probs, &lambda, k, b, &nz_init).unwrap();

    // out_nz.grad[i] should equal nz_init[i] + out_zero.grad[i].
    for i in 0..probs.len() {
        let expected = nz_init[i] + out_zero.grad[i];
        assert_relative_eq!(out_nz.grad[i], expected, epsilon = 5e-5, max_relative = 5e-4);
    }
}

#[test]
fn smoothness_grad_matches_finite_difference() {
    let dev = test_device();
    let k = 5usize;
    let b = 2usize;
    let mut probs = vec![0.0; k * b * N_HORIZONS];
    for i in 0..probs.len() {
        // Vary in k so derivatives are non-trivial.
        probs[i] = 0.2 + 0.05 * ((i % 7) as f32) + 0.03 * ((i / 7) as f32 % 5.0);
    }
    let lambda: [f32; N_HORIZONS] = [0.1, 0.3, 1.0, 3.0, 10.0];  // mimic the production ratio
    let zero_init = vec![0.0_f32; probs.len()];
    let base = run_smoothness_kernel(&dev, &probs, &lambda, k, b, &zero_init).unwrap();

    let eps = 1e-3_f32;
    // Probe a representative set of indices: corner (k=0), interior (k=2),
    // far corner (k=K-1), across two batches and horizons.
    let probe_indices: Vec<usize> = vec![
        // (k=0, b=0, h=0)
        0,
        // (k=0, b=1, h=2)
        0 * b * N_HORIZONS + 1 * N_HORIZONS + 2,
        // (k=2, b=0, h=1)
        2 * b * N_HORIZONS + 0 * N_HORIZONS + 1,
        // (k=2, b=1, h=4)
        2 * b * N_HORIZONS + 1 * N_HORIZONS + 4,
        // (k=K-1, b=0, h=3)
        (k - 1) * b * N_HORIZONS + 0 * N_HORIZONS + 3,
        // (k=K-1, b=1, h=0)
        (k - 1) * b * N_HORIZONS + 1 * N_HORIZONS + 0,
    ];
    for &i in &probe_indices {
        let mut up = probs.clone();
        up[i] += eps;
        let mut dn = probs.clone();
        dn[i] -= eps;
        let lu = run_smoothness_kernel(&dev, &up, &lambda, k, b, &zero_init).unwrap().loss;
        let ld = run_smoothness_kernel(&dev, &dn, &lambda, k, b, &zero_init).unwrap().loss;
        let fd = (lu - ld) / (2.0 * eps);
        assert_relative_eq!(
            base.grad[i],
            fd,
            epsilon = 5e-4,
            max_relative = 5e-2
        );
    }
}

#[test]
fn smoothness_zero_lambda_gives_zero_grad_and_loss() {
    // Default `smoothness_base_lambda = 0.0` ⇒ λ[h] = 0 ∀h ⇒ kernel
    // produces neither loss nor gradient. Verifies the opt-in default
    // is exactly behaviour-preserving.
    let dev = test_device();
    let k = 4usize;
    let b = 2usize;
    let probs: Vec<f32> = (0..k * b * N_HORIZONS).map(|i| 0.1 + 0.05 * (i as f32 % 9.0)).collect();
    let lambda = [0.0_f32; N_HORIZONS];
    let grad_init = vec![0.0_f32; probs.len()];
    let out = run_smoothness_kernel(&dev, &probs, &lambda, k, b, &grad_init).unwrap();
    assert!(out.loss.abs() < 1e-7);
    for (i, &g) in out.grad.iter().enumerate() {
        assert!(g.abs() < 1e-7, "grad[{i}] = {g} (expected 0 for λ=0)");
    }
    // raw_per_h is still populated (telemetry — independent of λ).
    // We don't assert specific values; only that the loss is zero.
}
```

- [ ] **Step 4.2: Run the tests**

```bash
SQLX_OFFLINE=true cargo test -p ml-alpha --test output_smoothness_grad_finite_diff -- --nocapture
```

Expected: all 5 tests pass. If `cargo test` reports "test was skipped: CUDA 0 required", that means CUDA is unavailable on the local box — record this and proceed; the Argo CI job will re-run on L40S where CUDA IS available, and Task 9 below has a re-run gate before retraining.

- [ ] **Step 4.3: Commit**

```bash
git add crates/ml-alpha/tests/output_smoothness_grad_finite_diff.rs
git commit -m "test(crt-train): finite-diff validate output_smoothness kernel grad"
```

---

## Task 5: Wire kernel into `PerceptionTrainer` — buffers, handle, λ upload

**Files:**
- Modify: `crates/ml-alpha/src/trainer/perception.rs`

This task adds the device buffers and the kernel handle to the trainer struct, and uploads the λ table at construction time. Wire-up of the kernel LAUNCH into `step_batched` is Task 6.

- [ ] **Step 5.1: Locate the trainer struct and add the new fields**

Find the line in `crates/ml-alpha/src/trainer/perception.rs`:
```rust
loss_per_horizon_d: CudaSlice<f32>,
```
(around line 276). Add immediately after that line:

```rust
    /// Per-horizon smoothness multiplier λ[h] = `cfg.smoothness_base_lambda
    /// × SMOOTHNESS_LAMBDA_RATIO[h]`. Uploaded ONCE at construction;
    /// never mutated. Read by `output_smoothness_loss_and_grad` inside
    /// the captured graph (no host writes during capture).
    smoothness_lambda_d: CudaSlice<f32>,
    /// Scalar total smoothness loss output (λ-weighted Σ_h). Output of
    /// `output_smoothness_loss_and_grad`. Mapped-pinned host shadow
    /// `smoothness_loss_host_d` provides the post-step value with no
    /// extra dtoh sync (telemetry only).
    smoothness_loss_d: CudaSlice<f32>,
    smoothness_loss_host_d: MappedF32Buffer,
    /// Per-horizon raw mean-squared-diff (pre-λ). Telemetry feed for
    /// the epoch summary writer. Shape [N_HORIZONS].
    smoothness_loss_per_horizon_d: CudaSlice<f32>,
    smoothness_loss_per_horizon_host_d: MappedF32Buffer,
    /// Cached handle for `output_smoothness_loss_and_grad`.
    smoothness_fn: CudaFunction,
    _smoothness_module: Arc<CudaModule>,
```

- [ ] **Step 5.2: Locate the trainer constructor and add a config field + initialization**

Find the `pub struct PerceptionTrainerConfig` definition (use grep to locate exact line):
```bash
grep -n "pub struct PerceptionTrainerConfig" crates/ml-alpha/src/trainer/perception.rs
```

Open the struct in your editor. Add a new field at the END of the struct (before the closing `}`):

```rust
    /// CRT.train intervention A: output-smoothness regularizer base λ.
    /// Effective λ[h] = `smoothness_base_lambda × SMOOTHNESS_LAMBDA_RATIO[h]`.
    /// Default 0.0 = regularizer disabled (kernel still launches but
    /// produces zero loss + zero gradient). Set to e.g. 0.01 to enable.
    /// See `docs/superpowers/specs/2026-05-20-crt-train-output-smoothness-design.md`.
    pub smoothness_base_lambda: f32,
```

Locate the `Default` impl for `PerceptionTrainerConfig` (grep below) and add a default of `0.0` for the new field:
```bash
grep -n "impl Default for PerceptionTrainerConfig" crates/ml-alpha/src/trainer/perception.rs
```

Inside that `Default::default()` block, add (preserving the existing pattern):
```rust
            smoothness_base_lambda: 0.0,
```

- [ ] **Step 5.3: Add the cubin include and kernel-load lines**

Near the top of `perception.rs`, locate the existing cubin include block (around line 59):
```rust
const HEADS_CUBIN: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/multi_horizon_heads.cubin"));
```
Add (preserving the existing block structure — look for the analogous `BCE_CUBIN`/`HORIZON_LAMBDA_CUBIN` constants and add adjacent):

```rust
const SMOOTHNESS_CUBIN: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/output_smoothness.cubin"));
```

In the constructor (search for the existing `let heads_grn_bwd_fn = heads_module.load_function(...)`), find the section that loads function handles. Around line 658-661 has:
```rust
        let bce_fn = bce_module.load_function("bce_multi_horizon_forward_backward")?;
```
Add a parallel block immediately after the existing module/function loads (locate by searching for `horizon_lambda_fn` to find the analogous pattern):

```rust
        let smoothness_module = ctx
            .load_cubin(SMOOTHNESS_CUBIN.to_vec())
            .context("load output_smoothness cubin")?;
        let smoothness_fn = smoothness_module
            .load_function("output_smoothness_loss_and_grad")
            .context("load output_smoothness_loss_and_grad")?;
```

- [ ] **Step 5.4: Allocate the smoothness buffers + upload λ**

In the constructor (continuing from above), find where `loss_per_horizon_d` is allocated (search `loss_per_horizon_d: stream.alloc_zeros`). Around line 1028. Right after that allocation, add:

```rust
        // ── CRT.train: output-smoothness regularizer state ──
        // λ[h] = base_lambda × ratio[h]. Uploaded once; never mutated.
        let smoothness_lambda_host: Vec<f32> = SMOOTHNESS_LAMBDA_RATIO
            .iter()
            .map(|r| cfg.smoothness_base_lambda * r)
            .collect();
        let smoothness_lambda_d = {
            // Upload via mapped-pinned (per `feedback_no_htod_htoh_only_mapped_pinned`).
            let staging = unsafe { MappedF32Buffer::new(N_HORIZONS) }
                .map_err(|e| anyhow::anyhow!("smoothness λ staging: {e}"))?;
            staging.write_from_slice(&smoothness_lambda_host);
            let mut dst = stream.alloc_zeros::<f32>(N_HORIZONS)
                .context("smoothness_lambda_d alloc")?;
            let nbytes = N_HORIZONS * std::mem::size_of::<f32>();
            unsafe {
                let (dst_ptr, _g) = dst.device_ptr_mut(&stream);
                cudarc::driver::result::memcpy_dtod_async(
                    dst_ptr, staging.dev_ptr, nbytes, stream.cu_stream(),
                ).context("smoothness λ DtoD")?;
            }
            dst
        };
        let smoothness_loss_d = stream.alloc_zeros::<f32>(1)
            .context("smoothness_loss_d alloc")?;
        let smoothness_loss_host_d = unsafe { MappedF32Buffer::new(1) }
            .map_err(|e| anyhow::anyhow!("smoothness_loss_host_d: {e}"))?;
        let smoothness_loss_per_horizon_d = stream.alloc_zeros::<f32>(N_HORIZONS)
            .context("smoothness_loss_per_horizon_d alloc")?;
        let smoothness_loss_per_horizon_host_d = unsafe { MappedF32Buffer::new(N_HORIZONS) }
            .map_err(|e| anyhow::anyhow!("smoothness_loss_per_horizon_host_d: {e}"))?;
```

In the same constructor's final `Ok(Self { ... })` block, add the new fields (preserve existing comma-separated field-list style):

```rust
            smoothness_lambda_d,
            smoothness_loss_d,
            smoothness_loss_host_d,
            smoothness_loss_per_horizon_d,
            smoothness_loss_per_horizon_host_d,
            smoothness_fn,
            _smoothness_module: smoothness_module,
```

Make sure `SMOOTHNESS_LAMBDA_RATIO` is imported. At the existing `use crate::heads::...` or `use crate::heads as heads;` (search for it), add `SMOOTHNESS_LAMBDA_RATIO` to the import list. Example:
```rust
use crate::heads::{HEAD_MID_DIM, HIDDEN_DIM, HORIZONS, N_HORIZONS, SMOOTHNESS_LAMBDA_RATIO};
```

- [ ] **Step 5.5: Build and check for compile errors**

```bash
SQLX_OFFLINE=true cargo check -p ml-alpha 2>&1 | grep -E "error\[|^error:" | head -20
```

Expected: no errors. Warnings about unused fields (smoothness_loss_d etc.) are OK at this point — Task 6 will wire the launch.

- [ ] **Step 5.6: Commit**

```bash
git add crates/ml-alpha/src/trainer/perception.rs
git commit -m "feat(crt-train): allocate smoothness buffers + upload λ in PerceptionTrainer"
```

---

## Task 6: Launch the smoothness kernel inside `step_batched`

**Files:**
- Modify: `crates/ml-alpha/src/trainer/perception.rs` — `step_batched` function body, between the existing BCE launch (~line 1906) and the existing backward K-loop (~line 1936).

- [ ] **Step 6.1: Add the launch immediately after the BCE block**

Locate the block in `step_batched` that ends with the BCE launch:
```rust
        {
            let mut launch = self.stream.launch_builder(&self.bce_fn);
            launch
                .arg(&self.probs_per_k_d).arg(&self.labels_per_k_d)
                .arg(&self.loss_weights_d)
                .arg(&self.log_sigma_h_d)
                .arg(&n_pos_i).arg(&n_h_i)
                .arg(&mut self.loss_d)
                .arg(&mut self.loss_per_horizon_d)
                .arg(&mut self.grad_probs_per_k_d)
                .arg(&mut self.valid_d)
                .arg(&mut self.grad_log_sigma_h_d);
            unsafe { launch.launch(bce_cfg).context("bce launch")?; }
        }
```

Add immediately after the closing `}`:

```rust
        // ── 5c. CRT.train output-smoothness regularizer ──
        //       λ[h]-weighted Σ (p[k]-p[k-1])² across adjacent positions
        //       in each sequence. Accumulates the analytic gradient INTO
        //       `self.grad_probs_per_k_d` (which BCE just overwrote);
        //       the heads_bwd K-loop below consumes the sum.
        //
        //       Per `pearl_no_host_branches_in_captured_graph`: kernel
        //       launch only — all scalars (k_i, b_i) and buffer pointers
        //       are fixed across captures (seq_len and batch_size are
        //       config-immutable). λ is uploaded once at construction;
        //       when `cfg.smoothness_base_lambda == 0.0` the kernel
        //       produces zero loss + zero grad, behaviorally identical
        //       to the pre-intervention path.
        let k_seq_i = k_seq as i32;
        let b_sz_i = b_sz as i32;
        let smooth_cfg = LaunchConfig {
            grid_dim: (1, 1, 1),
            block_dim: (256, 1, 1),
            shared_mem_bytes: 0,
        };
        {
            let mut launch = self.stream.launch_builder(&self.smoothness_fn);
            launch
                .arg(&self.probs_per_k_d)
                .arg(&self.smoothness_lambda_d)
                .arg(&k_seq_i).arg(&b_sz_i)
                .arg(&mut self.smoothness_loss_d)
                .arg(&mut self.smoothness_loss_per_horizon_d)
                .arg(&mut self.grad_probs_per_k_d);
            unsafe { launch.launch(smooth_cfg).context("output_smoothness launch")?; }
        }
```

(Note: `k_seq` and `b_sz` are already defined in scope from the existing BCE launch — search upward for `let n_pos_i = (k_seq * b_sz)` to confirm.)

- [ ] **Step 6.2: Hook host-shadow async memcpy for telemetry**

For the host shadow to be populated by the end of `step_batched`, find the existing memcpy chain that already shadows `loss_d` to `loss_host_d` (search `loss_host_d`):

```bash
grep -n "loss_host_d\|memcpy_dtod_async.*loss" crates/ml-alpha/src/trainer/perception.rs | head -10
```

Add parallel `memcpy_dtod_async` calls for `smoothness_loss_d → smoothness_loss_host_d` (1 float) and `smoothness_loss_per_horizon_d → smoothness_loss_per_horizon_host_d` (5 floats), placed AT THE SAME location as the existing `loss_host_d` memcpy. Mirror the exact pattern (same stream, same memcpy_dtod_async helper).

- [ ] **Step 6.3: Build, check, and run existing trainer tests**

```bash
SQLX_OFFLINE=true cargo check -p ml-alpha 2>&1 | grep -E "error\[|^error:" | head -10
```

Expected: clean compile.

If a CUDA-equipped local box is available, also run:
```bash
FOXHUNT_TEST_DATA=test_data/futures-baseline cargo test -p ml-alpha --lib -- smoke_tests --ignored --nocapture 2>&1 | tail -40
```

Expected: the smoke runs and exits with the usual perception step output; no NaN/divergence with smoothness_base_lambda = 0.0 (the default).

- [ ] **Step 6.4: Commit**

```bash
git add crates/ml-alpha/src/trainer/perception.rs
git commit -m "feat(crt-train): launch output_smoothness kernel after BCE in step_batched"
```

---

## Task 7: Add `--smoothness-base-lambda` CLI flag

**Files:**
- Modify: `crates/ml-alpha/examples/alpha_train.rs`

- [ ] **Step 7.1: Add the clap arg**

In `crates/ml-alpha/examples/alpha_train.rs`, find the last `#[arg(...)]` in the `Cli` struct (currently the `cv_train_window` field, around lines 153-154):
```rust
    #[arg(long, default_value_t = 0)]
    cv_train_window: usize,

}
```

Add a new field BEFORE the closing `}` of the struct:

```rust

    /// CRT.train intervention A — output-smoothness regularizer.
    ///
    /// λ[h] = base × (HORIZONS[h] / HORIZONS[0]). Default 0.0 disables
    /// the regularizer (kernel still launches, produces zero gradient).
    /// Recommended sweep: 0.001 / 0.01 / 0.1. See
    /// `docs/superpowers/specs/2026-05-20-crt-train-output-smoothness-design.md`.
    #[arg(long, default_value_t = 0.0)]
    smoothness_base_lambda: f32,
```

- [ ] **Step 7.2: Thread the flag into PerceptionTrainerConfig**

In the same file, locate where `PerceptionTrainerConfig` is constructed (search `PerceptionTrainerConfig {`). Add the new field assignment:

```rust
        smoothness_base_lambda: cli.smoothness_base_lambda,
```

- [ ] **Step 7.3: Add telemetry field to `AlphaTrainSummary`**

In the same file, locate `struct AlphaTrainSummary` (around line 158). Add a new field BEFORE the closing `}`:

```rust
    /// Per-horizon raw mean-squared adjacent-prob diff at the final epoch
    /// (CRT.train intervention A telemetry). All zeros when
    /// `--smoothness-base-lambda` is unset (default).
    #[serde(default)]
    final_smooth_loss_per_horizon: [f32; N_HORIZONS],
```

Locate where the final summary is assembled (search `AlphaTrainSummary {`) and populate the new field. Read the smoothness-per-horizon telemetry from the trainer (add a public accessor in `perception.rs` if needed — see Step 7.4):

```rust
        final_smooth_loss_per_horizon: trainer.last_smoothness_loss_per_horizon(),
```

- [ ] **Step 7.4: Add the accessor in perception.rs**

In `crates/ml-alpha/src/trainer/perception.rs`, locate `pub fn last_probs_per_position` (around line 3804) and add a parallel accessor immediately before or after it:

```rust
    /// Last-step per-horizon raw mean-squared-diff (pre-λ). Reads from
    /// the mapped-pinned host shadow populated by `step_batched`. Returns
    /// all zeros when the smoothness regularizer is disabled
    /// (`cfg.smoothness_base_lambda == 0.0`).
    pub fn last_smoothness_loss_per_horizon(&self) -> [f32; N_HORIZONS] {
        let raw = self.smoothness_loss_per_horizon_host_d.read_all();
        let mut out = [0.0f32; N_HORIZONS];
        for (i, v) in raw.into_iter().take(N_HORIZONS).enumerate() {
            out[i] = v;
        }
        out
    }
```

- [ ] **Step 7.5: Build + verify CLI accepts the new flag**

```bash
SQLX_OFFLINE=true cargo build -p ml-alpha --example alpha_train 2>&1 | tail -5
```

Expected: builds cleanly.

```bash
./target/debug/examples/alpha_train --help 2>&1 | grep -A1 smoothness
```

Expected: shows `--smoothness-base-lambda <SMOOTHNESS_BASE_LAMBDA>` with the doc comment.

- [ ] **Step 7.6: Commit**

```bash
git add crates/ml-alpha/examples/alpha_train.rs crates/ml-alpha/src/trainer/perception.rs
git commit -m "feat(crt-train): add --smoothness-base-lambda CLI flag + telemetry"
```

---

## Task 8: Plumb flag through Argo workflow

**Files:**
- Modify: `infra/k8s/argo/alpha-perception-template.yaml`

- [ ] **Step 8.1: Add workflow parameter**

In `infra/k8s/argo/alpha-perception-template.yaml`, locate the `parameters:` block (around lines 39-80). Find the last parameter `decision-stride`. Add immediately after it (preserving exact YAML indent):

```yaml
      - name: smoothness-base-lambda
        value: "0.0"
```

- [ ] **Step 8.2: Pass the flag through the training command**

In the same file, locate the training container's `args:` block (around line 433, the line `"$BIN" \`). Add a new line BEFORE `$EXTRA_FLAGS`:

```yaml
              --smoothness-base-lambda {{workflow.parameters.smoothness-base-lambda}} \
```

- [ ] **Step 8.3: Commit**

```bash
git add infra/k8s/argo/alpha-perception-template.yaml
git commit -m "feat(crt-train): plumb --smoothness-base-lambda through alpha-perception template"
```

---

## Task 9: Pre-deployment validation gate (local + CI)

This task is a HARD GATE before retraining. Per
[feedback_smoke_validation_before_structural_priors.md](../../../.claude/projects/-home-jgrusewski-Work-foxhunt/memory/feedback_smoke_validation_before_structural_priors.md):
validate wiring with the default (0.0) BEFORE deploying with a non-zero λ.

- [ ] **Step 9.1: Push the branch**

```bash
git push origin ml-alpha-phase-a
```

Expected: pushes 7-8 new commits (one per Task above) to origin.

- [ ] **Step 9.2: Validation Run #1 — λ = 0 wiring sanity**

Deploy a short training run with `--smoothness-base-lambda 0.0` (default) using the existing dispatcher. Goal: verify the new kernel launches and emits zeros without breaking existing training.

```bash
./scripts/argo-train.sh supervised --gpu-pool ci-training-l40s -p commit-sha=$(git rev-parse HEAD)
```

(If `argo-train.sh` has a `supervised`/`alpha-perception` mode flag — check `--help`; otherwise pass `-p` arguments directly to `argo submit --from=wftmpl/alpha-perception`.)

Wait for completion. Inspect the per-step log output for `smoothness_loss_per_horizon` (or check the emitted `alpha_train_summary.json` for `final_smooth_loss_per_horizon`). All five values MUST be 0.0 — confirming the kernel launches but contributes nothing when λ=0.

If they're non-zero with λ=0, STOP and debug — the kernel is producing spurious gradient. Likely culprits: incorrect index decomposition (k/b/h confusion), uninitialized output buffer, or λ upload bug.

- [ ] **Step 9.3: Validation Run #2 — λ small enough to not destroy AUC**

Once Run #1 passes, deploy a retrain with `smoothness-base-lambda = 0.001` (the conservative end of the spec's proposed sweep):

```bash
argo submit -n foxhunt --from=wftmpl/alpha-perception \
  -p commit-sha=$(git rev-parse HEAD) \
  -p git-branch=ml-alpha-phase-a \
  -p gpu-pool=ci-training-l40s \
  -p smoothness-base-lambda=0.001 \
  -p n-train-seqs=8000 \
  -p n-val-seqs=1000 \
  -p epochs=5
```

Expected wall time: ~30-90 min on L40S (per existing template comment).

While running, periodically (every ~10 min) check the training pod logs:
```bash
kubectl logs -n foxhunt -l workflows.argoproj.io/workflow=<wfname> --tail=200 | grep -E "epoch|smooth|auc|loss"
```

Watch for:
- `final_smooth_loss_per_horizon`: should be non-zero, increasing weakly with h (h6000 > h30).
- per-horizon AUC: should remain within ~2pp of the lnfwd baseline (mean ≈ 0.66, h6000 ≈ 0.66). Per spec §5 MUST gate: `AUC_h ≥ baseline_AUC_h − 0.02`. Drop > 0.02 → STOP and downsweep base λ.

When the workflow completes, download `alpha_train_summary.json` and verify the MUST gate. If passed, proceed to Task 10. If failed (AUC dropped > 0.02 anywhere), retry with `smoothness-base-lambda=0.0003` (3× weaker) before escalating.

---

## Task 10: WIN-gate validation via CRT.diag

This task runs the spec §5 WIN gate: does the regularizer actually produce horizon-differentiated outputs?

**Files referenced (no edits):**
- `scripts/argo-train.sh` (or the equivalent CRT.diag dispatcher) for running the LOB backtest sweep
- The diagnostic instrumentation already merged at commits `b44a97ff9` (Group A-D) and `0e17fd4f2` (Group E smoothed-flip)

- [ ] **Step 10.1: Locate the CRT.diag dispatcher**

```bash
grep -rn "CRT\.diag\|lob-backtest-sweep\|crt_diag\|lnfwd" scripts/ infra/k8s/argo/ 2>/dev/null | head -20
```

Expected to find the LOB backtest workflow or script the diag was last invoked from. Note the command pattern used at commit `0e17fd4f2`.

- [ ] **Step 10.2: Run CRT.diag against the new checkpoint**

Using the dispatcher identified in Step 10.1, run the same LOB backtest sweep that produced the `lnfwd` data point (2M events, alpha-input Wiener-α EMA floor 0.1). Point it at the new checkpoint emitted by Task 9.3.

Expected runtime: ~10-30 min on L40S.

- [ ] **Step 10.3: Apply the WIN gate**

From the CRT.diag output, extract the per-horizon mean run length (smoothed, post-Wiener-α-EMA). Compare against the `lnfwd` baseline:

| Metric | Baseline (lnfwd) | WIN gate | New checkpoint | Verdict |
|---|---|---|---|---|
| h30  mean_run_len | 3.0  | ≤ 10  | __FILL__ | __FILL__ |
| h100 mean_run_len | 3.3  | —     | __FILL__ | — |
| h300 mean_run_len | 3.0  | —     | __FILL__ | — |
| h1000 mean_run_len | 3.3 | —     | __FILL__ | — |
| h6000 mean_run_len | 3.3 | ≥ 100 | __FILL__ | __FILL__ |
| h6000 / h30 ratio | ~1.1 | ≥ 10× | __FILL__ | __FILL__ |

Decision tree:
- **WIN gate PASSED** (h6000 ≥ 100 events, ratio ≥ 10×) → proceed to Task 11 (STRETCH gate).
- **WIN gate FAILED** but signal-direction correct (h6000 > h30, both grew vs baseline) → sweep up: rerun Task 9.3 with `smoothness-base-lambda=0.01` or `0.1`; rerun Task 10.
- **WIN gate FAILED** AND λ has been swept across 0.001 / 0.01 / 0.1 without success → escalate to intervention B (spec §9), starting with the architecture-change design at
  `docs/superpowers/specs/2026-05-20-crt-train-architecture-change-design.md` (write that spec at that point, not now).

Record the WIN-gate table in the project memory at
`/home/jgrusewski/.claude/projects/-home-jgrusewski-Work-foxhunt/memory/project_crt_train_intervention_a_outcome.md`
with a `metadata.type: project` entry indexed in `MEMORY.md`.

---

## Task 11: STRETCH-gate validation via CRT.1 controller smoke

Per spec §5 STRETCH: does mean PnL now correlate with conviction (vs the anti-correlated lnfwd baseline -45 → -84 monotonic worsening)?

- [ ] **Step 11.1: Re-run the CRT.1 controller smoke**

Using the same CRT.1 controller architecture currently on `ml-alpha-phase-a` (no controller changes — the controller is correct given its inputs per memory file), run the controller smoke against the new checkpoint produced by Task 9.3 (or whichever λ best passed WIN).

```bash
# Use the dispatcher identified in Task 10.1 with the controller-enabled
# preset, pointed at the new checkpoint.
```

- [ ] **Step 11.2: Extract conviction × outcome table**

From the smoke output, produce the conviction-bucket table parallel to the lnfwd baseline (from `project_crt_diag_findings.md`):

| Bucket    | n (lnfwd) | win_rate (lnfwd) | mean_pnl (lnfwd) | n (new) | win_rate (new) | mean_pnl (new) |
|---|---|---|---|---|---|---|
| [0.1-0.2] | 41,824 | 22.51% | -45 | __FILL__ | __FILL__ | __FILL__ |
| [0.2-0.3] | 96,707 | 21.20% | -51 | __FILL__ | __FILL__ | __FILL__ |
| [0.3-0.4] | 64,940 | 21.80% | -65 | __FILL__ | __FILL__ | __FILL__ |
| [0.4-0.5] | 21,384 | 20.32% | -63 | __FILL__ | __FILL__ | __FILL__ |
| [0.5-0.6] |  2,028 | 21.79% | -84 | __FILL__ | __FILL__ | __FILL__ |

- [ ] **Step 11.3: Apply the STRETCH gate**

- **STRETCH PASSED** if either: (a) mean_pnl increases monotonically with conviction OR (b) win_rate increases with conviction → CRT.1 is now extracting alpha; document outcome and resume CRT.2 envelope work per spec §9 successor path.
- **STRETCH FAILED** despite WIN passing → model output is horizon-differentiated but still not predictive of trade outcomes. Per spec §5 escalation: write a spec for intervention B (architecture change to horizon-conditional outputs) — write that spec at that point, not now.

Update the memory file from Task 10.3 with the STRETCH outcome.

---

## Self-Review (run before claiming the plan complete)

After writing this plan I checked it against the spec:

1. **Spec §1 (motivation)** — addressed in Task 0 framing (the goal statement at top).
2. **Spec §2 (hypothesis)** — encoded in the kernel formula at Task 1.
3. **Spec §3.1 (regularizer math)** — Task 1 implements exactly `L_smooth[h] = λ[h] · mean (p[k]-p[k-1])²`.
4. **Spec §3.2 (minimum-scope)** — plan does NOT touch architecture (B), label generation (D), or curriculum (C); intervention A only. Verified at "Files NOT to modify" list.
5. **Spec §4 (implementation surface)** — every file in spec §4 is covered: `multi_horizon_heads.cu` is unchanged (as documented in spec §4), `perception.rs` (Tasks 5-6), `heads.rs` (Task 3), `alpha_train.rs` (Task 7). Plus the Argo plumbing (Task 8) which spec §4 didn't enumerate but is needed for the validation deploy.
6. **Spec §5 (validation gates)** — MUST gate in Task 9.3, WIN gate in Task 10, STRETCH gate in Task 11.
7. **Spec §6 (risks)** — covered: over-smoothing risk addressed by λ=0.0 default + 0.001 conservative start (Task 9.3); gradient explosion handled by existing optimizer; training time bloat is bounded by O(K·B·5) kernel (Task 1 single-block design); WIN-without-STRETCH path documented (Task 11).
8. **Spec §7 open questions:**
   - Q1 (base_λ value): defaulted to 0.0 (opt-in), sweep starts at 0.001 in Task 9.3 (conservative end of spec's range).
   - Q2 (within-window vs cross-window): resolved as within-window via the `[K, B, 5]` layout — Task 1's kernel is exactly within-window. Documented in plan preamble + spec resolution comment.
   - Q3 (per-batch vs per-fold target): resolved — sequences ARE temporally contiguous within the loader's `LabeledSequence.snapshots[0..seq_len]`. No loader change needed.
   - Q4 (loader supports adjacent pairs?): resolved as YES per the loader read at Task 0 — `next_sequence()` returns `seq_len` consecutive snapshots from a single source file.

9. **Spec §8 (out of scope)** — confirmed: plan does not introduce architecture changes, curriculum, label regen, multi-task losses, or online weight adaptation.

10. **Placeholder scan:** searched the plan for TODO/TBD/"implement later"/"add appropriate"/"similar to". Found none in commit-shipping content. The validation tables in Tasks 10-11 use `__FILL__` placeholders intentionally — those are operator-fill values produced by running the validation runs.

11. **Type consistency:**
    - `smoothness_lambda_d`, `smoothness_loss_d`, `smoothness_loss_per_horizon_d` — same name in Tasks 5, 6, 7 ✓
    - `smoothness_fn` — same in Tasks 5, 6 ✓
    - `last_smoothness_loss_per_horizon()` — accessor name consistent in Task 7 (steps 7.3 and 7.4) ✓
    - `--smoothness-base-lambda` — same flag name in Tasks 7, 8, 9, 10 ✓
    - `SMOOTHNESS_LAMBDA_RATIO` — same constant name in Tasks 3, 5 ✓

12. **Memory-rule compliance:**
    - `feedback_no_atomicadd`: kernel uses warp-shuffle + single-block tree-reduce ✓
    - `feedback_no_htod_htoh_only_mapped_pinned`: λ upload uses MappedF32Buffer DtoD ✓
    - `feedback_no_nvrtc`: kernel pre-compiled via build.rs ✓
    - `pearl_build_rs_rerun_if_env_changed`: KERNELS entry triggers existing rerun-if-changed glob ✓
    - `pearl_no_host_branches_in_captured_graph`: kernel launch only, no host branches inside step_batched insertion point ✓
    - `pearl_first_observation_bootstrap`: not applicable — no EMA state in this kernel
    - `pearl_one_unbounded_signal_per_reward`: not applicable — this is a loss term, not a reward; squared diffs are bounded ≤ 1 per pair
    - `feedback_no_stubs`/`no_todo_fixme`/`no_hiding`: every step ships complete code, no TODO markers, no `_` hiding ✓
    - `feedback_single_source_of_truth_no_duplicates`: smoothness lives in ONE new kernel; no v2/sigma variants ✓
    - `feedback_default_to_l40s_pool`: Task 9.2 and Task 9.3 specify `ci-training-l40s` ✓
    - `feedback_push_before_deploy`: Task 9.1 pushes before Task 9.2 deploys ✓
    - `feedback_smoke_validation_before_structural_priors`: Task 9.2 validates λ=0 wiring BEFORE Task 9.3 enables the prior ✓

All checks pass. Plan ready to execute.
