fix(rl): mapped-pinned for all R7d/R8 CPU↔GPU paths + diag JSONL + guard

Two concerns in one commit since they're entangled:

## 1. feedback_no_htod_htoh_only_mapped_pinned violations

R7d (PER push/sample) + R8 (CLI binary) + the new per-step diag dump
shipped with 8 raw `stream.memcpy_htod` / `stream.memcpy_dtoh` calls.
The rule is explicit: "mapped-pinned only for CPU↔GPU; tests not
exempt." A raw `stream.memcpy_*` on a regular `&[T]` / `&mut [T]` is
NOT mapped-pinned — the source/dest slice isn't page-locked, so the
CUDA driver does an internal blocking HtoD/DtoH that stalls the
stream.

Refactored all 8 violations to use the mapped-pinned + DtoD pattern
(cuMemHostAlloc DEVICEMAP — host writes via `host_ptr`, kernel reads
`dev_ptr`, DtoD between them via `cudarc::driver::result::memcpy_dtod_async`).
New shared helpers in `trainer/integrated.rs`:

  * `read_slice_i32_d` — DtoH for `i32` device buffers via
    `MappedI32Buffer` staging. Counterpart to the existing
    `read_slice_d` (f32 version).
  * `write_slice_f32_d` — CPU→GPU upload for `f32` via
    `MappedF32Buffer.write_from_slice` + DtoD into destination.
  * `write_slice_i32_d` — CPU→GPU upload for `i32` via
    `MappedI32Buffer.host_slice_mut().copy_from_slice` + DtoD.

`pub fn` wrappers (`read_slice_*_d_pub`) expose the f32/i32 helpers
to the CLI binary so the per-step diag DtoH uses the same canonical
pattern.

Call-site refactors:

  * `push_to_replay`: 4× `stream.memcpy_dtoh` → `read_slice_*_d`.
  * `sample_and_gather`: 3× `stream.memcpy_htod` → `write_slice_*_d`.
  * `step_with_lobsim` pre-PER ISV refresh: raw `memcpy_dtoh` →
    `read_slice_d` (424 floats per step).
  * `step_with_lobsim` post-Q PER priority TD readback: raw
    `memcpy_dtoh` → `read_slice_d` (b_size floats per step).
  * `step_synthetic` ISV mirror refresh: raw `memcpy_dtoh` →
    `read_slice_d` (pre-existing pre-R9 violation; fixed in the
    same commit since it's the same pattern in the same file).
  * Init-time (one-shot) `prng_state` upload: raw `memcpy_htod` →
    inline mapped-pinned DtoD (custom because cast through i32 for
    the u32 buffer).
  * Init-time (one-shot) `atom_supports` upload: raw `memcpy_htod`
    → `write_slice_f32_d`.
  * `examples/alpha_rl_train.rs` per-step diag DtoH (3 calls) →
    `read_slice_*_d_pub`.

## 2. Pre-commit guard gap — diff-aware HtoD/DtoH check

The existing GPU hot-path guard (`scripts/gpu-hotpath-guard.sh`)
EXPLICITLY skips memcpy_htod/dtoh on the assumption that such calls
only appear in `cuda_pipeline/` (where mapped-pinned is the
convention). That assumption was falsified by R7d/R8 — the guard
shipped 8 violations green.

Added `check_no_raw_htod_dtoh` to `scripts/pre-commit-hook.sh` (the
real file behind the `.git/hooks/pre-commit` symlink). The check is
DIFF-AWARE: it greps only the `+` lines of `git diff --cached -U0`,
so pre-existing violations elsewhere (143 sites across the codebase)
don't block commits touching unrelated files. NEW additions of
`\.memcpy_(htod|dtoh)\(` are flagged with a clear error pointing at
the mapped-pinned alternative. Suppress per-line with `// gpu-ok:
<reason>` (same convention as the existing guards).

Pre-existing violations in `ml-alpha/src/aux_heads.rs`,
`mamba2_block.rs`, `cfc/`, `data/`, etc. are a separate cleanup —
not blocked by this commit's check because the diff-aware filter
ignores anything that was already on `HEAD~1`.

## Verified gates (post-fix, local sm_86)

  G1  isv_bootstrap                 unchanged
  G3  controllers_emit              unchanged
  G4  target_soft_update            unchanged
  G6  r7d_per_wiring                unchanged (PER round-trips all
                                     mapped-pinned now)
  R3  ema/advantage (3 tests)       unchanged
  R4  action kernels (3 tests)      unchanged
  end integrated_trainer_smoke      unchanged

Mapped-pinned is semantically equivalent to raw memcpy_htod/dtoh —
just routed through page-locked staging so the driver doesn't have
to do its own internal pinning. Behaviour identical; the cost shifts
from "driver hidden HtoD per call" to "mapped-pinned alloc + DtoD
per call." For the smoke (b_size=1, 1000 steps), the cost difference
is in the microseconds.

## Per-step diag JSONL (separate concern, same commit)

Added `--diag-jsonl <PATH>` flag to `alpha_rl_train.rs` (default:
`<out>/diag.jsonl`). After each `step_with_lobsim`, writes one JSON
record capturing:

  * step number, elapsed wall time
  * all 5 head losses + λs
  * all 7 RL controller outputs (γ τ ε coef n_roll per_α scale)
  * all 5 per-head learning rates (lr_bce/q/pi/v/aux)
  * all 7 EMA inputs the controllers consume
  * replay buffer length
  * per-step reward stats (sum, max, min, abs_max)
  * per-step done count
  * per-step action histogram (9 action classes)

Critical for cluster smoke debugging — the prior CLI only flushed
an `eprintln` progress line every N steps (default 100), making
in-flight controller drift / replay stagnation / reward explosion
invisible until they produced a NaN abort. The JSONL is line-
buffered + flushed every `log_every` steps so `tail -f` shows
progress live.

The stderr progress line is also beefed up to include γ / ε / per_α /
reward_scale / dones / rew_sum at each tick so a casual `argo logs`
inspection sees the controller behaviour without parsing JSONL.

## Why R9 cluster submission needs this

Without the diag dump, an R9 1000-step smoke is "blind" — only the
final summary tells us what happened. With the dump, post-hoc
analysis can answer:
  * Did the controllers adapt or stay at bootstrap?
  * Did the reward scale stabilise or saturate?
  * Did the PER buffer fill?
  * Was the action histogram dominated by any one action?
  * Where did the per-head losses converge to?

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-23 14:14:55 +02:00
parent fd415d9b17
commit ce1e13519b
3 changed files with 393 additions and 46 deletions

View File

@@ -227,10 +227,67 @@ check_no_isv_migrations() {
fi
}
# Diff-aware guard: any NEW `stream.memcpy_htod(` or `stream.memcpy_dtoh(`
# line added in this commit is a violation of
# `feedback_no_htod_htoh_only_mapped_pinned`. The only permitted CPU↔GPU
# path is mapped-pinned staging (cuMemHostAlloc DEVICEMAP — host writes
# via `host_ptr`, kernels read `dev_ptr`, DtoD copy between them via
# `cudarc::driver::result::memcpy_dtod_async`).
#
# The existing gpu-hotpath-guard.sh explicitly skips memcpy_htod/dtoh
# (see scripts/gpu-hotpath-guard.sh:103-105) on the assumption that
# such calls only appear in cuda_pipeline/ where mapped-pinned is the
# convention. That assumption was falsified in 2026-05-23 when the R7d
# PER push/sample + R8 CLI diag landed 8 raw `stream.memcpy_htod/dtoh`
# calls in ml-alpha/src/trainer/ and ml-alpha/examples/ — the guard
# was deliberately blind to them and they shipped.
#
# This diff-aware check fires on NEW + lines only, so pre-existing
# violations elsewhere don't block commits touching unrelated files.
# Suppress per-line with `// gpu-ok: <reason>`.
check_no_raw_htod_dtoh() {
local staged
staged=$(git diff --cached --name-only --diff-filter=ACM | grep '\.rs$' || true)
if [ -z "$staged" ]; then return 0; fi
local bad=""
while IFS= read -r f; do
[ -z "$f" ] && continue
local hits
hits=$(git diff --cached -U0 -- "$f" 2>/dev/null \
| grep -E '^\+[^+]' \
| grep -E '\.memcpy_(htod|dtoh)\(' \
| grep -v 'gpu-ok:' \
|| true)
if [ -n "$hits" ]; then
bad+="${bad:+$'\n'}$f:"$'\n'"$hits"
fi
done <<< "$staged"
if [ -n "$bad" ]; then
echo "❌ feedback_no_htod_htoh_only_mapped_pinned violation: raw"
echo " stream.memcpy_htod/dtoh on a regular &[T]/&mut [T] is"
echo " forbidden. The source/dest slice is not page-locked, so the"
echo " CUDA driver does an internal blocking HtoD/DtoH copy."
echo " Use the mapped-pinned pattern instead: MappedF32Buffer /"
echo " MappedI32Buffer (host_ptr write + dev_ptr DtoD via"
echo " cudarc::driver::result::memcpy_dtod_async). See the"
echo " read_slice_d / write_slice_*_d helpers in"
echo " crates/ml-alpha/src/trainer/integrated.rs for the canonical"
echo " reference implementation."
echo ""
echo " Lines flagged (NEW additions only; pre-existing violations"
echo " are a separate cleanup tracked outside this guard):"
echo "$bad" | sed 's/^/ /'
return 1
fi
}
check_audit_doc_updates || exit 1
check_no_todo_fixme || exit 1
check_no_isv_migrations || exit 1
check_no_dtod_via_pinned || exit 1
check_no_raw_htod_dtoh || exit 1
check_sp18_consumer_audit || exit 1
echo "✅ All pre-commit checks passed!"