From 6dce4ac28b9866862e6c2ba0842e7dd20eaaf368 Mon Sep 17 00:00:00 2001
From: jgrusewski <jgrusewski@localhost>
Date: Fri, 8 May 2026 17:35:37 +0200
Subject: [PATCH] docs(audit): note ensure-binary cache key bug for cross-arch
 resubmits
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Discovered during SP16 validation: ensure-binary caches by SHA only,
so cross-pool resubmissions (L40S → H100) cache-hit the wrong-arch
binary and fail with CUDA_ERROR_NO_BINARY_FOR_GPU on rmsnorm.

Build.rs rerun-if-env-changed fix (98a81981a) handles Cargo's
incremental cache, but not the Argo PVC binary cache. Documented for
future infrastructure cleanup.

Bumping SHA to force fresh SM 90 build for SP16 H100 validation.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---
 docs/dqn-wire-up-audit.md | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/docs/dqn-wire-up-audit.md b/docs/dqn-wire-up-audit.md
index a941bbc32..54375ecf2 100644
--- a/docs/dqn-wire-up-audit.md
+++ b/docs/dqn-wire-up-audit.md
@@ -8723,3 +8723,26 @@ Per `pearl_canary_input_freshness_launch_order.md`: this launch order is the smo
 2. **Hold-cost consumer site found at exactly the canonical site.** `experience_kernels.cu:3089` (segment_complete branch) was the source of the `cost=0.006251` diagnostic value emitted at HEALTH_DIAG[2] in pfh9n. The 2 sibling sites (3553, 3617) cover the per-bar positioned-Hold and per-bar flat-Hold branches. All three are migrated atomically with the same fallback semantics so the per-bar Hold cost subtraction is consistent across all 3 reward composition paths.
 
 3. **Producer launch ordering.** The SP16-P1 producer fires AFTER the per-step `hold_rate_observer` chain via the `if (collector.executed) {…}` block in training_loop.rs. The new SP16-P2 producer fires immediately after SP16-P1 inside the same block, so the launch ordering is correct by construction. The `pearl_canary_input_freshness_launch_order.md` invariant is preserved.
+
+## Build infra: ensure-binary cache key bug (2026-05-08)
+
+Discovered during SP16 validation (4gb88 → 8mzkj forced switch from L40S to H100).
+
+The build.rs fix landed in commit `98a81981a` made Cargo's incremental
+build properly arch-aware (added `cargo:rerun-if-env-changed=CUDA_COMPUTE_CAP`
+to 6 build.rs files). However, the Argo `ensure-binary` step caches the
+compiled binary at `/data/bin/$SHORT_SHA/` keyed by **commit SHA only**.
+
+When a workflow on L40S (sm=89) populates the cache for a given SHA, then
+a subsequent workflow on H100 (sm=90) for the SAME SHA cache-hits and
+reuses the SM 89 binary. Training fails at runtime with:
+
+```
+rmsnorm cubin load: DriverError(CUDA_ERROR_NO_BINARY_FOR_GPU)
+```
+
+**Workaround used**: bump SHA with no-op commit to force fresh build.
+
+**Proper fix**: extend cache key in `scripts/argo-train.sh` (or workflow
+template) to include `cuda-compute-cap`, e.g., `/data/bin/$SHORT_SHA-sm$CC/`.
+Filed as future infrastructure cleanup.