diff --git a/docs/dqn-wire-up-audit.md b/docs/dqn-wire-up-audit.md index a941bbc32..54375ecf2 100644 --- a/docs/dqn-wire-up-audit.md +++ b/docs/dqn-wire-up-audit.md @@ -8723,3 +8723,26 @@ Per `pearl_canary_input_freshness_launch_order.md`: this launch order is the smo 2. **Hold-cost consumer site found at exactly the canonical site.** `experience_kernels.cu:3089` (segment_complete branch) was the source of the `cost=0.006251` diagnostic value emitted at HEALTH_DIAG[2] in pfh9n. The 2 sibling sites (3553, 3617) cover the per-bar positioned-Hold and per-bar flat-Hold branches. All three are migrated atomically with the same fallback semantics so the per-bar Hold cost subtraction is consistent across all 3 reward composition paths. 3. **Producer launch ordering.** The SP16-P1 producer fires AFTER the per-step `hold_rate_observer` chain via the `if (collector.executed) {…}` block in training_loop.rs. The new SP16-P2 producer fires immediately after SP16-P1 inside the same block, so the launch ordering is correct by construction. The `pearl_canary_input_freshness_launch_order.md` invariant is preserved. + +## Build infra: ensure-binary cache key bug (2026-05-08) + +Discovered during SP16 validation (4gb88 → 8mzkj forced switch from L40S to H100). + +The build.rs fix landed in commit `98a81981a` made Cargo's incremental +build properly arch-aware (added `cargo:rerun-if-env-changed=CUDA_COMPUTE_CAP` +to 6 build.rs files). However, the Argo `ensure-binary` step caches the +compiled binary at `/data/bin/$SHORT_SHA/` keyed by **commit SHA only**. + +When a workflow on L40S (sm=89) populates the cache for a given SHA, then +a subsequent workflow on H100 (sm=90) for the SAME SHA cache-hits and +reuses the SM 89 binary. Training fails at runtime with: + +``` +rmsnorm cubin load: DriverError(CUDA_ERROR_NO_BINARY_FOR_GPU) +``` + +**Workaround used**: bump SHA with no-op commit to force fresh build. + +**Proper fix**: extend cache key in `scripts/argo-train.sh` (or workflow +template) to include `cuda-compute-cap`, e.g., `/data/bin/$SHORT_SHA-sm$CC/`. +Filed as future infrastructure cleanup.