gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(audit): note ensure-binary cache key bug for cross-arch resubmits

Browse Source 
Discovered during SP16 validation: ensure-binary caches by SHA only,
so cross-pool resubmissions (L40S → H100) cache-hit the wrong-arch
binary and fail with CUDA_ERROR_NO_BINARY_FOR_GPU on rmsnorm.

Build.rs rerun-if-env-changed fix (98a81981a) handles Cargo's
incremental cache, but not the Argo PVC binary cache. Documented for
future infrastructure cleanup.

Bumping SHA to force fresh SM 90 build for SP16 H100 validation.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-08 17:35:37 +02:00
parent 1a3bcf97b8
commit 6dce4ac28b
1 changed files with 23 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

23
docs/dqn-wire-up-audit.md
View File

@@ -8723,3 +8723,26 @@ Per `pearl_canary_input_freshness_launch_order.md`: this launch order is the smo
2. **Hold-cost consumer site found at exactly the canonical site.** `experience_kernels.cu:3089` (segment_complete branch) was the source of the `cost=0.006251` diagnostic value emitted at HEALTH_DIAG[2] in pfh9n. The 2 sibling sites (3553, 3617) cover the per-bar positioned-Hold and per-bar flat-Hold branches. All three are migrated atomically with the same fallback semantics so the per-bar Hold cost subtraction is consistent across all 3 reward composition paths.
3. **Producer launch ordering.** The SP16-P1 producer fires AFTER the per-step `hold_rate_observer` chain via the `if (collector.executed) {…}` block in training_loop.rs. The new SP16-P2 producer fires immediately after SP16-P1 inside the same block, so the launch ordering is correct by construction. The `pearl_canary_input_freshness_launch_order.md` invariant is preserved.
## Build infra: ensure-binary cache key bug (2026-05-08)
Discovered during SP16 validation (4gb88 → 8mzkj forced switch from L40S to H100).
The build.rs fix landed in commit `98a81981a` made Cargo's incremental
build properly arch-aware (added `cargo:rerun-if-env-changed=CUDA_COMPUTE_CAP`
to 6 build.rs files). However, the Argo `ensure-binary` step caches the
compiled binary at `/data/bin/$SHORT_SHA/` keyed by **commit SHA only**.
When a workflow on L40S (sm=89) populates the cache for a given SHA, then
a subsequent workflow on H100 (sm=90) for the SAME SHA cache-hits and
reuses the SM 89 binary. Training fails at runtime with:
```
rmsnorm cubin load: DriverError(CUDA_ERROR_NO_BINARY_FOR_GPU)
```
**Workaround used**: bump SHA with no-op commit to force fresh build.
**Proper fix**: extend cache key in `scripts/argo-train.sh` (or workflow
template) to include `cuda-compute-cap`, e.g., `/data/bin/$SHORT_SHA-sm$CC/`.
Filed as future infrastructure cleanup.