- argo-train.sh: --profile flag forces multi-seed render path so the
nsys wrapper + foxhunt-training-artifacts upload step are visible in
--dry-run YAML without cluster contact (test surface).
- train-multi-seed-template.yaml: new `profile` parameter (default
"false") gates the per-(seed, fold) `nsys profile
--capture-range=cudaProfilerApi` wrapper and the `mc cp` upload to
foxhunt-training-artifacts/profiles/<sha>/. mc binary fetched
on-demand (ci-builder image lacks it). MinIO creds optional —
upload warn-skips if absent.
- Dockerfile.foxhunt-training-runtime: install nsight-systems-cli
unpinned (pinning the stale 2024.4.1.61-1 from earlier plans
breaks builds when apt index advances).
- minio.yaml: add foxhunt-training-artifacts bucket to minio-init.
- compare-nsys-profiles.py: V0 regression detector — compares
cuda_gpu_kern_sum total_ns / epoch_count between two profiles;
exits 1 on >20% slowdown. NVTX per-epoch ranges deferred to T5.
- tests/test_nsys_harness.sh: dry-run grep test — verifies both
required strings appear when --profile is set, and that the
default (no --profile) path keeps profile=false in the rendered
template.
- dqn-wire-up-audit.md: Plan 5 Task 3 row added documenting the
harness + the baseline-capture deferral to T5.
Backward compat: test_multi_seed_harness.sh from P5T1 still PASS.
Root cause: weight tensors packed sequentially in the flat params buffer
had non-aligned start offsets when preceding tensors had odd element counts
(e.g. bias of 51 atoms = 204 bytes, 204 % 16 = 12). cublasLtMatmul with
CUBLAS_COMPUTE_32F_FAST_TF32 requires 16-byte aligned buffer pointers.
Fix: pad each tensor to 4-element boundary (16 bytes) in both
f32_weight_ptrs_from_base (pointer computation) and compute_total_params
(buffer allocation). Added align4() and padded_byte_offset() helpers,
fixed shrink_perturb skip range and bottleneck gradient offset.
Switched compute type: CUBLAS_COMPUTE_32F → CUBLAS_COMPUTE_32F_FAST_TF32
(forward + backward). Explicit TF32 tensor core path, required by cuBLAS
13.0 on H100 SM90.
Deleted dead bf16_weight_ptrs function.
19/19 smoke tests pass on RTX 3050.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Align all 4 Docker images with local dev environment:
- ci-builder: CUDA 12.4.1/Ubuntu 22.04 → 12.9.1/Ubuntu 24.04
- ci-builder-cpu: Debian bookworm → Ubuntu 24.04 (+ explicit rustup)
- foxhunt-runtime: Debian bookworm → Ubuntu 24.04
- foxhunt-training-runtime: CUDA 12.6.3 → 12.9.1, nvrtc 12-6 → 12-9
All images now have glibc 2.39, matching the local build machine.
Binaries compiled locally or in CI will run in any of these containers
without GLIBC_2.3x version mismatch errors.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three fixes validated by 20/20 hyperopt trials on H100 (zero OOM):
1. Workspace default-features: ml-core, ml-dqn, ml-ppo, ml-supervised
workspace deps now have default-features=false. Prevents cudarc
(which requires nvcc) from leaking into CPU service builds via
Cargo feature unification. CI compile-services was failing with
"Failed to execute nvcc: No such file or directory" (exit 101).
2. BF16 comparison fix: Candle's gt()/le() don't support BF16 operands.
Cast ADX/CUSUM features to F32 before threshold comparison in
regime classification. Previous approach (cast threshold to BF16)
failed due to Candle broadcast_as reverting dtype.
3. CI pipeline: expand ML change detection to all 14 sub-crates,
add component:compile labels for sccache network policy matching,
bump training runtime to CUDA 12.6 + Ubuntu 24.04 (glibc 2.39).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
