2606506cd8
- argo-train.sh: --profile flag forces multi-seed render path so the nsys wrapper + foxhunt-training-artifacts upload step are visible in --dry-run YAML without cluster contact (test surface). - train-multi-seed-template.yaml: new `profile` parameter (default "false") gates the per-(seed, fold) `nsys profile --capture-range=cudaProfilerApi` wrapper and the `mc cp` upload to foxhunt-training-artifacts/profiles/<sha>/. mc binary fetched on-demand (ci-builder image lacks it). MinIO creds optional — upload warn-skips if absent. - Dockerfile.foxhunt-training-runtime: install nsight-systems-cli unpinned (pinning the stale 2024.4.1.61-1 from earlier plans breaks builds when apt index advances). - minio.yaml: add foxhunt-training-artifacts bucket to minio-init. - compare-nsys-profiles.py: V0 regression detector — compares cuda_gpu_kern_sum total_ns / epoch_count between two profiles; exits 1 on >20% slowdown. NVTX per-epoch ranges deferred to T5. - tests/test_nsys_harness.sh: dry-run grep test — verifies both required strings appear when --profile is set, and that the default (no --profile) path keeps profile=false in the rendered template. - dqn-wire-up-audit.md: Plan 5 Task 3 row added documenting the harness + the baseline-capture deferral to T5. Backward compat: test_multi_seed_harness.sh from P5T1 still PASS.
35 lines
1.3 KiB
Docker
35 lines
1.3 KiB
Docker
# Generic GPU runtime base image for Foxhunt training pods
|
|
# Contains: CUDA 13.0 + NVRTC + rclone (no cuDNN — zero conv ops in codebase)
|
|
# Binaries are fetched by initContainer at pod startup, not baked into this image.
|
|
# Rebuild: only on CUDA version bump
|
|
|
|
FROM nvidia/cuda:13.0.0-runtime-ubuntu24.04
|
|
|
|
ENV DEBIAN_FRONTEND=noninteractive
|
|
|
|
RUN apt-get update && apt-get install -y --no-install-recommends \
|
|
ca-certificates \
|
|
libssl3t64 \
|
|
curl \
|
|
unzip \
|
|
cuda-nvrtc-12-9 \
|
|
nsight-systems-cli \
|
|
&& curl -fsSL https://downloads.rclone.org/v1.69.1/rclone-v1.69.1-linux-amd64.zip -o /tmp/rclone.zip \
|
|
&& unzip -j /tmp/rclone.zip '*/rclone' -d /usr/local/bin/ \
|
|
&& rm /tmp/rclone.zip \
|
|
&& apt-get purge -y unzip \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
# Plan 5 Task 3 (A.4.1): nsys profile harness.
|
|
# `nsight-systems-cli` is installed unpinned because the CUDA 13 runtime base
|
|
# already pulls in the nsys version that matches its NVRTC stack; pinning a
|
|
# stale apt version (e.g. 2024.4.1.61-1 from older plans) causes the build to
|
|
# fail when the apt index advances. Verify post-build with `nsys --version`.
|
|
|
|
RUN usermod -l foxhunt ubuntu && groupmod -n foxhunt ubuntu
|
|
|
|
ENV NVIDIA_VISIBLE_DEVICES=all
|
|
ENV NVIDIA_DRIVER_CAPABILITIES=compute,utility
|
|
|
|
USER foxhunt
|