74392e3744
Root cause: weight tensors packed sequentially in the flat params buffer had non-aligned start offsets when preceding tensors had odd element counts (e.g. bias of 51 atoms = 204 bytes, 204 % 16 = 12). cublasLtMatmul with CUBLAS_COMPUTE_32F_FAST_TF32 requires 16-byte aligned buffer pointers. Fix: pad each tensor to 4-element boundary (16 bytes) in both f32_weight_ptrs_from_base (pointer computation) and compute_total_params (buffer allocation). Added align4() and padded_byte_offset() helpers, fixed shrink_perturb skip range and bottleneck gradient offset. Switched compute type: CUBLAS_COMPUTE_32F → CUBLAS_COMPUTE_32F_FAST_TF32 (forward + backward). Explicit TF32 tensor core path, required by cuBLAS 13.0 on H100 SM90. Deleted dead bf16_weight_ptrs function. 19/19 smoke tests pass on RTX 3050. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
28 lines
912 B
Docker
28 lines
912 B
Docker
# Generic GPU runtime base image for Foxhunt training pods
|
|
# Contains: CUDA 13.0 + NVRTC + rclone (no cuDNN — zero conv ops in codebase)
|
|
# Binaries are fetched by initContainer at pod startup, not baked into this image.
|
|
# Rebuild: only on CUDA version bump
|
|
|
|
FROM nvidia/cuda:13.0.0-runtime-ubuntu24.04
|
|
|
|
ENV DEBIAN_FRONTEND=noninteractive
|
|
|
|
RUN apt-get update && apt-get install -y --no-install-recommends \
|
|
ca-certificates \
|
|
libssl3t64 \
|
|
curl \
|
|
unzip \
|
|
cuda-nvrtc-12-9 \
|
|
&& curl -fsSL https://downloads.rclone.org/v1.69.1/rclone-v1.69.1-linux-amd64.zip -o /tmp/rclone.zip \
|
|
&& unzip -j /tmp/rclone.zip '*/rclone' -d /usr/local/bin/ \
|
|
&& rm /tmp/rclone.zip \
|
|
&& apt-get purge -y unzip \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
RUN usermod -l foxhunt ubuntu && groupmod -n foxhunt ubuntu
|
|
|
|
ENV NVIDIA_VISIBLE_DEVICES=all
|
|
ENV NVIDIA_DRIVER_CAPABILITIES=compute,utility
|
|
|
|
USER foxhunt
|