c731bef759
RL models (DQN/PPO) only need basic CUDA runtime (~2GB), while supervised models need cuDNN (~4GB). Splitting saves ~2GB pull time per RL job. Rename the L4 GPU pool from ci-compile to ci-rl to reflect its actual use. Add DaemonSet image pre-puller to cache training images on GPU nodes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
46 lines
1.6 KiB
Docker
46 lines
1.6 KiB
Docker
# Supervised training runtime image — TFT, Mamba2, TGGN, TLOB, Liquid, KAN, xLSTM, Diffusion (~4GB)
|
|
# Includes cuDNN for deep neural network operations (convolutions, attention)
|
|
# Binaries are compiled in CI compile-training job with --features ml/cuda
|
|
# Usage: kaniko --context dir://build-out/training --dockerfile Dockerfile.training-supervised
|
|
# Run: docker run --gpus all training-supervised train_baseline_supervised --model tft [args...]
|
|
|
|
FROM nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04
|
|
|
|
ENV DEBIAN_FRONTEND=noninteractive
|
|
|
|
RUN apt-get update && apt-get install -y --no-install-recommends \
|
|
ca-certificates \
|
|
libssl3 \
|
|
curl \
|
|
unzip \
|
|
&& curl -fsSL https://downloads.rclone.org/v1.69.1/rclone-v1.69.1-linux-amd64.zip -o /tmp/rclone.zip \
|
|
&& unzip -j /tmp/rclone.zip '*/rclone' -d /usr/local/bin/ \
|
|
&& chmod +x /usr/local/bin/rclone \
|
|
&& rm /tmp/rclone.zip \
|
|
&& apt-get purge -y unzip \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
RUN groupadd -g 1000 foxhunt \
|
|
&& useradd -u 1000 -g foxhunt -m -s /bin/false foxhunt
|
|
|
|
COPY train_baseline_supervised \
|
|
evaluate_supervised \
|
|
hyperopt_baseline_supervised \
|
|
training_uploader \
|
|
/usr/local/bin/
|
|
|
|
RUN chmod +x /usr/local/bin/train_baseline_supervised \
|
|
/usr/local/bin/evaluate_supervised \
|
|
/usr/local/bin/hyperopt_baseline_supervised \
|
|
/usr/local/bin/training_uploader
|
|
|
|
# CUDA runtime environment
|
|
ENV NVIDIA_VISIBLE_DEVICES=all
|
|
ENV NVIDIA_DRIVER_CAPABILITIES=compute,utility
|
|
|
|
USER foxhunt
|
|
|
|
WORKDIR /data
|
|
|
|
CMD ["echo", "Specify training command via K8s Job args"]
|