5a27bde9dd
The GPU experience collection kernels (dqn_experience_kernel.cu, ppo_experience_kernel.cu) use cudarc::nvrtc::compile_ptx() at runtime. Without libnvrtc.so the kernel compile fails silently and falls back to CPU experience collection. Adding cuda-nvrtc-12-4 (~30MB) enables full GPU-accelerated experience collection. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
54 lines
1.8 KiB
Docker
54 lines
1.8 KiB
Docker
# Unified training runtime image — all 10 models (RL + supervised) with CUDA GPU pipeline
|
|
# Binaries are compiled in CI compile-training job with --features ml/cuda
|
|
# cuDNN required: candle CUDA backend links against libcudnn for kernel ops
|
|
# NVRTC required: GPU experience collection kernels are JIT-compiled at runtime
|
|
# Usage: kaniko --context dir://build-out/training --dockerfile Dockerfile.training
|
|
# Run: docker run --gpus all training train_baseline_rl --model dqn [args...]
|
|
|
|
FROM nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04
|
|
|
|
ENV DEBIAN_FRONTEND=noninteractive
|
|
|
|
RUN apt-get update && apt-get install -y --no-install-recommends \
|
|
ca-certificates \
|
|
libssl3 \
|
|
curl \
|
|
unzip \
|
|
cuda-nvrtc-12-4 \
|
|
&& curl -fsSL https://downloads.rclone.org/v1.69.1/rclone-v1.69.1-linux-amd64.zip -o /tmp/rclone.zip \
|
|
&& unzip -j /tmp/rclone.zip '*/rclone' -d /usr/local/bin/ \
|
|
&& chmod +x /usr/local/bin/rclone \
|
|
&& rm /tmp/rclone.zip \
|
|
&& apt-get purge -y unzip \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
RUN groupadd -g 1000 foxhunt \
|
|
&& useradd -u 1000 -g foxhunt -m -s /bin/false foxhunt
|
|
|
|
COPY train_baseline_rl \
|
|
train_baseline_supervised \
|
|
evaluate_baseline \
|
|
evaluate_supervised \
|
|
hyperopt_baseline_rl \
|
|
hyperopt_baseline_supervised \
|
|
training_uploader \
|
|
/usr/local/bin/
|
|
|
|
RUN chmod +x /usr/local/bin/train_baseline_rl \
|
|
/usr/local/bin/train_baseline_supervised \
|
|
/usr/local/bin/evaluate_baseline \
|
|
/usr/local/bin/evaluate_supervised \
|
|
/usr/local/bin/hyperopt_baseline_rl \
|
|
/usr/local/bin/hyperopt_baseline_supervised \
|
|
/usr/local/bin/training_uploader
|
|
|
|
# CUDA runtime environment
|
|
ENV NVIDIA_VISIBLE_DEVICES=all
|
|
ENV NVIDIA_DRIVER_CAPABILITIES=compute,utility
|
|
|
|
USER foxhunt
|
|
|
|
WORKDIR /data
|
|
|
|
CMD ["echo", "Specify training command via K8s Job args"]
|