86ed7af58f
Added is_final parameter to checkpoint callback to distinguish final
checkpoints from regular epoch checkpoints. Early stopping now saves
as dqn_final_epoch{N}.safetensors instead of dqn_epoch_{N}.safetensors.
Changes:
- Updated callback signature: Fn(usize, Vec<u8>, bool)
- Early stopping passes is_final=true
- Regular checkpoints pass is_final=false
- Callback uses final naming when is_final=true
Fixes checkpoint overwrite bug where final model was indistinguishable
from regular epoch checkpoints.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
165 lines
7.3 KiB
Docker
165 lines
7.3 KiB
Docker
# =============================================================================
|
|
# RUNPOD DEPLOYMENT DOCKERFILE - VOLUME MOUNT ARCHITECTURE
|
|
# =============================================================================
|
|
# Purpose: Provides CUDA 13.0 + cuDNN development environment for pre-built binaries
|
|
# Size: ~8.4GB (includes CUDA 13.0 development libraries)
|
|
# Build time: ~2 minutes (vs 20+ minutes with compilation)
|
|
#
|
|
# Architecture:
|
|
# - Docker image: CUDA runtime environment + entrypoint script
|
|
# - Training binaries: Pre-uploaded to Runpod Network Volume at /runpod-volume/binaries/
|
|
# - Test data: Pre-uploaded to Runpod Network Volume at /runpod-volume/test_data/
|
|
# - Credentials: Pre-uploaded to Runpod Network Volume at /runpod-volume/.env
|
|
#
|
|
# NO COMPILATION IN DOCKER - Binaries are pre-built locally with CUDA support
|
|
# =============================================================================
|
|
|
|
# Base image: CUDA 13.0 devel on Ubuntu 24.04
|
|
# CRITICAL FIX: Binary compiled with CUDA 13.0 locally, requires libcublas.so.13
|
|
# CUDA 13.0 provides libcublas.so.13 (exact match with local compilation environment)
|
|
# Includes: libcuda.so.1, libcurand.so.10, libcublas.so.13, libcublasLt.so.13
|
|
# Note: Using Ubuntu 24.04 for GLIBC 2.39 (matches local build environment)
|
|
# Note: Using 'devel' instead of 'runtime' because our binaries need libcublas/libcublasLt
|
|
FROM nvidia/cuda:13.0.0-devel-ubuntu24.04
|
|
|
|
# Prevent interactive prompts during apt installations
|
|
ENV DEBIAN_FRONTEND=noninteractive
|
|
|
|
# Install minimal runtime dependencies
|
|
# Runtime dependencies identified from ldd output:
|
|
# - libcuda.so.1 (from base image)
|
|
# - libcurand.so.10 (from base image)
|
|
# - libcublas.so.13 (from CUDA 12.2 base image)
|
|
# - libcublasLt.so.13 (from CUDA 12.2 base image)
|
|
# - libcudnn.so.8 (installed below, compatible with CUDA 12.2)
|
|
# - libstdc++6, libgcc-s1 (C++ runtime, from base)
|
|
# - ca-certificates (HTTPS support)
|
|
RUN apt-get update && apt-get install -y \
|
|
ca-certificates \
|
|
wget \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
# Install cuDNN 9 for CUDA 13.0 (matches binary compilation environment)
|
|
# Note: cuDNN 9 is standard for CUDA 13.x, provides libcublas.so.13
|
|
RUN apt-get update && apt-get install -y \
|
|
libcudnn9-cuda-13 \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
# Set CUDA environment variables (for runtime library loading)
|
|
ENV CUDA_HOME=/usr/local/cuda
|
|
ENV PATH="${CUDA_HOME}/bin:${PATH}"
|
|
ENV LD_LIBRARY_PATH="${CUDA_HOME}/lib64:${LD_LIBRARY_PATH}"
|
|
|
|
# NVIDIA runtime configuration
|
|
ENV NVIDIA_VISIBLE_DEVICES=all
|
|
ENV NVIDIA_DRIVER_CAPABILITIES=compute,utility
|
|
ENV CUDA_VISIBLE_DEVICES=0
|
|
|
|
# Rust runtime environment
|
|
ENV RUST_BACKTRACE=1
|
|
ENV RUST_LOG=info
|
|
|
|
# Copy entrypoint scripts (self-terminate wrapper + generic base)
|
|
COPY entrypoint-generic.sh /entrypoint-generic.sh
|
|
COPY entrypoint-self-terminate.sh /entrypoint.sh
|
|
RUN chmod +x /entrypoint-generic.sh /entrypoint.sh
|
|
|
|
# Create workspace directory
|
|
WORKDIR /workspace
|
|
|
|
# Health check to verify GPU is accessible
|
|
HEALTHCHECK --interval=60s --timeout=10s --start-period=30s --retries=3 \
|
|
CMD nvidia-smi || exit 1
|
|
|
|
# Entrypoint executes training binary from volume mount
|
|
# NO DOWNLOADS - binaries and data are already on the mounted Runpod Network Volume
|
|
# Override binary via environment variable: BINARY_NAME (default: train_tft_parquet)
|
|
ENTRYPOINT ["/entrypoint.sh"]
|
|
|
|
# Default CMD shows help (deployment script overrides this with dockerStartCmd)
|
|
CMD ["--help"]
|
|
|
|
# =============================================================================
|
|
# BUILD AND RUN INSTRUCTIONS - VOLUME MOUNT ARCHITECTURE
|
|
# =============================================================================
|
|
#
|
|
# 1. BUILD IMAGE (minimal runtime, NO binaries or data embedded):
|
|
# docker build -f Dockerfile.runpod -t jgrusewski/foxhunt:latest .
|
|
# # Image size: ~2GB (CUDA runtime only)
|
|
# # Build time: ~2 minutes
|
|
#
|
|
# 2. PUSH TO DOCKER HUB:
|
|
# docker login # Login to Docker Hub as jgrusewski
|
|
# docker push jgrusewski/foxhunt:latest
|
|
# # IMPORTANT: Set repository to PRIVATE in Docker Hub settings
|
|
#
|
|
# 3. RUNPOD DEPLOYMENT (binaries and data already uploaded to volume):
|
|
# - GPU: Tesla V100-PCIE-16GB (16GB VRAM, $0.29/hr)
|
|
# - Docker Image: jgrusewski/foxhunt:latest (PRIVATE, requires Docker Hub auth)
|
|
# - Volume Mount: Select Runpod Network Volume, mount at /runpod-volume
|
|
# - Environment Variables:
|
|
# * BINARY_NAME=train_tft_parquet (default, or train_mamba2_parquet/train_dqn/train_ppo)
|
|
# * RUST_LOG=info (logging level)
|
|
#
|
|
# 4. TRAIN DIFFERENT MODELS (all binaries on mounted volume):
|
|
# Override BINARY_NAME environment variable:
|
|
# - TFT: BINARY_NAME=train_tft_parquet
|
|
# - MAMBA-2: BINARY_NAME=train_mamba2_parquet
|
|
# - DQN: BINARY_NAME=train_dqn
|
|
# - PPO: BINARY_NAME=train_ppo
|
|
#
|
|
# 5. AVAILABLE DATA ON MOUNTED VOLUME (pre-uploaded, NO download required):
|
|
# /runpod-volume/
|
|
# ├── .env (512B, credentials, chmod 600)
|
|
# ├── binaries/ (77MB total)
|
|
# │ ├── train_tft_parquet (23MB, TFT-225 features)
|
|
# │ ├── train_mamba2_parquet (22MB, MAMBA-2 model)
|
|
# │ ├── train_dqn (22MB, Deep Q-Network)
|
|
# │ └── train_ppo (13MB, Proximal Policy Optimization)
|
|
# ├── test_data/ (14MB total, 9 Parquet files)
|
|
# │ ├── ES_FUT_180d.parquet (2.9MB, 180 days)
|
|
# │ ├── NQ_FUT_180d.parquet (4.4MB, 180 days)
|
|
# │ ├── 6E_FUT_180d.parquet (2.8MB, 180 days)
|
|
# │ ├── ZN_FUT_90d.parquet (2.8MB, 90 days)
|
|
# │ └── [5 more small test files]
|
|
# └── models/ (Empty, populated by training runs)
|
|
#
|
|
# 6. UPDATING BINARIES (NO Docker rebuild required):
|
|
# # Simply upload new binary to Runpod Network Volume via S3 API
|
|
# # Next pod start will use updated binary automatically
|
|
# # Instant deployment (seconds, not minutes)
|
|
#
|
|
# =============================================================================
|
|
# OPTIMIZATION NOTES - VOLUME MOUNT ARCHITECTURE
|
|
# =============================================================================
|
|
#
|
|
# Memory Optimization:
|
|
# - Runtime-only image reduces size from ~10GB to ~2GB
|
|
# - NO binaries or data embedded (stored on mounted volume)
|
|
# - Only runtime CUDA libraries included (no build tools, no Rust)
|
|
# - Minimal image = faster pod startup (~30s vs 2-3min)
|
|
#
|
|
# Build Speed:
|
|
# - No compilation = 2 minute build (vs 20+ minutes)
|
|
# - Image build is one-time only (binaries update via volume upload)
|
|
# - No Docker rebuild needed for code changes (instant deployment)
|
|
#
|
|
# Deployment Speed:
|
|
# - Docker image: Build once, use forever (~2GB, CUDA runtime only)
|
|
# - Binary updates: Upload to volume (seconds, NO rebuild)
|
|
# - Pod startup: ~30 seconds (volume already mounted)
|
|
# - Total deployment time: <60 seconds for new binary
|
|
#
|
|
# GPU Compatibility:
|
|
# - CUDA 13.0 compatible with Tesla V100, RTX 4090, A100, H100
|
|
# - cuDNN 9 for optimal neural network performance
|
|
# - Target: Tesla V100-PCIE-16GB (16GB VRAM, $0.29/hr)
|
|
#
|
|
# Security:
|
|
# - PRIVATE Docker Hub repository required
|
|
# - PRIVATE Runpod Network Volume (binaries + data isolated per account)
|
|
# - No sensitive data in image (only CUDA runtime)
|
|
# - Minimal attack surface (runtime-only image, no build tools)
|
|
#
|
|
# =============================================================================
|