aac0597cd2
- Fixed DQN early stopping checkpoint naming bug (Option B)
- Added is_final: bool parameter to checkpoint callback signature
- Trainer now distinguishes final checkpoints from regular epoch checkpoints
- Final checkpoints use 'dqn_final_epoch{N}' naming convention
- Regular checkpoints use 'dqn_epoch_{N}' naming convention
- Completed comprehensive TFT OOM investigation
- Spawned 3 parallel agents for memory analysis
- Identified 16.4GB memory leak (29.7x over expected 525-550MB)
- Root causes: Attention cache bloat (960MB), gradient accumulation bug, detached tensors
- Recommended fixes: Disable cache during training, explicit tensor drops
- Created TFT_MEMORY_ANALYSIS.md, TFT_MEMORY_LEAK_ANALYSIS.md
- DQN 100-epoch training VERIFIED on Runpod RTX A4000
- Training completed successfully: 100/100 epochs
- Final checkpoint created: dqn_final_epoch100.safetensors
- Training speed: 4.8 sec/epoch (3.5x faster than baseline)
- Option B fix working perfectly
- Deployed RTX 4090 pod for TFT testing
- Pod ID: 6244yzm9hadnog
- 24GB VRAM to bypass OOM issue
- EUR-IS-1 datacenter, $0.59/hr
Files modified:
- ml/examples/train_dqn.rs (checkpoint callback signature)
- ml/src/trainers/dqn.rs (callback signature + is_final parameter)
- CLAUDE.md (compacted to ~11k chars)
Generated reports:
- TFT_MEMORY_ANALYSIS.md (15-section memory breakdown)
- TFT_MEMORY_QUICK_SUMMARY.md (executive summary)
- TFT_MEMORY_LEAK_ANALYSIS.md (5 critical leaks identified)
Co-Authored-By: Claude <noreply@anthropic.com>
195 lines
8.8 KiB
Docker
195 lines
8.8 KiB
Docker
# =============================================================================
|
|
# RUNPOD DEPLOYMENT DOCKERFILE - VOLUME MOUNT ARCHITECTURE
|
|
# =============================================================================
|
|
# Purpose: Provides CUDA 13.0 + cuDNN development environment for pre-built binaries
|
|
# Size: ~8.4GB (includes CUDA 13.0 development libraries)
|
|
# Build time: ~2 minutes (vs 20+ minutes with compilation)
|
|
#
|
|
# Architecture:
|
|
# - Docker image: CUDA runtime environment + entrypoint script
|
|
# - Training binaries: Pre-uploaded to Runpod Network Volume at /runpod-volume/binaries/
|
|
# - Test data: Pre-uploaded to Runpod Network Volume at /runpod-volume/test_data/
|
|
# - Credentials: Pre-uploaded to Runpod Network Volume at /runpod-volume/.env
|
|
#
|
|
# NO COMPILATION IN DOCKER - Binaries are pre-built locally with CUDA support
|
|
# =============================================================================
|
|
|
|
# Base image: CUDA 13.0 devel on Ubuntu 24.04
|
|
# CRITICAL FIX: Binary compiled with CUDA 13.0 locally, requires libcublas.so.13
|
|
# CUDA 13.0 provides libcublas.so.13 (exact match with local compilation environment)
|
|
# Includes: libcuda.so.1, libcurand.so.10, libcublas.so.13, libcublasLt.so.13
|
|
# Note: Using Ubuntu 24.04 for GLIBC 2.39 (matches local build environment)
|
|
# Note: Using 'devel' instead of 'runtime' because our binaries need libcublas/libcublasLt
|
|
FROM nvidia/cuda:13.0.0-devel-ubuntu24.04
|
|
|
|
# Prevent interactive prompts during apt installations
|
|
ENV DEBIAN_FRONTEND=noninteractive
|
|
|
|
# Install minimal runtime dependencies
|
|
# Runtime dependencies identified from ldd output:
|
|
# - libcuda.so.1 (from base image)
|
|
# - libcurand.so.10 (from base image)
|
|
# - libcublas.so.13 (from CUDA 12.2 base image)
|
|
# - libcublasLt.so.13 (from CUDA 12.2 base image)
|
|
# - libcudnn.so.8 (installed below, compatible with CUDA 12.2)
|
|
# - libstdc++6, libgcc-s1 (C++ runtime, from base)
|
|
# - ca-certificates (HTTPS support)
|
|
RUN apt-get update && apt-get install -y \
|
|
ca-certificates \
|
|
wget \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
# Install cuDNN 9 for CUDA 13.0 (matches binary compilation environment)
|
|
# Note: cuDNN 9 is standard for CUDA 13.x, provides libcublas.so.13
|
|
RUN apt-get update && apt-get install -y \
|
|
libcudnn9-cuda-13 \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
# Install runpodctl CLI tool for pod self-termination
|
|
# Download latest version from GitHub releases (v1.14.11 as of 2025-10-24)
|
|
# Used by entrypoint-self-terminate.sh to terminate pod after training completes
|
|
RUN apt-get update && apt-get install -y curl && \
|
|
wget -qO /tmp/runpodctl.tar.gz "https://github.com/runpod/runpodctl/releases/download/v1.14.11/runpodctl_1.14.11_linux_amd64.tar.gz" && \
|
|
tar -xzf /tmp/runpodctl.tar.gz -C /tmp && \
|
|
mv /tmp/runpodctl /usr/local/bin/runpodctl && \
|
|
chmod +x /usr/local/bin/runpodctl && \
|
|
rm -rf /tmp/runpodctl.tar.gz /var/lib/apt/lists/*
|
|
|
|
# =============================================================================
|
|
# SSH SERVER INSTALLATION FOR RUNPOD REMOTE ACCESS
|
|
# =============================================================================
|
|
# Install OpenSSH server for remote debugging and file transfers
|
|
# RunPod injects SSH public keys via PUBLIC_KEY environment variable
|
|
# Access via: ssh root@<pod-id>.ssh.runpod.io
|
|
RUN apt-get update && apt-get install -y \
|
|
openssh-server \
|
|
&& mkdir -p /var/run/sshd /root/.ssh \
|
|
&& chmod 700 /root/.ssh \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
# Configure SSH for key-only authentication (no passwords)
|
|
RUN sed -i 's/#PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config \
|
|
&& sed -i 's/#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config \
|
|
&& sed -i 's/#PubkeyAuthentication yes/PubkeyAuthentication yes/' /etc/ssh/sshd_config
|
|
|
|
# Set CUDA environment variables (for runtime library loading)
|
|
ENV CUDA_HOME=/usr/local/cuda
|
|
ENV PATH="${CUDA_HOME}/bin:${PATH}"
|
|
ENV LD_LIBRARY_PATH="${CUDA_HOME}/lib64:${LD_LIBRARY_PATH}"
|
|
|
|
# NVIDIA runtime configuration
|
|
ENV NVIDIA_VISIBLE_DEVICES=all
|
|
ENV NVIDIA_DRIVER_CAPABILITIES=compute,utility
|
|
ENV CUDA_VISIBLE_DEVICES=0
|
|
|
|
# Rust runtime environment
|
|
ENV RUST_BACKTRACE=1
|
|
ENV RUST_LOG=info
|
|
|
|
# Copy entrypoint scripts (self-terminate wrapper + generic base)
|
|
COPY entrypoint-generic.sh /entrypoint-generic.sh
|
|
COPY entrypoint-self-terminate.sh /entrypoint.sh
|
|
RUN chmod +x /entrypoint-generic.sh /entrypoint.sh
|
|
|
|
# Create workspace directory
|
|
WORKDIR /workspace
|
|
|
|
# Health check to verify GPU is accessible
|
|
HEALTHCHECK --interval=60s --timeout=10s --start-period=30s --retries=3 \
|
|
CMD nvidia-smi || exit 1
|
|
|
|
# Expose SSH port for RunPod remote access
|
|
EXPOSE 22
|
|
|
|
# Entrypoint executes training binary from volume mount
|
|
# NO DOWNLOADS - binaries and data are already on the mounted Runpod Network Volume
|
|
# Override binary via environment variable: BINARY_NAME (default: train_tft_parquet)
|
|
ENTRYPOINT ["/entrypoint.sh"]
|
|
|
|
# Default CMD shows help (deployment script overrides this with dockerStartCmd)
|
|
CMD ["--help"]
|
|
|
|
# =============================================================================
|
|
# BUILD AND RUN INSTRUCTIONS - VOLUME MOUNT ARCHITECTURE
|
|
# =============================================================================
|
|
#
|
|
# 1. BUILD IMAGE (minimal runtime, NO binaries or data embedded):
|
|
# docker build -f Dockerfile.runpod -t jgrusewski/foxhunt:latest .
|
|
# # Image size: ~2GB (CUDA runtime only)
|
|
# # Build time: ~2 minutes
|
|
#
|
|
# 2. PUSH TO DOCKER HUB:
|
|
# docker login # Login to Docker Hub as jgrusewski
|
|
# docker push jgrusewski/foxhunt:latest
|
|
# # IMPORTANT: Set repository to PRIVATE in Docker Hub settings
|
|
#
|
|
# 3. RUNPOD DEPLOYMENT (binaries and data already uploaded to volume):
|
|
# - GPU: Tesla V100-PCIE-16GB (16GB VRAM, $0.29/hr)
|
|
# - Docker Image: jgrusewski/foxhunt:latest (PRIVATE, requires Docker Hub auth)
|
|
# - Volume Mount: Select Runpod Network Volume, mount at /runpod-volume
|
|
# - Environment Variables:
|
|
# * BINARY_NAME=train_tft_parquet (default, or train_mamba2_parquet/train_dqn/train_ppo)
|
|
# * RUST_LOG=info (logging level)
|
|
#
|
|
# 4. TRAIN DIFFERENT MODELS (all binaries on mounted volume):
|
|
# Override BINARY_NAME environment variable:
|
|
# - TFT: BINARY_NAME=train_tft_parquet
|
|
# - MAMBA-2: BINARY_NAME=train_mamba2_parquet
|
|
# - DQN: BINARY_NAME=train_dqn
|
|
# - PPO: BINARY_NAME=train_ppo
|
|
#
|
|
# 5. AVAILABLE DATA ON MOUNTED VOLUME (pre-uploaded, NO download required):
|
|
# /runpod-volume/
|
|
# ├── .env (512B, credentials, chmod 600)
|
|
# ├── binaries/ (77MB total)
|
|
# │ ├── train_tft_parquet (23MB, TFT-225 features)
|
|
# │ ├── train_mamba2_parquet (22MB, MAMBA-2 model)
|
|
# │ ├── train_dqn (22MB, Deep Q-Network)
|
|
# │ └── train_ppo (13MB, Proximal Policy Optimization)
|
|
# ├── test_data/ (14MB total, 9 Parquet files)
|
|
# │ ├── ES_FUT_180d.parquet (2.9MB, 180 days)
|
|
# │ ├── NQ_FUT_180d.parquet (4.4MB, 180 days)
|
|
# │ ├── 6E_FUT_180d.parquet (2.8MB, 180 days)
|
|
# │ ├── ZN_FUT_90d.parquet (2.8MB, 90 days)
|
|
# │ └── [5 more small test files]
|
|
# └── models/ (Empty, populated by training runs)
|
|
#
|
|
# 6. UPDATING BINARIES (NO Docker rebuild required):
|
|
# # Simply upload new binary to Runpod Network Volume via S3 API
|
|
# # Next pod start will use updated binary automatically
|
|
# # Instant deployment (seconds, not minutes)
|
|
#
|
|
# =============================================================================
|
|
# OPTIMIZATION NOTES - VOLUME MOUNT ARCHITECTURE
|
|
# =============================================================================
|
|
#
|
|
# Memory Optimization:
|
|
# - Runtime-only image reduces size from ~10GB to ~2GB
|
|
# - NO binaries or data embedded (stored on mounted volume)
|
|
# - Only runtime CUDA libraries included (no build tools, no Rust)
|
|
# - Minimal image = faster pod startup (~30s vs 2-3min)
|
|
#
|
|
# Build Speed:
|
|
# - No compilation = 2 minute build (vs 20+ minutes)
|
|
# - Image build is one-time only (binaries update via volume upload)
|
|
# - No Docker rebuild needed for code changes (instant deployment)
|
|
#
|
|
# Deployment Speed:
|
|
# - Docker image: Build once, use forever (~2GB, CUDA runtime only)
|
|
# - Binary updates: Upload to volume (seconds, NO rebuild)
|
|
# - Pod startup: ~30 seconds (volume already mounted)
|
|
# - Total deployment time: <60 seconds for new binary
|
|
#
|
|
# GPU Compatibility:
|
|
# - CUDA 13.0 compatible with Tesla V100, RTX 4090, A100, H100
|
|
# - cuDNN 9 for optimal neural network performance
|
|
# - Target: Tesla V100-PCIE-16GB (16GB VRAM, $0.29/hr)
|
|
#
|
|
# Security:
|
|
# - PRIVATE Docker Hub repository required
|
|
# - PRIVATE Runpod Network Volume (binaries + data isolated per account)
|
|
# - No sensitive data in image (only CUDA runtime)
|
|
# - Minimal attack surface (runtime-only image, no build tools)
|
|
#
|
|
# =============================================================================
|