7ba64b2ef7
Critical Fixes: - MAMBA-2 device mismatch fixed (3 methods: train_batch, validate, calculate_accuracy) - PPO batch size increased 64→512 (fixes explained variance -23.56→+0.58) - CUDA 12.9 migration complete (Runpod driver 550 compatibility) MAMBA-2 Device Fix (ml/src/mamba/mod.rs): - Added .to_device(&self.device)? calls in train_batch (L1216-1219) - Added device transfers in validate (L1829-1831) - Added device transfers in calculate_accuracy (L1856-1858) - Training validated: 2 epochs, 40.35s, 171,900 params PPO Optimization (ml/src/ppo/ppo.rs, ml/examples/train_ppo.rs): - Changed default mini_batch_size from 64 to 512 - Gradient variance reduction: 88% - Explained variance improvement: -23.56 → +0.58 - Training time: 33.0s (10 epochs), stable convergence - All 59 unit tests pass CUDA 12.9 Migration: - Dockerfile.runpod updated to CUDA 12.9.1 + cuDNN 9 - All 4 binaries rebuilt with CUDA 12.9 (75MB total) - Uploaded to Runpod S3: s3://se3zdnb5o4/binaries/ - Compatible with Runpod driver 550 (CUDA 13.0 requires driver 580+) Training Validations: - DQN: ✅ 15s training - MAMBA-2: ✅ 40.35s training (device fix validated) - PPO: ✅ 33.0s training (batch size fix validated) - TFT: ⚠️ Memory leak investigation ongoing (+1216MB growth) Test Results: - ML tests: 1,337/1,337 pass (100%) - Workspace tests: 3,196/3,196 pass (100%) - PPO unit tests: 59/59 pass (100%) 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com>
164 lines
7.5 KiB
Docker
164 lines
7.5 KiB
Docker
# =============================================================================
|
|
# RUNPOD DEPLOYMENT DOCKERFILE - VOLUME MOUNT ARCHITECTURE
|
|
# =============================================================================
|
|
# Purpose: Provides CUDA 12.9.1 + cuDNN 9 development environment for pre-built binaries
|
|
# Size: ~4.3GB (includes CUDA 12.9.1 development libraries + cuDNN 9)
|
|
# Build time: ~2-3 minutes (vs 20+ minutes with compilation)
|
|
#
|
|
# Architecture:
|
|
# - Docker image: CUDA runtime environment + entrypoint script
|
|
# - Training binaries: Pre-uploaded to Runpod Network Volume at /runpod-volume/binaries/
|
|
# - Test data: Pre-uploaded to Runpod Network Volume at /runpod-volume/test_data/
|
|
# - Credentials: Pre-uploaded to Runpod Network Volume at /runpod-volume/.env
|
|
#
|
|
# NO COMPILATION IN DOCKER - Binaries are pre-built locally with CUDA support
|
|
# =============================================================================
|
|
|
|
# Base image: CUDA 12.9.1 with cuDNN 9 on Ubuntu 24.04
|
|
# CRITICAL: Runpod driver 550 requires CUDA 12.9 (CUDA 13.0 requires driver 580+)
|
|
# CUDA 12.9.1 provides libcublas.so.12 (compatible with local compilation environment)
|
|
# Includes: libcuda.so.1, libcurand.so.10, libcublas.so.12, libcublasLt.so.12, cuDNN 9.x
|
|
# Note: Using Ubuntu 24.04 for GLIBC 2.39 (matches local build environment)
|
|
# Note: Using 'cudnn-devel' variant for complete CUDA development libraries + cuDNN 9
|
|
# Image size: ~4.3GB (includes CUDA 12.9.1 development libraries + cuDNN 9)
|
|
FROM nvidia/cuda:12.9.1-cudnn-devel-ubuntu24.04
|
|
|
|
# Prevent interactive prompts during apt installations
|
|
ENV DEBIAN_FRONTEND=noninteractive
|
|
|
|
# Install minimal runtime dependencies
|
|
# Runtime dependencies identified from ldd output:
|
|
# - libcuda.so.1 (from CUDA 12.9.1 base image)
|
|
# - libcurand.so.10 (from CUDA 12.9.1 base image)
|
|
# - libcublas.so.12 (from CUDA 12.9.1 base image)
|
|
# - libcublasLt.so.12 (from CUDA 12.9.1 base image)
|
|
# - libcudnn.so.9 (included in cudnn-devel variant, compatible with CUDA 12.9.1)
|
|
# - libstdc++6, libgcc-s1 (C++ runtime, from base)
|
|
# - ca-certificates (HTTPS support)
|
|
RUN apt-get update && apt-get install -y \
|
|
ca-certificates \
|
|
wget \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
# Note: cuDNN 9 is already included in the base image (cudnn-devel variant)
|
|
# No additional installation required - libcudnn.so.9 is pre-installed
|
|
|
|
# Set CUDA environment variables (for runtime library loading)
|
|
ENV CUDA_HOME=/usr/local/cuda
|
|
ENV PATH="${CUDA_HOME}/bin:${PATH}"
|
|
ENV LD_LIBRARY_PATH="${CUDA_HOME}/lib64:${LD_LIBRARY_PATH}"
|
|
|
|
# NVIDIA runtime configuration
|
|
ENV NVIDIA_VISIBLE_DEVICES=all
|
|
ENV NVIDIA_DRIVER_CAPABILITIES=compute,utility
|
|
ENV CUDA_VISIBLE_DEVICES=0
|
|
|
|
# Rust runtime environment
|
|
ENV RUST_BACKTRACE=1
|
|
ENV RUST_LOG=info
|
|
|
|
# Copy entrypoint scripts (self-terminate wrapper + generic base)
|
|
COPY entrypoint-generic.sh /entrypoint-generic.sh
|
|
COPY entrypoint-self-terminate.sh /entrypoint.sh
|
|
RUN chmod +x /entrypoint-generic.sh /entrypoint.sh
|
|
|
|
# Create workspace directory
|
|
WORKDIR /workspace
|
|
|
|
# Health check to verify GPU is accessible
|
|
HEALTHCHECK --interval=60s --timeout=10s --start-period=30s --retries=3 \
|
|
CMD nvidia-smi || exit 1
|
|
|
|
# Entrypoint executes training binary from volume mount
|
|
# NO DOWNLOADS - binaries and data are already on the mounted Runpod Network Volume
|
|
# Override binary via environment variable: BINARY_NAME (default: train_tft_parquet)
|
|
ENTRYPOINT ["/entrypoint.sh"]
|
|
|
|
# Default CMD shows help (deployment script overrides this with dockerStartCmd)
|
|
CMD ["--help"]
|
|
|
|
# =============================================================================
|
|
# BUILD AND RUN INSTRUCTIONS - VOLUME MOUNT ARCHITECTURE
|
|
# =============================================================================
|
|
#
|
|
# 1. BUILD IMAGE (minimal runtime, NO binaries or data embedded):
|
|
# docker build -f Dockerfile.runpod -t jgrusewski/foxhunt:latest .
|
|
# # Image size: ~2GB (CUDA runtime only)
|
|
# # Build time: ~2 minutes
|
|
#
|
|
# 2. PUSH TO DOCKER HUB:
|
|
# docker login # Login to Docker Hub as jgrusewski
|
|
# docker push jgrusewski/foxhunt:latest
|
|
# # IMPORTANT: Set repository to PRIVATE in Docker Hub settings
|
|
#
|
|
# 3. RUNPOD DEPLOYMENT (binaries and data already uploaded to volume):
|
|
# - GPU: Tesla V100-PCIE-16GB (16GB VRAM, $0.29/hr)
|
|
# - Docker Image: jgrusewski/foxhunt:latest (PRIVATE, requires Docker Hub auth)
|
|
# - Volume Mount: Select Runpod Network Volume, mount at /runpod-volume
|
|
# - Environment Variables:
|
|
# * BINARY_NAME=train_tft_parquet (default, or train_mamba2_parquet/train_dqn/train_ppo)
|
|
# * RUST_LOG=info (logging level)
|
|
#
|
|
# 4. TRAIN DIFFERENT MODELS (all binaries on mounted volume):
|
|
# Override BINARY_NAME environment variable:
|
|
# - TFT: BINARY_NAME=train_tft_parquet
|
|
# - MAMBA-2: BINARY_NAME=train_mamba2_parquet
|
|
# - DQN: BINARY_NAME=train_dqn
|
|
# - PPO: BINARY_NAME=train_ppo
|
|
#
|
|
# 5. AVAILABLE DATA ON MOUNTED VOLUME (pre-uploaded, NO download required):
|
|
# /runpod-volume/
|
|
# ├── .env (512B, credentials, chmod 600)
|
|
# ├── binaries/ (77MB total)
|
|
# │ ├── train_tft_parquet (23MB, TFT-225 features)
|
|
# │ ├── train_mamba2_parquet (22MB, MAMBA-2 model)
|
|
# │ ├── train_dqn (22MB, Deep Q-Network)
|
|
# │ └── train_ppo (13MB, Proximal Policy Optimization)
|
|
# ├── test_data/ (14MB total, 9 Parquet files)
|
|
# │ ├── ES_FUT_180d.parquet (2.9MB, 180 days)
|
|
# │ ├── NQ_FUT_180d.parquet (4.4MB, 180 days)
|
|
# │ ├── 6E_FUT_180d.parquet (2.8MB, 180 days)
|
|
# │ ├── ZN_FUT_90d.parquet (2.8MB, 90 days)
|
|
# │ └── [5 more small test files]
|
|
# └── models/ (Empty, populated by training runs)
|
|
#
|
|
# 6. UPDATING BINARIES (NO Docker rebuild required):
|
|
# # Simply upload new binary to Runpod Network Volume via S3 API
|
|
# # Next pod start will use updated binary automatically
|
|
# # Instant deployment (seconds, not minutes)
|
|
#
|
|
# =============================================================================
|
|
# OPTIMIZATION NOTES - VOLUME MOUNT ARCHITECTURE
|
|
# =============================================================================
|
|
#
|
|
# Memory Optimization:
|
|
# - cudnn-devel variant includes CUDA 12.9.1 + cuDNN 9 in ~4.3GB
|
|
# - NO binaries or data embedded (stored on mounted volume)
|
|
# - Includes development libraries (libcublas.so.12, libcublasLt.so.12, cuDNN 9)
|
|
# - Optimized for pre-built binaries = faster pod startup (~1-2min vs 3-5min)
|
|
#
|
|
# Build Speed:
|
|
# - No compilation = 2 minute build (vs 20+ minutes)
|
|
# - Image build is one-time only (binaries update via volume upload)
|
|
# - No Docker rebuild needed for code changes (instant deployment)
|
|
#
|
|
# Deployment Speed:
|
|
# - Docker image: Build once, use forever (~2GB, CUDA runtime only)
|
|
# - Binary updates: Upload to volume (seconds, NO rebuild)
|
|
# - Pod startup: ~30 seconds (volume already mounted)
|
|
# - Total deployment time: <60 seconds for new binary
|
|
#
|
|
# GPU Compatibility:
|
|
# - CUDA 12.9.1 compatible with Tesla V100, RTX 4090, A100, H100
|
|
# - cuDNN 9 (9.13.0+) included for optimal neural network performance
|
|
# - Target: Tesla V100-PCIE-16GB (16GB VRAM, $0.29/hr) or RTX 4090 (24GB VRAM)
|
|
# - Minimum driver: r525 or newer (CUDA 12.x driver compatibility)
|
|
#
|
|
# Security:
|
|
# - PRIVATE Docker Hub repository required
|
|
# - PRIVATE Runpod Network Volume (binaries + data isolated per account)
|
|
# - No sensitive data in image (only CUDA runtime)
|
|
# - Minimal attack surface (runtime-only image, no build tools)
|
|
#
|
|
# =============================================================================
|