6da9d262db
CRITICAL P0 FIXES (Validated - Loss 0.87 → 0.07): - Add sigmoid activation to inference and training (ml/src/mamba/mod.rs:798, 1538) - Fix config.total_decay_steps (was hardcoded 10000) (ml/src/mamba/mod.rs:2271) - Update d_state: 16→64, 32→64 (Mamba-2 spec) (ml/src/mamba/mod.rs:178, 730) HYPERPARAMETER OPTIMIZATION: - Implement 13-parameter Bayesian optimization with argmin - Add async data loading with 3-batch prefetch (+20-30% speedup) - Create hyperopt adapter: ml/src/hyperopt/adapters/mamba2.rs - Add example: ml/examples/hyperopt_mamba2_demo.rs VALIDATION: - Local test: Loss 0.07 vs 0.87 (12× improvement) - Val loss: 0.04-0.14 vs 1.2 (27× improvement) - Accuracy: 12-30% vs 1-5% (3-6× improvement) - All binaries rebuilt and uploaded to Runpod S3 DEPLOYMENT: - RTX 4090 pod active (n0fq2ikt4uk0zy) - Training: 10 trials × 50 epochs, batch_size=256 - Expected: 1.3 days, $10.41 cost Fixes #P0-sigmoid #P0-decay-steps #hyperopt-mamba2
166 lines
7.7 KiB
Docker
166 lines
7.7 KiB
Docker
# =============================================================================
|
|
# RUNPOD DEPLOYMENT DOCKERFILE - VOLUME MOUNT ARCHITECTURE
|
|
# =============================================================================
|
|
# Purpose: Provides CUDA 12.9.1 + cuDNN 9 development environment for pre-built binaries
|
|
# Size: ~4.3GB (includes CUDA 12.9.1 development libraries + cuDNN 9)
|
|
# Build time: ~2-3 minutes (vs 20+ minutes with compilation)
|
|
#
|
|
# Architecture:
|
|
# - Docker image: CUDA runtime environment + entrypoint script
|
|
# - Training binaries: Pre-uploaded to Runpod Network Volume at /runpod-volume/binaries/
|
|
# - Test data: Pre-uploaded to Runpod Network Volume at /runpod-volume/test_data/
|
|
# - Credentials: Pre-uploaded to Runpod Network Volume at /runpod-volume/.env
|
|
#
|
|
# NO COMPILATION IN DOCKER - Binaries are pre-built locally with CUDA support
|
|
# =============================================================================
|
|
|
|
# Base image: CUDA 12.9.1 with cuDNN 9 on Ubuntu 24.04
|
|
# CRITICAL: Runpod driver 550 requires CUDA 12.9 (CUDA 13.0 requires driver 580+)
|
|
# CUDA 12.9.1 provides libcublas.so.12 (compatible with local compilation environment)
|
|
# Includes: libcuda.so.1, libcurand.so.10, libcublas.so.12, libcublasLt.so.12, cuDNN 9.x
|
|
# Note: Using Ubuntu 24.04 for GLIBC 2.39 (matches local build environment)
|
|
# Note: Using 'cudnn-devel' variant for complete CUDA development libraries + cuDNN 9
|
|
# Image size: ~4.3GB (includes CUDA 12.9.1 development libraries + cuDNN 9)
|
|
FROM nvidia/cuda:12.9.1-cudnn-devel-ubuntu24.04
|
|
|
|
# Prevent interactive prompts during apt installations
|
|
ENV DEBIAN_FRONTEND=noninteractive
|
|
|
|
# Install minimal runtime dependencies
|
|
# Runtime dependencies identified from ldd output:
|
|
# - libcuda.so.1 (from CUDA 12.9.1 base image)
|
|
# - libcurand.so.10 (from CUDA 12.9.1 base image)
|
|
# - libcublas.so.12 (from CUDA 12.9.1 base image)
|
|
# - libcublasLt.so.12 (from CUDA 12.9.1 base image)
|
|
# - libcudnn.so.9 (included in cudnn-devel variant, compatible with CUDA 12.9.1)
|
|
# - libstdc++6, libgcc-s1 (C++ runtime, from base)
|
|
# - ca-certificates (HTTPS support)
|
|
RUN apt-get update && apt-get install -y \
|
|
ca-certificates \
|
|
wget \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
# Note: cuDNN 9 is already included in the base image (cudnn-devel variant)
|
|
# No additional installation required - libcudnn.so.9 is pre-installed
|
|
|
|
# Set CUDA environment variables (for runtime library loading)
|
|
# CRITICAL: Include compat path FIRST to fix CUDA_ERROR_UNSUPPORTED_PTX_VERSION
|
|
# Driver 550 needs compat libraries for CUDA 12.9 PTX (see CUDA_PTX_VERSION_DEEP_INVESTIGATION.md)
|
|
ENV CUDA_HOME=/usr/local/cuda
|
|
ENV PATH="${CUDA_HOME}/bin:${PATH}"
|
|
ENV LD_LIBRARY_PATH="${CUDA_HOME}/compat:${CUDA_HOME}/lib64:${LD_LIBRARY_PATH}"
|
|
|
|
# NVIDIA runtime configuration
|
|
ENV NVIDIA_VISIBLE_DEVICES=all
|
|
ENV NVIDIA_DRIVER_CAPABILITIES=compute,utility
|
|
ENV CUDA_VISIBLE_DEVICES=0
|
|
|
|
# Rust runtime environment
|
|
ENV RUST_BACKTRACE=1
|
|
ENV RUST_LOG=info
|
|
|
|
# Copy entrypoint scripts (self-terminate wrapper + generic base)
|
|
COPY entrypoint-generic.sh /entrypoint-generic.sh
|
|
COPY entrypoint-self-terminate.sh /entrypoint.sh
|
|
RUN chmod +x /entrypoint-generic.sh /entrypoint.sh
|
|
|
|
# Create workspace directory
|
|
WORKDIR /workspace
|
|
|
|
# Health check to verify GPU is accessible
|
|
HEALTHCHECK --interval=60s --timeout=10s --start-period=30s --retries=3 \
|
|
CMD nvidia-smi || exit 1
|
|
|
|
# Entrypoint executes training binary from volume mount
|
|
# NO DOWNLOADS - binaries and data are already on the mounted Runpod Network Volume
|
|
# Override binary via environment variable: BINARY_NAME (default: train_tft_parquet)
|
|
ENTRYPOINT ["/entrypoint.sh"]
|
|
|
|
# Default CMD shows help (deployment script overrides this with dockerStartCmd)
|
|
CMD ["--help"]
|
|
|
|
# =============================================================================
|
|
# BUILD AND RUN INSTRUCTIONS - VOLUME MOUNT ARCHITECTURE
|
|
# =============================================================================
|
|
#
|
|
# 1. BUILD IMAGE (minimal runtime, NO binaries or data embedded):
|
|
# docker build -f Dockerfile.runpod -t jgrusewski/foxhunt:latest .
|
|
# # Image size: ~2GB (CUDA runtime only)
|
|
# # Build time: ~2 minutes
|
|
#
|
|
# 2. PUSH TO DOCKER HUB:
|
|
# docker login # Login to Docker Hub as jgrusewski
|
|
# docker push jgrusewski/foxhunt:latest
|
|
# # IMPORTANT: Set repository to PRIVATE in Docker Hub settings
|
|
#
|
|
# 3. RUNPOD DEPLOYMENT (binaries and data already uploaded to volume):
|
|
# - GPU: Tesla V100-PCIE-16GB (16GB VRAM, $0.29/hr)
|
|
# - Docker Image: jgrusewski/foxhunt:latest (PRIVATE, requires Docker Hub auth)
|
|
# - Volume Mount: Select Runpod Network Volume, mount at /runpod-volume
|
|
# - Environment Variables:
|
|
# * BINARY_NAME=train_tft_parquet (default, or train_mamba2_parquet/train_dqn/train_ppo)
|
|
# * RUST_LOG=info (logging level)
|
|
#
|
|
# 4. TRAIN DIFFERENT MODELS (all binaries on mounted volume):
|
|
# Override BINARY_NAME environment variable:
|
|
# - TFT: BINARY_NAME=train_tft_parquet
|
|
# - MAMBA-2: BINARY_NAME=train_mamba2_parquet
|
|
# - DQN: BINARY_NAME=train_dqn
|
|
# - PPO: BINARY_NAME=train_ppo
|
|
#
|
|
# 5. AVAILABLE DATA ON MOUNTED VOLUME (pre-uploaded, NO download required):
|
|
# /runpod-volume/
|
|
# ├── .env (512B, credentials, chmod 600)
|
|
# ├── binaries/ (77MB total)
|
|
# │ ├── train_tft_parquet (23MB, TFT-225 features)
|
|
# │ ├── train_mamba2_parquet (22MB, MAMBA-2 model)
|
|
# │ ├── train_dqn (22MB, Deep Q-Network)
|
|
# │ └── train_ppo (13MB, Proximal Policy Optimization)
|
|
# ├── test_data/ (14MB total, 9 Parquet files)
|
|
# │ ├── ES_FUT_180d.parquet (2.9MB, 180 days)
|
|
# │ ├── NQ_FUT_180d.parquet (4.4MB, 180 days)
|
|
# │ ├── 6E_FUT_180d.parquet (2.8MB, 180 days)
|
|
# │ ├── ZN_FUT_90d.parquet (2.8MB, 90 days)
|
|
# │ └── [5 more small test files]
|
|
# └── models/ (Empty, populated by training runs)
|
|
#
|
|
# 6. UPDATING BINARIES (NO Docker rebuild required):
|
|
# # Simply upload new binary to Runpod Network Volume via S3 API
|
|
# # Next pod start will use updated binary automatically
|
|
# # Instant deployment (seconds, not minutes)
|
|
#
|
|
# =============================================================================
|
|
# OPTIMIZATION NOTES - VOLUME MOUNT ARCHITECTURE
|
|
# =============================================================================
|
|
#
|
|
# Memory Optimization:
|
|
# - cudnn-devel variant includes CUDA 12.9.1 + cuDNN 9 in ~4.3GB
|
|
# - NO binaries or data embedded (stored on mounted volume)
|
|
# - Includes development libraries (libcublas.so.12, libcublasLt.so.12, cuDNN 9)
|
|
# - Optimized for pre-built binaries = faster pod startup (~1-2min vs 3-5min)
|
|
#
|
|
# Build Speed:
|
|
# - No compilation = 2 minute build (vs 20+ minutes)
|
|
# - Image build is one-time only (binaries update via volume upload)
|
|
# - No Docker rebuild needed for code changes (instant deployment)
|
|
#
|
|
# Deployment Speed:
|
|
# - Docker image: Build once, use forever (~2GB, CUDA runtime only)
|
|
# - Binary updates: Upload to volume (seconds, NO rebuild)
|
|
# - Pod startup: ~30 seconds (volume already mounted)
|
|
# - Total deployment time: <60 seconds for new binary
|
|
#
|
|
# GPU Compatibility:
|
|
# - CUDA 12.9.1 compatible with Tesla V100, RTX 4090, A100, H100
|
|
# - cuDNN 9 (9.13.0+) included for optimal neural network performance
|
|
# - Target: Tesla V100-PCIE-16GB (16GB VRAM, $0.29/hr) or RTX 4090 (24GB VRAM)
|
|
# - Minimum driver: r525 or newer (CUDA 12.x driver compatibility)
|
|
#
|
|
# Security:
|
|
# - PRIVATE Docker Hub repository required
|
|
# - PRIVATE Runpod Network Volume (binaries + data isolated per account)
|
|
# - No sensitive data in image (only CUDA runtime)
|
|
# - Minimal attack surface (runtime-only image, no build tools)
|
|
#
|
|
# =============================================================================
|