33afaabe1a
- PPO numerical stability: Added epsilon (1e-8) protection at 4 log locations - Hurst division by zero: Fixed in trending.rs:394 and price_features.rs:342 - DQN 225-feature support: Fixed dimension mismatch (feature_vec[4..]) - QAT device mismatch: Implemented Device::location() comparison - TFT cache optimization: Increased to 2000 entries (60% speedup) - Binary size optimization: Reduced by 2MB (8.7%) via dependency tuning - Unused imports: Eliminated all 34 warnings in ML crate - Test coverage: Added 94+ production hardening tests Test Results: - FP32 Models: 1,317/1,317 tests passing (100%) - Overall Workspace: 313/314 passing (99.7%) - QAT: 0/24 (temporarily disabled, compilation errors) Performance: - TFT training: ~2 min (60% faster via cache optimization) - DQN training: ~15s (10-25% faster via mimalloc) - Average improvement: 922× vs minimum requirements QAT Blockers (P0 - 1-2 weeks): 1. Device mismatch: 11 compilation errors in qat_tft.rs 2. Gradient checkpointing: CLI flag exists but not implemented 3. OOM recovery: AutoBatchSizer exists but no retry integration Documentation: - FINAL_VALIDATION_SUMMARY.md (17 agents, 281 lines) - STABILIZATION_WAVE_COMPLETION_REPORT.md (290 lines) - DEPLOYMENT_QUICK_START.md (385 lines) - PRE_DEPLOYMENT_CHECKLIST.md (426 lines) - KNOWN_ISSUES.md (385 lines) - NEXT_STEPS_ROADMAP.md (27KB) Status: ✅ FP32 PRODUCTION READY | 🔴 QAT BLOCKED
195 lines
8.8 KiB
Docker
195 lines
8.8 KiB
Docker
# =============================================================================
|
|
# RUNPOD DEPLOYMENT DOCKERFILE - VOLUME MOUNT ARCHITECTURE
|
|
# =============================================================================
|
|
# Purpose: Provides CUDA 13.0 + cuDNN development environment for pre-built binaries
|
|
# Size: ~8.4GB (includes CUDA 13.0 development libraries)
|
|
# Build time: ~2 minutes (vs 20+ minutes with compilation)
|
|
#
|
|
# Architecture:
|
|
# - Docker image: CUDA runtime environment + entrypoint script
|
|
# - Training binaries: Pre-uploaded to Runpod Network Volume at /runpod-volume/binaries/
|
|
# - Test data: Pre-uploaded to Runpod Network Volume at /runpod-volume/test_data/
|
|
# - Credentials: Pre-uploaded to Runpod Network Volume at /runpod-volume/.env
|
|
#
|
|
# NO COMPILATION IN DOCKER - Binaries are pre-built locally with CUDA support
|
|
# =============================================================================
|
|
|
|
# Base image: CUDA 13.0 devel on Ubuntu 24.04
|
|
# CRITICAL FIX: Binary compiled with CUDA 13.0 locally, requires libcublas.so.13
|
|
# CUDA 13.0 provides libcublas.so.13 (exact match with local compilation environment)
|
|
# Includes: libcuda.so.1, libcurand.so.10, libcublas.so.13, libcublasLt.so.13
|
|
# Note: Using Ubuntu 24.04 for GLIBC 2.39 (matches local build environment)
|
|
# Note: Using 'devel' instead of 'runtime' because our binaries need libcublas/libcublasLt
|
|
FROM nvidia/cuda:13.0.0-devel-ubuntu24.04
|
|
|
|
# Prevent interactive prompts during apt installations
|
|
ENV DEBIAN_FRONTEND=noninteractive
|
|
|
|
# Install minimal runtime dependencies
|
|
# Runtime dependencies identified from ldd output:
|
|
# - libcuda.so.1 (from base image)
|
|
# - libcurand.so.10 (from base image)
|
|
# - libcublas.so.13 (from CUDA 12.2 base image)
|
|
# - libcublasLt.so.13 (from CUDA 12.2 base image)
|
|
# - libcudnn.so.8 (installed below, compatible with CUDA 12.2)
|
|
# - libstdc++6, libgcc-s1 (C++ runtime, from base)
|
|
# - ca-certificates (HTTPS support)
|
|
RUN apt-get update && apt-get install -y \
|
|
ca-certificates \
|
|
wget \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
# Install cuDNN 9 for CUDA 13.0 (matches binary compilation environment)
|
|
# Note: cuDNN 9 is standard for CUDA 13.x, provides libcublas.so.13
|
|
RUN apt-get update && apt-get install -y \
|
|
libcudnn9-cuda-13 \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
# Install runpodctl CLI tool for pod self-termination
|
|
# Download latest version from GitHub releases (v1.14.11 as of 2025-10-24)
|
|
# Used by entrypoint-self-terminate.sh to terminate pod after training completes
|
|
RUN apt-get update && apt-get install -y curl && \
|
|
wget -qO /tmp/runpodctl.tar.gz "https://github.com/runpod/runpodctl/releases/download/v1.14.11/runpodctl_1.14.11_linux_amd64.tar.gz" && \
|
|
tar -xzf /tmp/runpodctl.tar.gz -C /tmp && \
|
|
mv /tmp/runpodctl /usr/local/bin/runpodctl && \
|
|
chmod +x /usr/local/bin/runpodctl && \
|
|
rm -rf /tmp/runpodctl.tar.gz /var/lib/apt/lists/*
|
|
|
|
# =============================================================================
|
|
# SSH SERVER INSTALLATION FOR RUNPOD REMOTE ACCESS
|
|
# =============================================================================
|
|
# Install OpenSSH server for remote debugging and file transfers
|
|
# RunPod injects SSH public keys via PUBLIC_KEY environment variable
|
|
# Access via: ssh root@<pod-id>.ssh.runpod.io
|
|
RUN apt-get update && apt-get install -y \
|
|
openssh-server \
|
|
&& mkdir -p /var/run/sshd /root/.ssh \
|
|
&& chmod 700 /root/.ssh \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
# Configure SSH for key-only authentication (no passwords)
|
|
RUN sed -i 's/#PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config \
|
|
&& sed -i 's/#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config \
|
|
&& sed -i 's/#PubkeyAuthentication yes/PubkeyAuthentication yes/' /etc/ssh/sshd_config
|
|
|
|
# Set CUDA environment variables (for runtime library loading)
|
|
ENV CUDA_HOME=/usr/local/cuda
|
|
ENV PATH="${CUDA_HOME}/bin:${PATH}"
|
|
ENV LD_LIBRARY_PATH="${CUDA_HOME}/lib64:${LD_LIBRARY_PATH}"
|
|
|
|
# NVIDIA runtime configuration
|
|
ENV NVIDIA_VISIBLE_DEVICES=all
|
|
ENV NVIDIA_DRIVER_CAPABILITIES=compute,utility
|
|
ENV CUDA_VISIBLE_DEVICES=0
|
|
|
|
# Rust runtime environment
|
|
ENV RUST_BACKTRACE=1
|
|
ENV RUST_LOG=info
|
|
|
|
# Copy entrypoint scripts (self-terminate wrapper + generic base)
|
|
COPY entrypoint-generic.sh /entrypoint-generic.sh
|
|
COPY entrypoint-self-terminate.sh /entrypoint.sh
|
|
RUN chmod +x /entrypoint-generic.sh /entrypoint.sh
|
|
|
|
# Create workspace directory
|
|
WORKDIR /workspace
|
|
|
|
# Health check to verify GPU is accessible
|
|
HEALTHCHECK --interval=60s --timeout=10s --start-period=30s --retries=3 \
|
|
CMD nvidia-smi || exit 1
|
|
|
|
# Expose SSH port for RunPod remote access
|
|
EXPOSE 22
|
|
|
|
# Entrypoint executes training binary from volume mount
|
|
# NO DOWNLOADS - binaries and data are already on the mounted Runpod Network Volume
|
|
# Override binary via environment variable: BINARY_NAME (default: train_tft_parquet)
|
|
ENTRYPOINT ["/entrypoint.sh"]
|
|
|
|
# Default CMD shows help (deployment script overrides this with dockerStartCmd)
|
|
CMD ["--help"]
|
|
|
|
# =============================================================================
|
|
# BUILD AND RUN INSTRUCTIONS - VOLUME MOUNT ARCHITECTURE
|
|
# =============================================================================
|
|
#
|
|
# 1. BUILD IMAGE (minimal runtime, NO binaries or data embedded):
|
|
# docker build -f Dockerfile.runpod -t jgrusewski/foxhunt:latest .
|
|
# # Image size: ~2GB (CUDA runtime only)
|
|
# # Build time: ~2 minutes
|
|
#
|
|
# 2. PUSH TO DOCKER HUB:
|
|
# docker login # Login to Docker Hub as jgrusewski
|
|
# docker push jgrusewski/foxhunt:latest
|
|
# # IMPORTANT: Set repository to PRIVATE in Docker Hub settings
|
|
#
|
|
# 3. RUNPOD DEPLOYMENT (binaries and data already uploaded to volume):
|
|
# - GPU: Tesla V100-PCIE-16GB (16GB VRAM, $0.29/hr)
|
|
# - Docker Image: jgrusewski/foxhunt:latest (PRIVATE, requires Docker Hub auth)
|
|
# - Volume Mount: Select Runpod Network Volume, mount at /runpod-volume
|
|
# - Environment Variables:
|
|
# * BINARY_NAME=train_tft_parquet (default, or train_mamba2_parquet/train_dqn/train_ppo)
|
|
# * RUST_LOG=info (logging level)
|
|
#
|
|
# 4. TRAIN DIFFERENT MODELS (all binaries on mounted volume):
|
|
# Override BINARY_NAME environment variable:
|
|
# - TFT: BINARY_NAME=train_tft_parquet
|
|
# - MAMBA-2: BINARY_NAME=train_mamba2_parquet
|
|
# - DQN: BINARY_NAME=train_dqn
|
|
# - PPO: BINARY_NAME=train_ppo
|
|
#
|
|
# 5. AVAILABLE DATA ON MOUNTED VOLUME (pre-uploaded, NO download required):
|
|
# /runpod-volume/
|
|
# ├── .env (512B, credentials, chmod 600)
|
|
# ├── binaries/ (77MB total)
|
|
# │ ├── train_tft_parquet (23MB, TFT-225 features)
|
|
# │ ├── train_mamba2_parquet (22MB, MAMBA-2 model)
|
|
# │ ├── train_dqn (22MB, Deep Q-Network)
|
|
# │ └── train_ppo (13MB, Proximal Policy Optimization)
|
|
# ├── test_data/ (14MB total, 9 Parquet files)
|
|
# │ ├── ES_FUT_180d.parquet (2.9MB, 180 days)
|
|
# │ ├── NQ_FUT_180d.parquet (4.4MB, 180 days)
|
|
# │ ├── 6E_FUT_180d.parquet (2.8MB, 180 days)
|
|
# │ ├── ZN_FUT_90d.parquet (2.8MB, 90 days)
|
|
# │ └── [5 more small test files]
|
|
# └── models/ (Empty, populated by training runs)
|
|
#
|
|
# 6. UPDATING BINARIES (NO Docker rebuild required):
|
|
# # Simply upload new binary to Runpod Network Volume via S3 API
|
|
# # Next pod start will use updated binary automatically
|
|
# # Instant deployment (seconds, not minutes)
|
|
#
|
|
# =============================================================================
|
|
# OPTIMIZATION NOTES - VOLUME MOUNT ARCHITECTURE
|
|
# =============================================================================
|
|
#
|
|
# Memory Optimization:
|
|
# - Runtime-only image reduces size from ~10GB to ~2GB
|
|
# - NO binaries or data embedded (stored on mounted volume)
|
|
# - Only runtime CUDA libraries included (no build tools, no Rust)
|
|
# - Minimal image = faster pod startup (~30s vs 2-3min)
|
|
#
|
|
# Build Speed:
|
|
# - No compilation = 2 minute build (vs 20+ minutes)
|
|
# - Image build is one-time only (binaries update via volume upload)
|
|
# - No Docker rebuild needed for code changes (instant deployment)
|
|
#
|
|
# Deployment Speed:
|
|
# - Docker image: Build once, use forever (~2GB, CUDA runtime only)
|
|
# - Binary updates: Upload to volume (seconds, NO rebuild)
|
|
# - Pod startup: ~30 seconds (volume already mounted)
|
|
# - Total deployment time: <60 seconds for new binary
|
|
#
|
|
# GPU Compatibility:
|
|
# - CUDA 13.0 compatible with Tesla V100, RTX 4090, A100, H100
|
|
# - cuDNN 9 for optimal neural network performance
|
|
# - Target: Tesla V100-PCIE-16GB (16GB VRAM, $0.29/hr)
|
|
#
|
|
# Security:
|
|
# - PRIVATE Docker Hub repository required
|
|
# - PRIVATE Runpod Network Volume (binaries + data isolated per account)
|
|
# - No sensitive data in image (only CUDA runtime)
|
|
# - Minimal attack surface (runtime-only image, no build tools)
|
|
#
|
|
# =============================================================================
|