59cce96d9d
Apply explicit resource cleanup pattern to prevent memory accumulation between hyperopt trials. Fixes OOM crashes that occurred after 1-2 trials on RunPod GPU pods. Changes: - PPO adapter (ppo.rs:455-469): Add drop() for ppo_agent and val_trajectory_batch - TFT adapter (tft.rs:444-457): Add drop() for trainer - Both: CUDA synchronization with 100ms sleep to ensure GPU memory release - Validation: 5/5 trials completed successfully (vs 0-1 before fix) Pattern applied: 1. Explicit drop() of model/trainer objects 2. CUDA sync check + 100ms sleep 3. Resource cleanup logging Validation results (Pod b6kc3mc5lbjiro): - 5 trials completed without OOM (batch sizes 9-229) - Total runtime: 79 minutes - Best loss: 0.047 (Trial 3) - Memory cleanup working correctly between trials Note: MAMBA-2 and DQN adapters already had this fix applied. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
RunPod Deployment Script
Script: runpod_deploy.py
Automated deployment script for RunPod GPU pods in EUR-IS region (SECURE cloud).
Features
- Scans available GPUs with ≥16GB VRAM
- Auto-selects best value GPU (RTX 4090 preferred, then cheapest)
- Supports custom GPU selection
- Dry-run mode for testing
- Automatic network volume attachment
Requirements
pip install requests python-dotenv
Configuration
Create .env.runpod with:
RUNPOD_API_KEY=your_api_key
RUNPOD_VOLUME_ID=your_volume_id
Usage Examples
# Auto-select best value GPU (dry run)
./scripts/runpod_deploy.py --dry-run
# Deploy with default settings (RTX 4090 preferred)
./scripts/runpod_deploy.py
# Deploy with specific GPU
./scripts/runpod_deploy.py --gpu-type "RTX 3090"
# Custom image and larger disk
./scripts/runpod_deploy.py \
--image runpod/pytorch:2.1.0-py3.10-cuda11.8.0-devel \
--container-disk 100
# With custom command
./scripts/runpod_deploy.py --command "jupyter lab --allow-root"
Default Configuration
- Cloud Type: SECURE (no spot interruptions)
- Region: EUR-IS (Iceland - low latency to Europe)
- Image:
runpod/pytorch:2.4.0-py3.11-cuda12.4.1-devel-ubuntu22.04 - Container Disk: 50GB
- Network Volume: Attached from
.env.runpod - Ports: 8888/http (Jupyter)
GPU Selection Logic
- If
--gpu-typespecified and available → use it - Else if RTX 4090 available → use it (best value)
- Else → use cheapest available GPU
Output
✅ POD DEPLOYED SUCCESSFULLY
======================================================================
Pod ID: abc123-xyz789
GPU: RTX 4090 (24GB)
Cost: $0.340/hr
Image: runpod/pytorch:2.4.0
Status: RUNNING
======================================================================
📝 NEXT STEPS:
1. Wait 2-3 minutes for pod to initialize
2. Access Jupyter at: https://abc123-8888.proxy.runpod.net
3. SSH access: ssh root@abc123.ssh.runpod.io
4. Monitor pod: https://www.runpod.io/console/pods
Cost Warning
The script will display hourly costs. Remember to stop pods when done to avoid unnecessary charges.