59cce96d9d
Apply explicit resource cleanup pattern to prevent memory accumulation between hyperopt trials. Fixes OOM crashes that occurred after 1-2 trials on RunPod GPU pods. Changes: - PPO adapter (ppo.rs:455-469): Add drop() for ppo_agent and val_trajectory_batch - TFT adapter (tft.rs:444-457): Add drop() for trainer - Both: CUDA synchronization with 100ms sleep to ensure GPU memory release - Validation: 5/5 trials completed successfully (vs 0-1 before fix) Pattern applied: 1. Explicit drop() of model/trainer objects 2. CUDA sync check + 100ms sleep 3. Resource cleanup logging Validation results (Pod b6kc3mc5lbjiro): - 5 trials completed without OOM (batch sizes 9-229) - Total runtime: 79 minutes - Best loss: 0.047 (Trial 3) - Memory cleanup working correctly between trials Note: MAMBA-2 and DQN adapters already had this fix applied. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
61 lines
1.7 KiB
Bash
Executable File
61 lines
1.7 KiB
Bash
Executable File
#!/bin/bash
|
|
# Watch for GPU availability on RunPod
|
|
# Usage: ./watch_gpu_availability.sh [check_interval_seconds]
|
|
|
|
INTERVAL=${1:-30} # Default: check every 30 seconds
|
|
|
|
echo "==================================================================="
|
|
echo "RunPod GPU Availability Monitor"
|
|
echo "==================================================================="
|
|
echo "Checking every ${INTERVAL} seconds..."
|
|
echo "Press Ctrl+C to stop"
|
|
echo ""
|
|
|
|
while true; do
|
|
TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')
|
|
echo "[$TIMESTAMP] Scanning for GPUs with ≥16GB VRAM..."
|
|
|
|
# Run availability check
|
|
.venv/bin/python3 << 'EOF'
|
|
import os
|
|
from pathlib import Path
|
|
from dotenv import load_dotenv
|
|
import runpod
|
|
|
|
# Load environment
|
|
env_path = Path.cwd().parent / '.env.runpod'
|
|
load_dotenv(env_path)
|
|
runpod.api_key = os.getenv('RUNPOD_API_KEY')
|
|
|
|
# Get GPU types
|
|
gpu_types = runpod.get_gpus()
|
|
found_any = False
|
|
|
|
for gpu in gpu_types:
|
|
memory_gb = gpu.get('memoryInGb', 0)
|
|
secure = gpu.get('secureCloud', 0)
|
|
community = gpu.get('communityCloud', 0)
|
|
total = secure + community
|
|
name = gpu.get('displayName', 'Unknown')
|
|
|
|
if memory_gb >= 16 and total > 0:
|
|
price_info = gpu.get('lowestPrice', {})
|
|
price = price_info.get('uninterruptablePrice', 'N/A') if price_info else 'N/A'
|
|
|
|
cloud_info = []
|
|
if secure > 0:
|
|
cloud_info.append(f"secure:{secure}")
|
|
if community > 0:
|
|
cloud_info.append(f"community:{community}")
|
|
|
|
print(f" ✅ {name}: {total} available ({', '.join(cloud_info)}) @ ${price}/hr")
|
|
found_any = True
|
|
|
|
if not found_any:
|
|
print(" ❌ No GPUs with ≥16GB VRAM available")
|
|
EOF
|
|
|
|
echo ""
|
|
sleep $INTERVAL
|
|
done
|