Files

jgrusewski 59cce96d9d feat(ml): Fix OOM memory leaks in PPO and TFT hyperopt adapters

Apply explicit resource cleanup pattern to prevent memory accumulation between hyperopt trials. Fixes OOM crashes that occurred after 1-2 trials on RunPod GPU pods.

Changes:
- PPO adapter (ppo.rs:455-469): Add drop() for ppo_agent and val_trajectory_batch
- TFT adapter (tft.rs:444-457): Add drop() for trainer
- Both: CUDA synchronization with 100ms sleep to ensure GPU memory release
- Validation: 5/5 trials completed successfully (vs 0-1 before fix)

Pattern applied:
1. Explicit drop() of model/trainer objects
2. CUDA sync check + 100ms sleep
3. Resource cleanup logging

Validation results (Pod b6kc3mc5lbjiro):
- 5 trials completed without OOM (batch sizes 9-229)
- Total runtime: 79 minutes
- Best loss: 0.047 (Trial 3)
- Memory cleanup working correctly between trials

Note: MAMBA-2 and DQN adapters already had this fix applied.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

2025-10-29 19:35:10 +01:00

4.9 KiB

Raw Blame History

RunPod GPU Detection Fix - 2025-10-29

Problem

The runpod_deploy.py script was incorrectly reporting:

⏭ RTX 4090: Skipped (0 secure cloud instances)
⏭ RTX 4000 Ada: Skipped (0 secure cloud instances)
⏭ RTX 5090: Skipped (0 secure cloud instances)

Even when these GPUs were visible as available in the RunPod web UI.

Root Cause

The script was only checking secureCloud availability and ignoring communityCloud availability:

# OLD CODE (BROKEN)
secure_count = gpu.get('secureCloud', 0)
if secure_count == 0:
    logger.info(f"   ⏭ {gpu_name}: Skipped (0 secure cloud instances)")
    continue

This meant GPUs available in community cloud were being filtered out.

Fix Applied

Updated the script to check both secure AND community cloud availability:

# NEW CODE (FIXED)
secure_count = gpu.get('secureCloud', 0)
community_count = gpu.get('communityCloud', 0)
total_count = secure_count + community_count

if total_count == 0:
    logger.info(f"   ⏭ {gpu_name}: Skipped (0 instances available)")
    continue

# Show detailed availability info
cloud_type = []
if secure_count > 0:
    cloud_type.append(f"secure:{secure_count}")
if community_count > 0:
    cloud_type.append(f"community:{community_count}")
cloud_info = ", ".join(cloud_type)

logger.info(f"   ✓ {gpu_name}: Available ({memory_gb}GB VRAM, ${price:.3f}/hr, {cloud_info})")

Deployment Changes

Updated deploy_pod() function to try both cloud types:

# Try SECURE cloud first, then COMMUNITY cloud
for cloud_type in ["SECURE", "COMMUNITY"]:
    logger.info(f"\n🚀 Trying {cloud_type} cloud...")

    deploy_config = base_config.copy()
    deploy_config["cloud_type"] = cloud_type

    try:
        pod = runpod.create_pod(**deploy_config)
        if pod and 'id' in pod:
            logger.info(f"✓ Pod created successfully on {cloud_type} cloud!")
            return pod
    except Exception as e:
        logger.warning(f"⚠ {cloud_type} cloud deployment failed: {e}")

This ensures the script tries secure cloud first (more reliable), then falls back to community cloud if needed.

Changes Summary

Files modified:

/home/jgrusewski/Work/foxhunt/scripts/runpod_deploy.py

Key changes:

✅ Added communityCloud field check in GPU availability scan
✅ Updated availability filter to use total_count (secure + community)
✅ Enhanced logging to show breakdown of secure vs community instances
✅ Updated deploy_pod() to try both SECURE and COMMUNITY cloud types
✅ Updated script description and error messages to reflect both cloud types

Current Status (2025-10-29 10:57)

Fix Status: ✅ WORKING CORRECTLY

Current GPU Availability: ❌ ALL GPUS UNAVAILABLE

The script now correctly detects both secure and community cloud instances. However, at the time of this fix (10:57 AM), RunPod API reports 0 available instances for all GPUs with ≥16GB VRAM:

RTX 4090:      secure: 0, community: 0, total: 0
RTX 4000 Ada:  secure: 0, community: 0, total: 0
RTX 5090:      secure: 0, community: 0, total: 0
RTX A4000:     secure: 0, community: 0, total: 0
A100 PCIe:     secure: 0, community: 0, total: 0
H100 SXM:      secure: 0, community: 0, total: 0

Recommendation: Retry deployment in a few minutes. RunPod GPU availability changes frequently throughout the day.

Testing

To test the fix once GPUs become available:

# Quick GPU scan (dry-run, no actual deployment)
python3 scripts/runpod_deploy.py --dry-run

# Deploy with specific GPU
python3 scripts/runpod_deploy.py --gpu-type "RTX 4090" --command "YOUR_COMMAND"

# Deploy with auto GPU selection (cheapest available)
python3 scripts/runpod_deploy.py --command "YOUR_COMMAND"

The script will now correctly detect GPUs available in both secure and community clouds.

Example Output (When GPUs Available)

STEP 2: GPU AVAILABILITY SCAN
======================================================================
🔍 Querying available GPU types...
   ✓ RTX 4090: Available (24GB VRAM, $0.340/hr, secure:2, community:5)
   ✓ RTX 4000 Ada: Available (20GB VRAM, $0.280/hr, community:3)
   ✓ RTX 5090: Available (32GB VRAM, $0.450/hr, secure:1)

✓ Found 3 GPU type(s) to try
======================================================================

STEP 3: POD DEPLOYMENT
======================================================================
🎯 Attempting: RTX 4000 Ada ($0.280/hr)

🚀 Trying SECURE cloud...
⚠ SECURE cloud deployment failed: No capacity
🚀 Trying COMMUNITY cloud...
✓ Pod created successfully on COMMUNITY cloud! ID: abc123xyz

Notes

The script prioritizes SECURE cloud for reliability, but will use COMMUNITY cloud if secure is unavailable
The fix ensures we don't miss available GPUs due to cloud type filtering
GPU availability on RunPod is highly dynamic - what's unavailable now might be available in minutes

4.9 KiB Raw Blame History