jgrusewski
|
59cce96d9d
|
feat(ml): Fix OOM memory leaks in PPO and TFT hyperopt adapters
Apply explicit resource cleanup pattern to prevent memory accumulation between hyperopt trials. Fixes OOM crashes that occurred after 1-2 trials on RunPod GPU pods.
Changes:
- PPO adapter (ppo.rs:455-469): Add drop() for ppo_agent and val_trajectory_batch
- TFT adapter (tft.rs:444-457): Add drop() for trainer
- Both: CUDA synchronization with 100ms sleep to ensure GPU memory release
- Validation: 5/5 trials completed successfully (vs 0-1 before fix)
Pattern applied:
1. Explicit drop() of model/trainer objects
2. CUDA sync check + 100ms sleep
3. Resource cleanup logging
Validation results (Pod b6kc3mc5lbjiro):
- 5 trials completed without OOM (batch sizes 9-229)
- Total runtime: 79 minutes
- Best loss: 0.047 (Trial 3)
- Memory cleanup working correctly between trials
Note: MAMBA-2 and DQN adapters already had this fix applied.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
|
2025-10-29 19:35:10 +01:00 |
|