gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
9ed4fdd48971a7dd50fbb14acfe7efd2023351e7
foxhunt/scripts/deployment_log.txt
jgrusewski e61e8f54da feat(ml): Complete hyperopt infrastructure + documentation 
Changes:
- CLAUDE.md: Update OOM fix validation status
- Add comprehensive documentation (30+ markdown reports)
- LSTM encoder varmap bug fix (tft/lstm_encoder.rs:290)
- Quantized LSTM layer matching fix (tft/quantized_lstm.rs)
- Hyperopt paths module (ml/src/hyperopt/paths.rs)
- Training path tests for all adapters (DQN, MAMBA-2, PPO, TFT)
- Checkpoint integrity tests
- Script cleanup: Remove 29 obsolete deployment scripts
- Archive old scripts to scripts/archive/
- New deployment utilities: check_gpu_availability.py, monitor_hyperopt.sh

Validation:
- OOM fixes validated: 5/5 trials successful (pod b6kc3mc5lbjiro)
- Batch-size-max 256 tested successfully
- All hyperopt adapters working correctly

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-29 19:52:21 +01:00

56 lines
4.0 KiB
Plaintext
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
2025-10-29 10:57:04 - INFO - Restarting script with venv Python...
2025-10-29 10:57:06 - INFO - ✓ Loaded environment from /home/jgrusewski/Work/foxhunt/.env.runpod
2025-10-29 10:57:06 - INFO - ✓ Environment variables validated
2025-10-29 10:57:06 - INFO -
ℹ Skipping binary upload check (--skip-upload)
2025-10-29 10:57:06 - INFO -
2025-10-29 10:57:06 - INFO - ======================================================================
2025-10-29 10:57:06 - INFO - STEP 2: GPU AVAILABILITY SCAN
2025-10-29 10:57:06 - INFO - ======================================================================
2025-10-29 10:57:06 - INFO - 🔍 Querying available GPU types...
2025-10-29 10:57:06 - INFO - ⏭ MI300X: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ A100 PCIe: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ A100 SXM: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ A40: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ B200: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ RTX 3070: Skipped (VRAM 8GB < 16GB)
2025-10-29 10:57:06 - INFO - ⏭ RTX 3080: Skipped (VRAM 10GB < 16GB)
2025-10-29 10:57:06 - INFO - ⏭ RTX 3080 Ti: Skipped (VRAM 12GB < 16GB)
2025-10-29 10:57:06 - INFO - ⏭ RTX 3090: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ RTX 3090 Ti: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ RTX 4070 Ti: Skipped (VRAM 12GB < 16GB)
2025-10-29 10:57:06 - INFO - ⏭ RTX 4080: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ RTX 4080 SUPER: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ RTX 4090: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ RTX 5080: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ RTX 5090: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ H100 SXM: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ H100 NVL: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ H100 PCIe: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ H200 SXM: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ NVIDIA H200 NVL: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ L4: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ L40: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ L40S: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ RTX 2000 Ada: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ RTX 4000 Ada: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ RTX 4000 Ada SFF: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ RTX 5000 Ada: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ RTX 6000 Ada: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ RTX A2000: Skipped (VRAM 6GB < 16GB)
2025-10-29 10:57:06 - INFO - ⏭ RTX A4000: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ RTX A4500: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ RTX A5000: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ RTX A6000: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ RTX PRO 6000 MaxQ: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ RTX PRO 6000: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ RTX PRO 6000 WK: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ V100 FHHL: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ Tesla V100: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ V100 SXM2: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ V100 SXM2 32GB: Skipped (0 instances available)
2025-10-29 10:57:06 - INFO - ⏭ unknown: Skipped (VRAM 0GB < 16GB)
2025-10-29 10:57:06 - WARNING - ⚠ No GPU types found with ≥16GB VRAM
2025-10-29 10:57:06 - ERROR -
✗ No GPUs available with ≥16GB VRAM in SECURE or COMMUNITY cloud
Reference in New Issue View Git Blame Copy Permalink