3853988af7
- Fixed PSO budget calculation bug in ml/src/hyperopt/optimizer.rs - Root cause: Division by n_particles in sequential execution - Now correctly calculates max_iters = remaining_trials (no division) - Result: 50 trials complete instead of 23 (100% vs 46%) - Added comprehensive DQN hyperopt results analysis - 39/50 trials analyzed across 2 RunPod deployments - Best hyperparameters identified: LR 4.89e-5 (ultra-low) - Created DQN_HYPEROPT_RESULTS_SUMMARY.md with expert validation - GitLab CI/CD pipeline operational (48 lines fixed) - Fixed YAML syntax errors (unquoted colons) - All 7 jobs validated and working - Warning cleanup complete (136 → 0 warnings) - Removed 143 lines dead code - Fixed visibility, unused imports, Debug traits - Archived Wave D reports to docs/archive/ - 8 early stopping reports moved - Root directory cleaned up 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
136 lines
5.1 KiB
Plaintext
136 lines
5.1 KiB
Plaintext
===============================================================================
|
||
POD REDEPLOYMENT SUMMARY - 2025-11-01
|
||
===============================================================================
|
||
|
||
MISSION: Terminate invalid pods and redeploy with corrected CLI arguments
|
||
|
||
===============================================================================
|
||
OLD PODS (TERMINATED)
|
||
===============================================================================
|
||
|
||
1. MAMBA2 Pod: rolerffcwio5ti
|
||
Status: Already terminated (404 - not found)
|
||
|
||
2. DQN Pod: n1emkvj04k6ezj
|
||
Status: ✅ Terminated successfully
|
||
|
||
===============================================================================
|
||
NEW PODS (DEPLOYED)
|
||
===============================================================================
|
||
|
||
1. MAMBA2 HYPEROPT POD
|
||
Pod ID: qarw3nchfoz5mk
|
||
Status: RUNNING ✅
|
||
GPU: RTX A4000 (16GB VRAM)
|
||
Datacenter: EUR-IS-1
|
||
Cost: $0.25/hr
|
||
Docker Image: jgrusewski/foxhunt-hyperopt:latest
|
||
|
||
Command (CORRECTED):
|
||
-------------------
|
||
hyperopt_mamba2_demo \
|
||
--parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet \
|
||
--trials 50 \
|
||
--epochs 100 \
|
||
--batch-size-max 256 \
|
||
--base-dir /runpod-volume/ml_training/mamba2_hyperopt_batch256 \
|
||
--early-stopping-min-epochs 50
|
||
|
||
FIXES APPLIED:
|
||
- ❌ --timeout → REMOVED (not supported)
|
||
- ❌ --max-batch-size → ✅ --batch-size-max 256
|
||
- ❌ --output-dir → ✅ --base-dir /runpod-volume/ml_training/mamba2_hyperopt_batch256
|
||
- ❌ --checkpoint-dir → REMOVED (handled by base-dir)
|
||
- ✅ Added: --parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet
|
||
- ✅ Added: --epochs 100
|
||
- ✅ Added: --early-stopping-min-epochs 50
|
||
|
||
Expected Duration: 46-65 hours
|
||
Expected Cost: $11.50-$16.25
|
||
|
||
2. DQN HYPEROPT POD
|
||
Pod ID: iyh6whl578olaq
|
||
Status: RUNNING ✅
|
||
GPU: RTX A4000 (16GB VRAM)
|
||
Datacenter: EUR-IS-1
|
||
Cost: $0.25/hr
|
||
Docker Image: jgrusewski/foxhunt-hyperopt:latest
|
||
|
||
Command (CORRECTED):
|
||
-------------------
|
||
hyperopt_dqn_demo \
|
||
--parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet \
|
||
--trials 50 \
|
||
--epochs 100 \
|
||
--base-dir /runpod-volume/ml_training/dqn_hyperopt_fixed \
|
||
--early-stopping-min-epochs 50
|
||
|
||
FIXES APPLIED:
|
||
- ❌ --timeout → REMOVED (not supported)
|
||
- ❌ --output-dir → ✅ --base-dir /runpod-volume/ml_training/dqn_hyperopt_fixed
|
||
- ❌ --checkpoint-dir → REMOVED (handled by base-dir)
|
||
- ❌ --min-epochs-before-stopping → ✅ --early-stopping-min-epochs 50
|
||
- ❌ --learning-rate → REMOVED (determined by hyperopt)
|
||
- ✅ Added: --epochs 100
|
||
|
||
Expected Duration: 24-36 hours
|
||
Expected Cost: $6.00-$9.00
|
||
|
||
Success Criteria:
|
||
- Action distribution: 20-40% each (BUY/SELL/HOLD)
|
||
- Reward std > 0.1 (no constant reward warnings)
|
||
- Q-values balanced (divergence < 100)
|
||
- Final backtest: > 10% return, Sharpe > 1.5
|
||
|
||
===============================================================================
|
||
MONITORING
|
||
===============================================================================
|
||
|
||
Monitor logs (if available):
|
||
python3 scripts/python/runpod/monitor_logs.py qarw3nchfoz5mk # MAMBA2
|
||
python3 scripts/python/runpod/monitor_logs.py iyh6whl578olaq # DQN
|
||
|
||
RunPod Console:
|
||
https://www.runpod.io/console/pods
|
||
|
||
Jupyter Access (after 2-3 min initialization):
|
||
MAMBA2: https://qarw3nchfoz5mk-8888.proxy.runpod.net
|
||
DQN: https://iyh6whl578olaq-8888.proxy.runpod.net
|
||
|
||
SSH Access:
|
||
MAMBA2: ssh root@qarw3nchfoz5mk.ssh.runpod.io
|
||
DQN: ssh root@iyh6whl578olaq.ssh.runpod.io
|
||
|
||
===============================================================================
|
||
CRITICAL SUCCESS FACTORS
|
||
===============================================================================
|
||
|
||
✅ Both pods deployed to EUR-IS-1 (volume location)
|
||
✅ Correct CLI arguments (no invalid flags)
|
||
✅ Both using jgrusewski/foxhunt-hyperopt:latest image
|
||
✅ Network volume se3zdnb5o4 mounted at /runpod-volume
|
||
✅ Private Docker registry auth configured
|
||
✅ Status: RUNNING for both pods
|
||
|
||
===============================================================================
|
||
NEXT STEPS
|
||
===============================================================================
|
||
|
||
1. Wait 2-3 minutes for container initialization
|
||
2. Check logs via RunPod console or monitor_logs.py
|
||
3. Verify training starts without CLI argument errors
|
||
4. Monitor S3 bucket for checkpoints:
|
||
aws s3 ls s3://se3zdnb5o4/ml_training/mamba2_hyperopt_batch256/ --profile runpod --recursive
|
||
aws s3 ls s3://se3zdnb5o4/ml_training/dqn_hyperopt_fixed/ --profile runpod --recursive
|
||
5. Pods will auto-terminate when training completes (if configured)
|
||
|
||
===============================================================================
|
||
TOTAL EXPECTED COST
|
||
===============================================================================
|
||
|
||
MAMBA2: 46-65 hours × $0.25/hr = $11.50-$16.25
|
||
DQN: 24-36 hours × $0.25/hr = $6.00-$9.00
|
||
TOTAL: $17.50-$25.25 (assumes no errors, full completion)
|
||
|
||
===============================================================================
|