2df1ea92e1
BREAKING CHANGES: - Removed orphaned dqn.rs monolithic trainer (4,975 lines) - Removed orphaned dqn_ensemble.rs module (816 lines) - Removed orphaned tft.rs and tft_complete_int8_integration_test.rs - TFT trainer split into modular directory structure DQN Module Refactoring: - Split trainers/dqn.rs into modular structure (config.rs, statistics.rs, trainer.rs) - Fixed hyperopt 39D search space (continuous params only) - Boolean flags (use_dueling, use_double_dqn, use_per, use_noisy_nets) are now FIXED architectural decisions - use_distributional defaults to false (Candle BUG #36 - scatter_add gradient issues) Clean Module Structure: - ml/src/trainers/dqn/ directory with proper mod.rs exports - ml/src/trainers/tft/ directory with config.rs, types.rs, model.rs, trainer.rs, tests.rs - All P0 features validated: TD-error clamping, batch diversity, LR scheduler, priority staleness Documentation: - Added comprehensive docs in docs/codebase-cleanup/ - ADR-001 for DQN refactoring decisions - Rainbow DQN component matrix and quick reference guides Build Status: Compiles with zero errors 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
60 lines
1.9 KiB
Bash
Executable File
60 lines
1.9 KiB
Bash
Executable File
#!/bin/bash
|
|
# Verify hyperopt training is actually running (not just pod running)
|
|
source .env.runpod
|
|
|
|
DQN_POD_ID="dy2bn5ninzaxma"
|
|
PPO_POD_ID="dytpb1mcqwj54t"
|
|
|
|
echo "========================================="
|
|
echo "VERIFYING HYPEROPT TRAINING EXECUTION"
|
|
echo "========================================="
|
|
echo ""
|
|
|
|
# Wait for pods to initialize
|
|
echo "Waiting 30 seconds for pods to initialize..."
|
|
sleep 30
|
|
|
|
echo ""
|
|
echo "Checking DQN pod logs for training activity..."
|
|
echo "----------------------------------------"
|
|
DQN_LOGS=$(curl -s "https://www.runpod.io/console/pods/${DQN_POD_ID}/logs" 2>/dev/null || echo "Log API not accessible")
|
|
|
|
if echo "$DQN_LOGS" | grep -q "hyperopt_dqn_demo"; then
|
|
echo "✅ DQN training command detected"
|
|
elif echo "$DQN_LOGS" | grep -q "Trial"; then
|
|
echo "✅ DQN trials running"
|
|
else
|
|
echo "⚠️ Cannot verify DQN training from logs (check dashboard)"
|
|
fi
|
|
|
|
echo ""
|
|
echo "Checking PPO pod logs for training activity..."
|
|
echo "----------------------------------------"
|
|
PPO_LOGS=$(curl -s "https://www.runpod.io/console/pods/${PPO_POD_ID}/logs" 2>/dev/null || echo "Log API not accessible")
|
|
|
|
if echo "$PPO_LOGS" | grep -q "hyperopt_ppo_demo"; then
|
|
echo "✅ PPO training command detected"
|
|
elif echo "$PPO_LOGS" | grep -q "Trial"; then
|
|
echo "✅ PPO trials running"
|
|
else
|
|
echo "⚠️ Cannot verify PPO training from logs (check dashboard)"
|
|
fi
|
|
|
|
echo ""
|
|
echo "========================================="
|
|
echo "MANUAL VERIFICATION"
|
|
echo "========================================="
|
|
echo ""
|
|
echo "Check DQN logs manually:"
|
|
echo " https://www.runpod.io/console/pods/${DQN_POD_ID}"
|
|
echo ""
|
|
echo "Check PPO logs manually:"
|
|
echo " https://www.runpod.io/console/pods/${PPO_POD_ID}"
|
|
echo ""
|
|
echo "Look for:"
|
|
echo " - 'Starting hyperopt_*_demo' messages"
|
|
echo " - 'Trial 1/50' progress indicators"
|
|
echo " - GPU memory allocation logs"
|
|
echo " - No error messages (OOM, CUDA errors)"
|
|
echo ""
|