2df1ea92e1
BREAKING CHANGES: - Removed orphaned dqn.rs monolithic trainer (4,975 lines) - Removed orphaned dqn_ensemble.rs module (816 lines) - Removed orphaned tft.rs and tft_complete_int8_integration_test.rs - TFT trainer split into modular directory structure DQN Module Refactoring: - Split trainers/dqn.rs into modular structure (config.rs, statistics.rs, trainer.rs) - Fixed hyperopt 39D search space (continuous params only) - Boolean flags (use_dueling, use_double_dqn, use_per, use_noisy_nets) are now FIXED architectural decisions - use_distributional defaults to false (Candle BUG #36 - scatter_add gradient issues) Clean Module Structure: - ml/src/trainers/dqn/ directory with proper mod.rs exports - ml/src/trainers/tft/ directory with config.rs, types.rs, model.rs, trainer.rs, tests.rs - All P0 features validated: TD-error clamping, batch diversity, LR scheduler, priority staleness Documentation: - Added comprehensive docs in docs/codebase-cleanup/ - ADR-001 for DQN refactoring decisions - Rainbow DQN component matrix and quick reference guides Build Status: Compiles with zero errors 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
73 lines
1.7 KiB
Bash
Executable File
73 lines
1.7 KiB
Bash
Executable File
#!/bin/bash
|
|
# Check status of both hyperopt pods
|
|
source .env.runpod
|
|
|
|
DQN_POD_ID="dy2bn5ninzaxma"
|
|
PPO_POD_ID="dytpb1mcqwj54t"
|
|
|
|
echo "========================================="
|
|
echo "HYPEROPT PODS STATUS"
|
|
echo "========================================="
|
|
echo ""
|
|
|
|
# GraphQL query to get pod status
|
|
QUERY=$(cat <<'EOF'
|
|
{
|
|
myself {
|
|
pods {
|
|
id
|
|
name
|
|
desiredStatus
|
|
runtime {
|
|
uptimeInSeconds
|
|
}
|
|
machine {
|
|
gpuType {
|
|
displayName
|
|
}
|
|
dataCenterId
|
|
}
|
|
costPerHr
|
|
}
|
|
}
|
|
}
|
|
EOF
|
|
)
|
|
|
|
# Query RunPod API
|
|
RESPONSE=$(curl -s -X POST https://api.runpod.io/graphql \
|
|
-H "Content-Type: application/json" \
|
|
-H "Authorization: Bearer ${RUNPOD_API_KEY}" \
|
|
-d "{\"query\": $(echo "$QUERY" | jq -Rs .)}")
|
|
|
|
# Parse and display
|
|
echo "DQN Pod (${DQN_POD_ID}):"
|
|
echo "$RESPONSE" | jq -r ".data.myself.pods[] | select(.id == \"${DQN_POD_ID}\") |
|
|
\" Status: \(.desiredStatus)
|
|
GPU: \(.machine.gpuType.displayName)
|
|
Datacenter: \(.machine.dataCenterId)
|
|
Cost: $\(.costPerHr)/hr
|
|
Uptime: \(.runtime.uptimeInSeconds)s\""
|
|
echo ""
|
|
|
|
echo "PPO Pod (${PPO_POD_ID}):"
|
|
echo "$RESPONSE" | jq -r ".data.myself.pods[] | select(.id == \"${PPO_POD_ID}\") |
|
|
\" Status: \(.desiredStatus)
|
|
GPU: \(.machine.gpuType.displayName)
|
|
Datacenter: \(.machine.dataCenterId)
|
|
Cost: $\(.costPerHr)/hr
|
|
Uptime: \(.runtime.uptimeInSeconds)s\""
|
|
echo ""
|
|
|
|
echo "========================================="
|
|
echo "MONITORING URLS"
|
|
echo "========================================="
|
|
echo "Dashboard: https://www.runpod.io/console/pods"
|
|
echo ""
|
|
echo "DQN Logs:"
|
|
echo " https://www.runpod.io/console/pods/${DQN_POD_ID}"
|
|
echo ""
|
|
echo "PPO Logs:"
|
|
echo " https://www.runpod.io/console/pods/${PPO_POD_ID}"
|
|
echo ""
|