foxhunt/DQN_RUNPOD_MONITORING_GUIDE.md

DQN Runpod Monitoring Quick Reference

Purpose: Quick commands for monitoring DQN training on Runpod

---

Real-Time Monitoring

View Training Logs (Automatic with --monitor)

The deployment script automatically streams logs when using --monitor flag.

Manual Log Monitoring:

# If you need to reconnect to logs
export PYTHONPATH=/home/jgrusewski/Work/foxhunt:$PYTHONPATH
source .venv/bin/activate

python3 scripts/monitor_logs.py <pod_id>

---

Key Metrics to Watch

1. Reward Variance (Every Epoch)

Look for: "Reward std=0.152"
✅ GOOD: std > 0.1 (healthy variance)
⚠️  BAD:  std < 0.01 (constant rewards - training bug)

2. Action Distribution (Every 10 Epochs)

Action Distribution [Epoch 10]:
  BUY=28.5% (1423) | SELL=31.2% (1556) | HOLD=40.3% (2011)

✅ GOOD: Each action 20-40%
⚠️  BAD:  One action < 10% or > 80%

3. Q-Value Balance (Every 10 Epochs)

Average Q-values [Epoch 10]:
  BUY=0.1234 | SELL=0.1189 | HOLD=0.1201

✅ GOOD: All within 50% of each other
⚠️  BAD:  One action > 10x another

4. Exploration Decay

Look for: "Final epsilon: 0.XXXX"

Epoch 1:   epsilon ~0.300
Epoch 10:  epsilon ~0.270
Epoch 50:  epsilon ~0.150
Epoch 100: epsilon ~0.050

5. Training Progress

Look for checkpoint saves:
💾 Checkpoint saved: dqn_epoch_10.safetensors (12345678 bytes)
💾 Checkpoint saved: dqn_epoch_20.safetensors (12345678 bytes)
...

---

S3 Checkpoint Verification

List All Checkpoints

aws s3 ls s3://se3zdnb5o4/ml_training/dqn_fixed_reward/checkpoints/ \
  --profile runpod \
  --endpoint-url https://s3api-eur-is-1.runpod.io \
  --recursive

Expected Output:

2025-11-01 10:15:32   12345678 checkpoints/dqn_epoch_10.safetensors
2025-11-01 10:25:45   12345678 checkpoints/dqn_epoch_20.safetensors
2025-11-01 10:35:58   12345678 checkpoints/dqn_epoch_30.safetensors
...

Download Latest Checkpoint

# List checkpoints sorted by time
aws s3 ls s3://se3zdnb5o4/ml_training/dqn_fixed_reward/ \
  --profile runpod \
  --endpoint-url https://s3api-eur-is-1.runpod.io \
  --recursive | sort -k1,2

# Download latest
aws s3 cp s3://se3zdnb5o4/ml_training/dqn_fixed_reward/dqn_final_epoch100.safetensors \
  ml/trained_models/ \
  --profile runpod \
  --endpoint-url https://s3api-eur-is-1.runpod.io

---

Pod Management

Check Pod Status

curl -H "Authorization: Bearer $RUNPOD_API_KEY" \
  https://rest.runpod.io/v1/pods

Response:

{
  "pods": [
    {
      "id": "abc123",
      "name": "foxhunt-training",
      "status": "RUNNING",
      "gpuType": "RTX A4000",
      "runtime": 3600  // seconds
    }
  ]
}

Stop Pod (Graceful)

POD_ID="<your_pod_id>"

curl -X POST \
  -H "Authorization: Bearer $RUNPOD_API_KEY" \
  https://rest.runpod.io/v1/pods/$POD_ID/stop

Terminate Pod (Force)

POD_ID="<your_pod_id>"

curl -X POST \
  -H "Authorization: Bearer $RUNPOD_API_KEY" \
  https://rest.runpod.io/v1/pods/$POD_ID/terminate

---

Training Health Checks

Check 1: Training Started

What to look for: "🏋️ Starting training..." When: Within 1-2 minutes of deployment If missing: Check pod logs for errors

Check 2: Replay Buffer Filling

What to look for: "Building replay buffer: X/104346" When: First 5-10 minutes If stuck: Data loading issue or OOM

Check 3: First Checkpoint

What to look for: "💾 Checkpoint saved: dqn_epoch_10.safetensors" When: ~10-15 minutes If missing: Check S3 credentials or disk space

Check 4: Reward Variance

What to look for: "Reward std > 0.1" When: Every epoch after 10 If failing: Reward function bug (constant rewards)

Check 5: Action Diversity

What to look for: "Action Distribution [Epoch X]" When: Every 10 epochs If imbalanced: Epsilon too low or Q-value bug

---

Cost Monitoring

Calculate Current Cost

# Get pod runtime in seconds
POD_RUNTIME_HOURS=$(echo "scale=2; $RUNTIME_SECONDS / 3600" | bc)

# RTX A4000 = $0.25/hr
COST=$(echo "scale=2; $POD_RUNTIME_HOURS * 0.25" | bc)

echo "Current cost: \$$COST"

Set Cost Alarm (Manual)

# Check every 10 minutes
while true; do
  RUNTIME=$(curl -s -H "Authorization: Bearer $RUNPOD_API_KEY" \
    https://rest.runpod.io/v1/pods/$POD_ID | jq -r '.runtime')

  HOURS=$(echo "scale=2; $RUNTIME / 3600" | bc)
  COST=$(echo "scale=2; $HOURS * 0.25" | bc)

  echo "[$(date)] Runtime: ${HOURS}h | Cost: \$${COST}"

  # Alert if cost > $1.00 (4 hours)
  if (( $(echo "$COST > 1.00" | bc -l) )); then
    echo "⚠️  WARNING: Cost exceeded \$1.00! Consider terminating pod."
  fi

  sleep 600  # 10 minutes
done

---

Troubleshooting Commands

Issue: No logs appearing

# Check if S3 credentials are set
grep -E "RUNPOD_S3" .env.runpod

# Try manual S3 list
aws s3 ls s3://se3zdnb5o4/ \
  --profile runpod \
  --endpoint-url https://s3api-eur-is-1.runpod.io

Issue: Pod stuck

# Check pod status
curl -H "Authorization: Bearer $RUNPOD_API_KEY" \
  https://rest.runpod.io/v1/pods/$POD_ID

# Force restart
curl -X POST \
  -H "Authorization: Bearer $RUNPOD_API_KEY" \
  https://rest.runpod.io/v1/pods/$POD_ID/restart

Issue: OOM errors

# Check pod logs for memory errors
python3 scripts/monitor_logs.py $POD_ID | grep -i "out of memory\|OOM\|killed"

# If OOM, reduce batch_size or buffer_size and redeploy

---

Expected Timeline

Time	Milestone	What to Check
0-2 min	Pod deployed	Logs start streaming
2-5 min	Training starts	"🏋️ Starting training..."
5-10 min	Replay buffer fills	"Building replay buffer: 500/104346"
10-15 min	Epoch 10 complete	First checkpoint saved
15-30 min	Action diversity logs	"Action Distribution [Epoch 10]"
30-60 min	Epoch 50 complete	Mid-training checkpoint
60-90 min	Epoch 100 complete	Final checkpoint saved
90-120 min	Training complete	"🎉 DQN training complete!"

---

Quick Checks (Copy-Paste)

# 1. Check if pod is running
curl -H "Authorization: Bearer $RUNPOD_API_KEY" \
  https://rest.runpod.io/v1/pods | jq -r '.pods[] | select(.name=="foxhunt-training") | .status'

# 2. Count checkpoints saved
aws s3 ls s3://se3zdnb5o4/ml_training/dqn_fixed_reward/checkpoints/ \
  --profile runpod \
  --endpoint-url https://s3api-eur-is-1.runpod.io \
  --recursive | wc -l

# 3. Get latest checkpoint timestamp
aws s3 ls s3://se3zdnb5o4/ml_training/dqn_fixed_reward/ \
  --profile runpod \
  --endpoint-url https://s3api-eur-is-1.runpod.io \
  --recursive | sort -k1,2 | tail -1

# 4. Estimate current cost
POD_RUNTIME=$(curl -s -H "Authorization: Bearer $RUNPOD_API_KEY" \
  https://rest.runpod.io/v1/pods | jq -r '.pods[] | select(.name=="foxhunt-training") | .runtime')
echo "Runtime: $(echo "scale=2; $POD_RUNTIME / 3600" | bc)h | Cost: \$$(echo "scale=2; $POD_RUNTIME / 3600 * 0.25" | bc)"

---

When to Terminate Pod

Terminate if:

• ✅ Training completes successfully (100 epochs)
• ❌ Constant reward warnings persist after epoch 20
• ❌ Action diversity < 10% for any action after epoch 30
• ❌ Q-value divergence > 1000 after epoch 50
• ❌ Cost exceeds $1.00 (4 hours) without progress
• ❌ OOM errors or repeated crashes

Keep running if:

• Training is progressing normally
• Checkpoints saving every 10 epochs
• Reward variance > 0.1
• Action diversity 20-40% each
• Cost < $0.50

---

References

• Deployment Script: deploy_dqn_retrain.sh
• Validation Checklist: DQN_RETRAIN_VALIDATION_CHECKLIST.md
• Runpod Quick Ref: RUNPOD_DEPLOY_QUICK_REF.md