foxhunt/DQN_RETRAIN_VALIDATION_CHECKLIST.md

DQN Retrain Validation Checklist

Purpose: Validate that the DQN retrain on Runpod is working correctly with fixed reward function and monitoring.

Date: 2025-11-01 Status: Ready for deployment

---

Pre-Deployment Validation

Prerequisites Check

• Task 1 complete: Reward function monitoring added to ml/src/trainers/dqn.rs
• Task 2 complete: Training monitor added with action/Q-value tracking
• Task 3 complete: min_replay_size and min_epochs_before_stopping configurable
• Code compiles: cargo build -p ml --example train_dqn --release
• Deployment script created: deploy_dqn_retrain.sh

Docker Image Verification

• Image has train_dqn binary at /usr/local/bin/train_dqn
• Image supports CUDA (RTX A4000 compatible)
• Volume mount configured: /runpod-volume/
• Test data available: /runpod-volume/test_data/ES_FUT_180d.parquet

Configuration Parameters

Epochs: 100
Min epochs before stopping: 50
Learning rate: 0.0001
Batch size: 32
Gamma: 0.9626
Epsilon: 0.3 → 0.05 (decay 0.995)
Buffer size: 104346
Min replay size: 500
Checkpoint frequency: 10

---

Deployment Validation

Step 1: Deploy Pod

./deploy_dqn_retrain.sh

Expected Output:

• ✅ Virtual environment activated
• ✅ runpod module available
• ✅ DQN code compiles
• ✅ Pod deployed to EUR-IS-1 (RTX A4000)
• ✅ Real-time log streaming starts

Step 2: Monitor Training Logs

Epoch 1-10 (Exploration Phase) Look for these indicators:

• Training starts: "🏋️ Starting training..."
• Replay buffer fills: "Building replay buffer: 500/104346"
• First checkpoint saved: "💾 Checkpoint saved: epoch_10.safetensors"
• Action distribution logged every 10 epochs

Healthy Signals:

Action Distribution [Epoch 10]:
  BUY=28.5% (1423) | SELL=31.2% (1556) | HOLD=40.3% (2011)

Average Q-values [Epoch 10]:
  BUY=0.1234 | SELL=0.1189 | HOLD=0.1201

Reward Variance Check:

Reward std=0.152 (HEALTHY - variance > 0.1)

RED FLAGS (should NOT appear):

⚠️  CONSTANT REWARDS DETECTED! std=0.001
⚠️  LOW ACTION DIVERSITY: BUY only 5.2%
⚠️  Q-VALUE DIVERGENCE: BUY=1500.2, SELL=0.5

Step 3: Mid-Training Check (Epoch 50)

Expected Behavior:

• Epsilon decayed to ~0.15 (50% of initial 0.3)
• Replay buffer full: 104346/104346
• Action distribution balanced (20-40% each)
• Q-values converging (all within 50% of each other)
• Reward std > 0.1

Check Checkpoints:

aws s3 ls s3://se3zdnb5o4/ml_training/dqn_fixed_reward/checkpoints/ \
  --profile runpod --recursive

Expected Files:

• dqn_epoch_10.safetensors
• dqn_epoch_20.safetensors
• dqn_epoch_30.safetensors
• dqn_epoch_40.safetensors
• dqn_epoch_50.safetensors

Step 4: Training Completion (Epoch 100)

Expected Final Output:

✅ Training completed successfully!

📊 Final Metrics:
  • Final loss: 0.XXXXXX
  • Epochs trained: 100
  • Training time: 45.2 min
  • Actual elapsed time: 50.1s
  • Convergence: ✅ Yes

  • Average Q-value: 0.XXXX
  • Final epsilon: 0.05
  • Average gradient norm: 0.XXXXXX

💾 Saving final model to: dqn_final_epoch100.safetensors
✅ Final model saved: 12345678 bytes

🎉 DQN training complete!

Final Validation:

• 100 epochs completed
• Final checkpoint saved
• No constant reward warnings
• Action diversity maintained (20-40% each)
• Q-values balanced

---

Post-Deployment Validation

Step 5: Download and Verify Checkpoints

# Download final checkpoint
aws s3 cp s3://se3zdnb5o4/ml_training/dqn_fixed_reward/dqn_final_epoch100.safetensors \
  ml/trained_models/ --profile runpod

# Verify checkpoint size (should be ~12-15 MB)
ls -lh ml/trained_models/dqn_final_epoch100.safetensors

Expected Size: 12-15 MB (225 features × 3 actions × network layers)

Step 6: Load and Test Model

# Test loading checkpoint
cargo run -p ml --example evaluate_dqn --release -- \
  --checkpoint ml/trained_models/dqn_final_epoch100.safetensors \
  --test-data test_data/ES_FUT_unseen.parquet

Expected Output:

• ✅ Model loads successfully
• ✅ Predictions generated for test data
• ✅ Action distribution balanced
• ✅ No errors/panics

---

Troubleshooting

Issue: "CONSTANT REWARDS DETECTED"

Cause: Reward function returning same value every step Fix:

1. Check if price data is loading correctly
2. Verify calculate_reward() uses actual price changes
3. Confirm next_close != current_close for most samples

Issue: "LOW ACTION DIVERSITY - BUY only 5%"

Cause: Model stuck in one action (HOLD bias) Fix:

1. Check epsilon decay (should be gradual)
2. Verify Q-value updates for all actions
3. Increase initial epsilon (currently 0.3)

Issue: "Q-VALUE DIVERGENCE"

Cause: One action's Q-values exploding Fix:

1. Check reward scaling (should be ±1.0 max)
2. Verify gamma parameter (0.9626 is correct)
3. Check gradient clipping in optimizer

Issue: Pod hangs at "Building replay buffer"

Cause: Data loading or memory issue Fix:

1. Check Runpod logs for OOM errors
2. Verify parquet file exists at /runpod-volume/test_data/ES_FUT_180d.parquet
3. Reduce buffer_size if OOM

---

Success Criteria

Training is successful if ALL of the following are true:

1. ✅ 100 epochs complete without crashes
2. ✅ No "CONSTANT REWARDS" warnings after epoch 10
3. ✅ Action distribution: 20-40% for EACH action (BUY/SELL/HOLD)
4. ✅ Reward std > 0.1 at all epochs
5. ✅ Q-values balanced (max divergence < 100)
6. ✅ Final checkpoint saved and loadable
7. ✅ Total cost < $0.50 (2 hours × $0.25/hr)

---

Cost Tracking

Estimated Cost: $0.25 - $0.50 Actual Cost: [Fill after deployment] Duration: [Fill after deployment] GPU Used: [Fill after deployment]

---

Next Steps After Successful Retrain

1. Compare with old model:

   • Load old checkpoint (stopped at epoch 50)
   • Compare action distributions
   • Verify new model has better balance

2. Backtest performance:

   cargo run -p ml --example backtest_dqn_replay -- \
     --checkpoint ml/trained_models/dqn_final_epoch100.safetensors \
     --test-data test_data/ES_FUT_unseen.parquet
   

3. Update CLAUDE.md:

   • Mark DQN as ✅ CERTIFIED
   • Update test pass rate
   • Add to production deployment checklist

4. Deploy to production:

   • Follow PRODUCTION_DEPLOYMENT_CHECKLIST.md
   • Enable Grafana monitoring
   • Start paper trading validation

---

References

• Deployment Script: deploy_dqn_retrain.sh
• Training Code: ml/examples/train_dqn.rs
• Trainer Logic: ml/src/trainers/dqn.rs
• Runpod Guide: RUNPOD_DEPLOY_QUICK_REF.md
• Docker Image: Dockerfile.foxhunt-build