# DQN Retrain Validation Checklist

**Purpose**: Validate that the DQN retrain on Runpod is working correctly with fixed reward function and monitoring.

**Date**: 2025-11-01
**Status**: Ready for deployment

---

## Pre-Deployment Validation

### Prerequisites Check
- [x] Task 1 complete: Reward function monitoring added to `ml/src/trainers/dqn.rs`
- [x] Task 2 complete: Training monitor added with action/Q-value tracking
- [x] Task 3 complete: `min_replay_size` and `min_epochs_before_stopping` configurable
- [x] Code compiles: `cargo build -p ml --example train_dqn --release`
- [x] Deployment script created: `deploy_dqn_retrain.sh`

### Docker Image Verification
- [x] Image has `train_dqn` binary at `/usr/local/bin/train_dqn`
- [x] Image supports CUDA (RTX A4000 compatible)
- [x] Volume mount configured: `/runpod-volume/`
- [x] Test data available: `/runpod-volume/test_data/ES_FUT_180d.parquet`

### Configuration Parameters
```bash
Epochs: 100
Min epochs before stopping: 50
Learning rate: 0.0001
Batch size: 32
Gamma: 0.9626
Epsilon: 0.3 → 0.05 (decay 0.995)
Buffer size: 104346
Min replay size: 500
Checkpoint frequency: 10
```

---

## Deployment Validation

### Step 1: Deploy Pod
```bash
./deploy_dqn_retrain.sh
```

**Expected Output**:
- ✅ Virtual environment activated
- ✅ runpod module available
- ✅ DQN code compiles
- ✅ Pod deployed to EUR-IS-1 (RTX A4000)
- ✅ Real-time log streaming starts

### Step 2: Monitor Training Logs

**Epoch 1-10 (Exploration Phase)**
Look for these indicators:
- [ ] Training starts: "🏋️  Starting training..."
- [ ] Replay buffer fills: "Building replay buffer: 500/104346"
- [ ] First checkpoint saved: "💾 Checkpoint saved: epoch_10.safetensors"
- [ ] Action distribution logged every 10 epochs

**Healthy Signals**:
```
Action Distribution [Epoch 10]:
  BUY=28.5% (1423) | SELL=31.2% (1556) | HOLD=40.3% (2011)

Average Q-values [Epoch 10]:
  BUY=0.1234 | SELL=0.1189 | HOLD=0.1201
```

**Reward Variance Check**:
```
Reward std=0.152 (HEALTHY - variance > 0.1)
```

**RED FLAGS** (should NOT appear):
```
⚠️  CONSTANT REWARDS DETECTED! std=0.001
⚠️  LOW ACTION DIVERSITY: BUY only 5.2%
⚠️  Q-VALUE DIVERGENCE: BUY=1500.2, SELL=0.5
```

### Step 3: Mid-Training Check (Epoch 50)

**Expected Behavior**:
- [ ] Epsilon decayed to ~0.15 (50% of initial 0.3)
- [ ] Replay buffer full: 104346/104346
- [ ] Action distribution balanced (20-40% each)
- [ ] Q-values converging (all within 50% of each other)
- [ ] Reward std > 0.1

**Check Checkpoints**:
```bash
aws s3 ls s3://se3zdnb5o4/ml_training/dqn_fixed_reward/checkpoints/ \
  --profile runpod --recursive
```

**Expected Files**:
- `dqn_epoch_10.safetensors`
- `dqn_epoch_20.safetensors`
- `dqn_epoch_30.safetensors`
- `dqn_epoch_40.safetensors`
- `dqn_epoch_50.safetensors`

### Step 4: Training Completion (Epoch 100)

**Expected Final Output**:
```
✅ Training completed successfully!

📊 Final Metrics:
  • Final loss: 0.XXXXXX
  • Epochs trained: 100
  • Training time: 45.2 min
  • Actual elapsed time: 50.1s
  • Convergence: ✅ Yes

  • Average Q-value: 0.XXXX
  • Final epsilon: 0.05
  • Average gradient norm: 0.XXXXXX

💾 Saving final model to: dqn_final_epoch100.safetensors
✅ Final model saved: 12345678 bytes

🎉 DQN training complete!
```

**Final Validation**:
- [ ] 100 epochs completed
- [ ] Final checkpoint saved
- [ ] No constant reward warnings
- [ ] Action diversity maintained (20-40% each)
- [ ] Q-values balanced

---

## Post-Deployment Validation

### Step 5: Download and Verify Checkpoints

```bash
# Download final checkpoint
aws s3 cp s3://se3zdnb5o4/ml_training/dqn_fixed_reward/dqn_final_epoch100.safetensors \
  ml/trained_models/ --profile runpod

# Verify checkpoint size (should be ~12-15 MB)
ls -lh ml/trained_models/dqn_final_epoch100.safetensors
```

**Expected Size**: 12-15 MB (225 features × 3 actions × network layers)

### Step 6: Load and Test Model

```bash
# Test loading checkpoint
cargo run -p ml --example evaluate_dqn --release -- \
  --checkpoint ml/trained_models/dqn_final_epoch100.safetensors \
  --test-data test_data/ES_FUT_unseen.parquet
```

**Expected Output**:
- ✅ Model loads successfully
- ✅ Predictions generated for test data
- ✅ Action distribution balanced
- ✅ No errors/panics

---

## Troubleshooting

### Issue: "CONSTANT REWARDS DETECTED"
**Cause**: Reward function returning same value every step
**Fix**:
1. Check if price data is loading correctly
2. Verify `calculate_reward()` uses actual price changes
3. Confirm `next_close != current_close` for most samples

### Issue: "LOW ACTION DIVERSITY - BUY only 5%"
**Cause**: Model stuck in one action (HOLD bias)
**Fix**:
1. Check epsilon decay (should be gradual)
2. Verify Q-value updates for all actions
3. Increase initial epsilon (currently 0.3)

### Issue: "Q-VALUE DIVERGENCE"
**Cause**: One action's Q-values exploding
**Fix**:
1. Check reward scaling (should be ±1.0 max)
2. Verify gamma parameter (0.9626 is correct)
3. Check gradient clipping in optimizer

### Issue: Pod hangs at "Building replay buffer"
**Cause**: Data loading or memory issue
**Fix**:
1. Check Runpod logs for OOM errors
2. Verify parquet file exists at `/runpod-volume/test_data/ES_FUT_180d.parquet`
3. Reduce `buffer_size` if OOM

---

## Success Criteria

**Training is successful if ALL of the following are true**:
1. ✅ 100 epochs complete without crashes
2. ✅ No "CONSTANT REWARDS" warnings after epoch 10
3. ✅ Action distribution: 20-40% for EACH action (BUY/SELL/HOLD)
4. ✅ Reward std > 0.1 at all epochs
5. ✅ Q-values balanced (max divergence < 100)
6. ✅ Final checkpoint saved and loadable
7. ✅ Total cost < $0.50 (2 hours × $0.25/hr)

---

## Cost Tracking

**Estimated Cost**: $0.25 - $0.50
**Actual Cost**: _[Fill after deployment]_
**Duration**: _[Fill after deployment]_
**GPU Used**: _[Fill after deployment]_

---

## Next Steps After Successful Retrain

1. **Compare with old model**:
   - Load old checkpoint (stopped at epoch 50)
   - Compare action distributions
   - Verify new model has better balance

2. **Backtest performance**:
   ```bash
   cargo run -p ml --example backtest_dqn_replay -- \
     --checkpoint ml/trained_models/dqn_final_epoch100.safetensors \
     --test-data test_data/ES_FUT_unseen.parquet
   ```

3. **Update CLAUDE.md**:
   - Mark DQN as ✅ CERTIFIED
   - Update test pass rate
   - Add to production deployment checklist

4. **Deploy to production**:
   - Follow PRODUCTION_DEPLOYMENT_CHECKLIST.md
   - Enable Grafana monitoring
   - Start paper trading validation

---

## References

- **Deployment Script**: `deploy_dqn_retrain.sh`
- **Training Code**: `ml/examples/train_dqn.rs`
- **Trainer Logic**: `ml/src/trainers/dqn.rs`
- **Runpod Guide**: `RUNPOD_DEPLOY_QUICK_REF.md`
- **Docker Image**: `Dockerfile.foxhunt-build`