# ML Training Validation - Quick Start Guide

**Wave 152 Agent 20** - Fast validation of all ML training pipelines

## 🚀 Quick Start (3 Steps)

### Step 1: Download Test Data (Agent 19)

```bash
cd /home/jgrusewski/Work/foxhunt
./scripts/train_all_models_fixed.sh
```

**Time**: ~10-15 minutes
**Downloads**: 3-month BTC/USD and ETH/USD historical data

### Step 2: Validate Training (Agent 20)

```bash
./scripts/validate_training.sh
```

**Time**: 10-30 minutes (depends on hardware)
**Tests**: All 4 models (DQN, PPO, MAMBA, TFT) with 2 epochs each

### Step 3: Check Results

**Success** (Exit 0):
```
========================================
✓ ALL TESTS PASSED
========================================

All 4 models trained successfully and saved .safetensors files
```

**Failure** (Exit 1):
```
========================================
✗ TESTS FAILED
========================================

Failed: 1/4 models

Check logs for details:
  - test_data/models/PPO_TIMESTAMP.log
```

## 📊 What Gets Tested

| Model | Architecture | Epochs | Output File |
|-------|-------------|--------|-------------|
| DQN | Deep Q-Network | 2 | `dqn_TIMESTAMP.safetensors` |
| PPO | Policy Gradient | 2 | `ppo_TIMESTAMP.safetensors` |
| MAMBA | State Space | 2 | `mamba_TIMESTAMP.safetensors` |
| TFT | Transformer | 2 | `tft_TIMESTAMP.safetensors` |

## 🎯 Success Criteria

✅ All models train without errors
✅ All models save `.safetensors` files
✅ Model files are non-empty (>1MB)
✅ Script exits with code 0

## 🔧 Troubleshooting

### "BTC data not found"

Run Agent 19 first:
```bash
./scripts/train_all_models_fixed.sh
```

### "cargo not found"

Install Rust:
```bash
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source $HOME/.cargo/env
```

### Training too slow

Check if GPU is available:
```bash
nvidia-smi  # Should show GPU utilization
```

If no GPU, reduce epochs or batch size in script.

## 📁 Output Files

After successful run:

```
test_data/models/
├── dqn_20251014_011545.safetensors      # ~15MB
├── ppo_20251014_011547.safetensors      # ~18MB
├── mamba_20251014_011552.safetensors    # ~42MB
├── tft_20251014_011555.safetensors      # ~28MB
├── DQN_20251014_011545.log              # Training logs
├── PPO_20251014_011547.log              # Training logs
├── MAMBA_20251014_011552.log            # Training logs
└── TFT_20251014_011555.log              # Training logs
```

## ⏱️ Expected Timing

| Hardware | Total Time |
|----------|------------|
| RTX 3090 | 8-12 min |
| RTX 3050 Ti | 12-18 min |
| AMD Ryzen 9 (CPU) | 25-35 min |
| Intel i7 (CPU) | 35-50 min |

## 📚 Documentation

- **Full guide**: `scripts/README_validate_training.md`
- **Technical details**: `WAVE_152_AGENT_20_SUMMARY.md`
- **Data preparation**: Agent 19 script

## 🔗 Related Scripts

- **Agent 19**: `train_all_models_fixed.sh` - Data download + full training
- **Agent 18**: `train_all_models_full.sh` - Original training script
- **Agent 17**: `test_dqn_training.sh` - DQN-only test

## ❓ FAQ

**Q: How long does validation take?**
A: 10-30 minutes depending on your hardware (GPU vs CPU).

**Q: Can I run it multiple times?**
A: Yes, each run creates new timestamped output files.

**Q: What if one model fails?**
A: Check the log file for that model in `test_data/models/MODEL_TIMESTAMP.log`.

**Q: Can I modify the number of epochs?**
A: Yes, edit `EPOCHS=2` at the top of `validate_training.sh`.

**Q: Do I need a GPU?**
A: No, but training is 3-5x faster with GPU.

**Q: Can I train models in parallel?**
A: Not in this version (would require 4x memory). Sequential is safer.

## 💡 Pro Tips

1. **Monitor GPU usage** during training:
   ```bash
   watch -n 1 nvidia-smi
   ```

2. **Check model file sizes** after completion:
   ```bash
   ls -lh test_data/models/*.safetensors
   ```

3. **Review training logs** for any warnings:
   ```bash
   grep -i "error\|warn" test_data/models/*.log
   ```

4. **Clean old models** to save disk space:
   ```bash
   rm test_data/models/*_old_timestamp.*
   ```

## 🎓 What This Tests

1. **Data Loading**: Parquet file reading
2. **Feature Engineering**: Technical indicators, OHLCV processing
3. **Model Initialization**: All 4 architectures
4. **Training Loop**: Forward pass, loss calculation, backprop
5. **Checkpoint Saving**: SafeTensors serialization
6. **Error Handling**: Graceful failures, proper logging

## ✅ Pre-Deployment Checklist

- [ ] Data downloaded (Agent 19)
- [ ] Validation script executed
- [ ] All 4 models passed (exit 0)
- [ ] Model files verified (>1MB each)
- [ ] No errors in logs
- [ ] GPU utilized (if available)
- [ ] Timing acceptable for hardware

## 🚢 Production Deployment

Once validation passes:

```bash
# Commit the validated models
git add test_data/models/*.safetensors
git commit -m "Validated ML models ready for production"

# Tag the release
git tag -a v1.0-ml-validated -m "ML training validated"

# Deploy to production
./scripts/deploy_production.sh
```

## 📞 Support

If you encounter issues:

1. Check the troubleshooting section in `README_validate_training.md`
2. Review model-specific log files
3. Verify system requirements (RAM, disk space, CUDA)
4. Consult WAVE_152_AGENT_20_SUMMARY.md for technical details

---

**Last Updated**: 2025-10-14 (Wave 152 Agent 20)
**Status**: Production Ready ✅