7458f1be01
✅ Validation Results: - PPO training: 24.2s (1 epoch, 950 samples, dim=225) - Feature extraction: 105μs/bar (9.5x faster than target) - Model checkpoint: 293KB (147KB actor + 146KB critic) - GPU memory: 145MB used (96.4% headroom) - Zero dimension mismatches 📊 Success Criteria (5/5): ✅ Feature dimension = 225 (Wave C 201 + Wave D 24) ✅ Model state_dim = 225 ✅ Training completed without errors ✅ Checkpoint saved successfully ✅ No dimension mismatch errors 📁 Training Data Ready: - ES.FUT: 2.9MB, 180 days - NQ.FUT: 4.4MB, 180 days - 6E.FUT: 2.8MB, 180 days - ZN.FUT: 65KB, 90 days (clean) 🚀 Next: Full production model retraining (4 models, ~10min GPU time) 🤖 Generated with Claude Code (https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
65 lines
1.2 KiB
Markdown
65 lines
1.2 KiB
Markdown
# GPU Errors Troubleshooting Guide
|
|
|
|
**Last Updated**: 2025-10-22
|
|
|
|
---
|
|
|
|
## GPU Not Detected
|
|
|
|
### Symptom: `nvidia-smi: command not found` in container
|
|
|
|
**Diagnosis**:
|
|
```bash
|
|
# Check GPU availability on node
|
|
kubectl get nodes -o json | jq '.items[].status.allocatable | select(.["nvidia.com/gpu"] != null)'
|
|
|
|
# Check NVIDIA device plugin
|
|
kubectl get pods -n kube-system | grep nvidia-device-plugin
|
|
```
|
|
|
|
**Resolution**:
|
|
```bash
|
|
# Install NVIDIA GPU Operator
|
|
helm install gpu-operator nvidia/gpu-operator \
|
|
--namespace gpu-operator-resources \
|
|
--create-namespace
|
|
|
|
# Verify GPU detected
|
|
kubectl describe node gpu-node-1 | grep nvidia.com/gpu
|
|
```
|
|
|
|
---
|
|
|
|
## CUDA Out of Memory
|
|
|
|
### Symptom: `CUDA error: out of memory`
|
|
|
|
**Diagnosis**:
|
|
```bash
|
|
# Check GPU memory usage
|
|
kubectl exec -n foxhunt foxhunt-ml-training-service-xxxxx -- nvidia-smi
|
|
|
|
# Output:
|
|
# GPU Memory-Usage
|
|
# 0 3800MiB / 4096MiB (93% - CRITICAL)
|
|
```
|
|
|
|
**Resolution**:
|
|
```python
|
|
# Reduce batch size
|
|
batch_size = 16 # Reduced from 32
|
|
|
|
# Enable gradient checkpointing
|
|
model.gradient_checkpointing_enable()
|
|
|
|
# Use mixed precision training
|
|
from torch.cuda.amp import autocast, GradScaler
|
|
scaler = GradScaler()
|
|
with autocast():
|
|
output = model(input)
|
|
```
|
|
|
|
---
|
|
|
|
**End of GPU Errors Troubleshooting Guide**
|