# GPU Errors Troubleshooting Guide

**Last Updated**: 2025-10-22

---

## GPU Not Detected

### Symptom: `nvidia-smi: command not found` in container

**Diagnosis**:
```bash
# Check GPU availability on node
kubectl get nodes -o json | jq '.items[].status.allocatable | select(.["nvidia.com/gpu"] != null)'

# Check NVIDIA device plugin
kubectl get pods -n kube-system | grep nvidia-device-plugin
```

**Resolution**:
```bash
# Install NVIDIA GPU Operator
helm install gpu-operator nvidia/gpu-operator \
    --namespace gpu-operator-resources \
    --create-namespace

# Verify GPU detected
kubectl describe node gpu-node-1 | grep nvidia.com/gpu
```

---

## CUDA Out of Memory

### Symptom: `CUDA error: out of memory`

**Diagnosis**:
```bash
# Check GPU memory usage
kubectl exec -n foxhunt foxhunt-ml-training-service-xxxxx -- nvidia-smi

# Output:
# GPU  Memory-Usage
#  0   3800MiB / 4096MiB  (93% - CRITICAL)
```

**Resolution**:
```python
# Reduce batch size
batch_size = 16  # Reduced from 32

# Enable gradient checkpointing
model.gradient_checkpointing_enable()

# Use mixed precision training
from torch.cuda.amp import autocast, GradScaler
scaler = GradScaler()
with autocast():
    output = model(input)
```

---

**End of GPU Errors Troubleshooting Guide**