7458f1be01
✅ Validation Results: - PPO training: 24.2s (1 epoch, 950 samples, dim=225) - Feature extraction: 105μs/bar (9.5x faster than target) - Model checkpoint: 293KB (147KB actor + 146KB critic) - GPU memory: 145MB used (96.4% headroom) - Zero dimension mismatches 📊 Success Criteria (5/5): ✅ Feature dimension = 225 (Wave C 201 + Wave D 24) ✅ Model state_dim = 225 ✅ Training completed without errors ✅ Checkpoint saved successfully ✅ No dimension mismatch errors 📁 Training Data Ready: - ES.FUT: 2.9MB, 180 days - NQ.FUT: 4.4MB, 180 days - 6E.FUT: 2.8MB, 180 days - ZN.FUT: 65KB, 90 days (clean) 🚀 Next: Full production model retraining (4 models, ~10min GPU time) 🤖 Generated with Claude Code (https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
57 lines
1.2 KiB
Markdown
57 lines
1.2 KiB
Markdown
# Service Crashes Troubleshooting Guide
|
|
|
|
**Last Updated**: 2025-10-22
|
|
|
|
---
|
|
|
|
## OOM Killed (Exit Code 137)
|
|
|
|
### Symptom: Container exits with code 137
|
|
|
|
**Diagnosis**:
|
|
```bash
|
|
# Check pod events
|
|
kubectl describe pod foxhunt-trading-service-xxxxx -n foxhunt | grep -A 10 Events
|
|
|
|
# Output:
|
|
# Reason: OOMKilled
|
|
# Message: Container exceeded memory limit
|
|
|
|
# Check memory limits
|
|
kubectl get pod foxhunt-trading-service-xxxxx -n foxhunt -o jsonpath='{.spec.containers[0].resources.limits.memory}'
|
|
```
|
|
|
|
**Resolution**:
|
|
```bash
|
|
# Increase memory limit
|
|
kubectl patch deployment foxhunt-trading-service -n foxhunt \
|
|
-p '{"spec":{"template":{"spec":{"containers":[{"name":"trading-service","resources":{"limits":{"memory":"4Gi"}}}]}}}}'
|
|
|
|
# OR enable swap (not recommended for production)
|
|
```
|
|
|
|
---
|
|
|
|
## Panic/Segfault
|
|
|
|
### Symptom: Container crashes with panic or segfault
|
|
|
|
**Diagnosis**:
|
|
```bash
|
|
# Check logs before crash
|
|
kubectl logs -n foxhunt foxhunt-trading-service-xxxxx --previous | tail -100
|
|
|
|
# Output:
|
|
# thread 'main' panicked at 'index out of bounds: the len is 10 but the index is 10', src/main.rs:42
|
|
```
|
|
|
|
**Resolution**:
|
|
```bash
|
|
# Fix code bug and deploy new version
|
|
# Add bounds checking in code
|
|
```
|
|
|
|
---
|
|
|
|
**End of Service Crashes Troubleshooting Guide**
|