7458f1be01
✅ Validation Results: - PPO training: 24.2s (1 epoch, 950 samples, dim=225) - Feature extraction: 105μs/bar (9.5x faster than target) - Model checkpoint: 293KB (147KB actor + 146KB critic) - GPU memory: 145MB used (96.4% headroom) - Zero dimension mismatches 📊 Success Criteria (5/5): ✅ Feature dimension = 225 (Wave C 201 + Wave D 24) ✅ Model state_dim = 225 ✅ Training completed without errors ✅ Checkpoint saved successfully ✅ No dimension mismatch errors 📁 Training Data Ready: - ES.FUT: 2.9MB, 180 days - NQ.FUT: 4.4MB, 180 days - 6E.FUT: 2.8MB, 180 days - ZN.FUT: 65KB, 90 days (clean) 🚀 Next: Full production model retraining (4 models, ~10min GPU time) 🤖 Generated with Claude Code (https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
79 lines
1.8 KiB
Markdown
79 lines
1.8 KiB
Markdown
# Memory Leak Troubleshooting Guide
|
|
|
|
**Last Updated**: 2025-10-22
|
|
**Target**: Stable memory usage over 24 hours
|
|
|
|
---
|
|
|
|
## Diagnosis
|
|
|
|
### Step 1: Identify Memory Growth
|
|
|
|
```bash
|
|
# Monitor memory usage over time
|
|
kubectl top pods -n foxhunt -l app.kubernetes.io/component=trading-service --watch
|
|
|
|
# Check memory metrics from Prometheus
|
|
curl -s "http://prometheus:9090/api/v1/query?query=container_memory_usage_bytes{pod=~\"foxhunt-trading-service.*\"}" | jq
|
|
```
|
|
|
|
### Step 2: Profile Memory Usage
|
|
|
|
```bash
|
|
# Run with memory profiler (Rust: valgrind + massif)
|
|
kubectl exec -it foxhunt-trading-service-xxxxx -n foxhunt -- /bin/bash
|
|
|
|
# Inside container:
|
|
valgrind --tool=massif --massif-out-file=/tmp/massif.out ./trading_service
|
|
|
|
# Analyze massif output
|
|
ms_print /tmp/massif.out | head -100
|
|
```
|
|
|
|
---
|
|
|
|
## Common Causes
|
|
|
|
### Cause 1: Unbounded Cache
|
|
|
|
**Symptom**: Memory grows linearly with time
|
|
|
|
**Diagnosis**:
|
|
```rust
|
|
// Check for unbounded HashMap/Vec
|
|
let cache: HashMap<String, Data> = HashMap::new(); // No size limit!
|
|
```
|
|
|
|
**Resolution**:
|
|
```rust
|
|
// Use LRU cache with size limit
|
|
use lru::LruCache;
|
|
let mut cache: LruCache<String, Data> = LruCache::new(1000);
|
|
```
|
|
|
|
### Cause 2: Connection Leaks
|
|
|
|
**Symptom**: Memory grows with database connections
|
|
|
|
**Diagnosis**:
|
|
```bash
|
|
# Check active connections
|
|
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
|
|
psql -U foxhunt -d foxhunt -c "SELECT count(*), state FROM pg_stat_activity GROUP BY state;"
|
|
```
|
|
|
|
**Resolution**:
|
|
```rust
|
|
// Ensure connections are returned to pool
|
|
async fn query_db(pool: &PgPool) -> Result<()> {
|
|
let mut conn = pool.acquire().await?;
|
|
let result = sqlx::query("SELECT * FROM orders").fetch_all(&mut conn).await?;
|
|
drop(conn); // Explicitly return to pool
|
|
Ok(())
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
**End of Memory Leak Troubleshooting Guide**
|