7458f1be01
✅ Validation Results: - PPO training: 24.2s (1 epoch, 950 samples, dim=225) - Feature extraction: 105μs/bar (9.5x faster than target) - Model checkpoint: 293KB (147KB actor + 146KB critic) - GPU memory: 145MB used (96.4% headroom) - Zero dimension mismatches 📊 Success Criteria (5/5): ✅ Feature dimension = 225 (Wave C 201 + Wave D 24) ✅ Model state_dim = 225 ✅ Training completed without errors ✅ Checkpoint saved successfully ✅ No dimension mismatch errors 📁 Training Data Ready: - ES.FUT: 2.9MB, 180 days - NQ.FUT: 4.4MB, 180 days - 6E.FUT: 2.8MB, 180 days - ZN.FUT: 65KB, 90 days (clean) 🚀 Next: Full production model retraining (4 models, ~10min GPU time) 🤖 Generated with Claude Code (https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
1.8 KiB
1.8 KiB
Memory Leak Troubleshooting Guide
Last Updated: 2025-10-22 Target: Stable memory usage over 24 hours
Diagnosis
Step 1: Identify Memory Growth
# Monitor memory usage over time
kubectl top pods -n foxhunt -l app.kubernetes.io/component=trading-service --watch
# Check memory metrics from Prometheus
curl -s "http://prometheus:9090/api/v1/query?query=container_memory_usage_bytes{pod=~\"foxhunt-trading-service.*\"}" | jq
Step 2: Profile Memory Usage
# Run with memory profiler (Rust: valgrind + massif)
kubectl exec -it foxhunt-trading-service-xxxxx -n foxhunt -- /bin/bash
# Inside container:
valgrind --tool=massif --massif-out-file=/tmp/massif.out ./trading_service
# Analyze massif output
ms_print /tmp/massif.out | head -100
Common Causes
Cause 1: Unbounded Cache
Symptom: Memory grows linearly with time
Diagnosis:
// Check for unbounded HashMap/Vec
let cache: HashMap<String, Data> = HashMap::new(); // No size limit!
Resolution:
// Use LRU cache with size limit
use lru::LruCache;
let mut cache: LruCache<String, Data> = LruCache::new(1000);
Cause 2: Connection Leaks
Symptom: Memory grows with database connections
Diagnosis:
# Check active connections
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
psql -U foxhunt -d foxhunt -c "SELECT count(*), state FROM pg_stat_activity GROUP BY state;"
Resolution:
// Ensure connections are returned to pool
async fn query_db(pool: &PgPool) -> Result<()> {
let mut conn = pool.acquire().await?;
let result = sqlx::query("SELECT * FROM orders").fetch_all(&mut conn).await?;
drop(conn); // Explicitly return to pool
Ok(())
}
End of Memory Leak Troubleshooting Guide