7458f1be01
✅ Validation Results: - PPO training: 24.2s (1 epoch, 950 samples, dim=225) - Feature extraction: 105μs/bar (9.5x faster than target) - Model checkpoint: 293KB (147KB actor + 146KB critic) - GPU memory: 145MB used (96.4% headroom) - Zero dimension mismatches 📊 Success Criteria (5/5): ✅ Feature dimension = 225 (Wave C 201 + Wave D 24) ✅ Model state_dim = 225 ✅ Training completed without errors ✅ Checkpoint saved successfully ✅ No dimension mismatch errors 📁 Training Data Ready: - ES.FUT: 2.9MB, 180 days - NQ.FUT: 4.4MB, 180 days - 6E.FUT: 2.8MB, 180 days - ZN.FUT: 65KB, 90 days (clean) 🚀 Next: Full production model retraining (4 models, ~10min GPU time) 🤖 Generated with Claude Code (https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
3.1 KiB
3.1 KiB
High Latency Troubleshooting Guide
Last Updated: 2025-10-22 Target: P99 latency <500ms
Diagnosis
Step 1: Identify Latency Source
# Check API Gateway latency
curl -w "@curl-format.txt" -o /dev/null -s http://localhost:8080/health
# Response time: 1250ms (HIGH)
# Check database query performance
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
psql -U foxhunt -d foxhunt -c "SELECT query, mean_exec_time, calls FROM pg_stat_statements ORDER BY mean_exec_time DESC LIMIT 10;"
# Check service-to-service latency
kubectl logs -n foxhunt -l app.kubernetes.io/component=trading-service | grep "request_duration_ms" | tail -100
Step 2: Check Resource Utilization
# CPU usage
kubectl top pods -n foxhunt --sort-by=cpu | head -20
# Memory usage
kubectl top pods -n foxhunt --sort-by=memory | head -20
# Disk I/O (PostgreSQL)
kubectl exec -n foxhunt foxhunt-postgresql-0 -- iostat -x 1 5
Step 3: Analyze Slow Queries
# Enable slow query logging (PostgreSQL)
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
psql -U foxhunt -d foxhunt -c "ALTER SYSTEM SET log_min_duration_statement = 100;" # Log queries >100ms
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
psql -U foxhunt -d foxhunt -c "SELECT pg_reload_conf();"
# View slow queries
kubectl logs -n foxhunt foxhunt-postgresql-0 | grep "duration:" | tail -50
Common Causes
Cause 1: Missing Database Index
Symptom: Queries take >1s, high CPU on database
Diagnosis:
-- Find sequential scans on large tables
SELECT schemaname, tablename, seq_scan, seq_tup_read, idx_scan
FROM pg_stat_user_tables
WHERE seq_scan > 0 AND seq_tup_read > 100000
ORDER BY seq_tup_read DESC
LIMIT 10;
Resolution:
-- Create missing index
CREATE INDEX CONCURRENTLY idx_orders_symbol_created_at ON orders(symbol, created_at DESC);
-- Verify index usage
EXPLAIN ANALYZE SELECT * FROM orders WHERE symbol='ES.FUT' ORDER BY created_at DESC LIMIT 100;
Cause 2: Connection Pool Exhausted
Symptom: timeout acquiring connection from pool
Diagnosis:
kubectl logs -n foxhunt -l app.kubernetes.io/component=trading-service | grep "connection pool"
Resolution:
# Increase pool size via environment variable
kubectl set env deployment/foxhunt-trading-service -n foxhunt DATABASE_MAX_CONNECTIONS=50
# OR scale horizontally
kubectl scale deployment/foxhunt-trading-service -n foxhunt --replicas=10
Cause 3: Network Latency
Symptom: High latency between services
Diagnosis:
# Test inter-service latency
kubectl run -it --rm debug --image=nicolaka/netshoot --restart=Never -- \
ping foxhunt-postgresql.foxhunt.svc.cluster.local
# Check for packet loss
kubectl run -it --rm debug --image=nicolaka/netshoot --restart=Never -- \
mtr --report --report-cycles 10 foxhunt-postgresql.foxhunt.svc.cluster.local
Resolution:
# Verify services are in same availability zone
kubectl get pods -n foxhunt -o wide
# Use node affinity to colocate services
End of High Latency Troubleshooting Guide