7458f1be01
✅ Validation Results: - PPO training: 24.2s (1 epoch, 950 samples, dim=225) - Feature extraction: 105μs/bar (9.5x faster than target) - Model checkpoint: 293KB (147KB actor + 146KB critic) - GPU memory: 145MB used (96.4% headroom) - Zero dimension mismatches 📊 Success Criteria (5/5): ✅ Feature dimension = 225 (Wave C 201 + Wave D 24) ✅ Model state_dim = 225 ✅ Training completed without errors ✅ Checkpoint saved successfully ✅ No dimension mismatch errors 📁 Training Data Ready: - ES.FUT: 2.9MB, 180 days - NQ.FUT: 4.4MB, 180 days - 6E.FUT: 2.8MB, 180 days - ZN.FUT: 65KB, 90 days (clean) 🚀 Next: Full production model retraining (4 models, ~10min GPU time) 🤖 Generated with Claude Code (https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
123 lines
3.1 KiB
Markdown
123 lines
3.1 KiB
Markdown
# High Latency Troubleshooting Guide
|
|
|
|
**Last Updated**: 2025-10-22
|
|
**Target**: P99 latency <500ms
|
|
|
|
---
|
|
|
|
## Diagnosis
|
|
|
|
### Step 1: Identify Latency Source
|
|
|
|
```bash
|
|
# Check API Gateway latency
|
|
curl -w "@curl-format.txt" -o /dev/null -s http://localhost:8080/health
|
|
# Response time: 1250ms (HIGH)
|
|
|
|
# Check database query performance
|
|
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
|
|
psql -U foxhunt -d foxhunt -c "SELECT query, mean_exec_time, calls FROM pg_stat_statements ORDER BY mean_exec_time DESC LIMIT 10;"
|
|
|
|
# Check service-to-service latency
|
|
kubectl logs -n foxhunt -l app.kubernetes.io/component=trading-service | grep "request_duration_ms" | tail -100
|
|
```
|
|
|
|
### Step 2: Check Resource Utilization
|
|
|
|
```bash
|
|
# CPU usage
|
|
kubectl top pods -n foxhunt --sort-by=cpu | head -20
|
|
|
|
# Memory usage
|
|
kubectl top pods -n foxhunt --sort-by=memory | head -20
|
|
|
|
# Disk I/O (PostgreSQL)
|
|
kubectl exec -n foxhunt foxhunt-postgresql-0 -- iostat -x 1 5
|
|
```
|
|
|
|
### Step 3: Analyze Slow Queries
|
|
|
|
```bash
|
|
# Enable slow query logging (PostgreSQL)
|
|
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
|
|
psql -U foxhunt -d foxhunt -c "ALTER SYSTEM SET log_min_duration_statement = 100;" # Log queries >100ms
|
|
|
|
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
|
|
psql -U foxhunt -d foxhunt -c "SELECT pg_reload_conf();"
|
|
|
|
# View slow queries
|
|
kubectl logs -n foxhunt foxhunt-postgresql-0 | grep "duration:" | tail -50
|
|
```
|
|
|
|
---
|
|
|
|
## Common Causes
|
|
|
|
### Cause 1: Missing Database Index
|
|
|
|
**Symptom**: Queries take >1s, high CPU on database
|
|
|
|
**Diagnosis**:
|
|
```sql
|
|
-- Find sequential scans on large tables
|
|
SELECT schemaname, tablename, seq_scan, seq_tup_read, idx_scan
|
|
FROM pg_stat_user_tables
|
|
WHERE seq_scan > 0 AND seq_tup_read > 100000
|
|
ORDER BY seq_tup_read DESC
|
|
LIMIT 10;
|
|
```
|
|
|
|
**Resolution**:
|
|
```sql
|
|
-- Create missing index
|
|
CREATE INDEX CONCURRENTLY idx_orders_symbol_created_at ON orders(symbol, created_at DESC);
|
|
|
|
-- Verify index usage
|
|
EXPLAIN ANALYZE SELECT * FROM orders WHERE symbol='ES.FUT' ORDER BY created_at DESC LIMIT 100;
|
|
```
|
|
|
|
### Cause 2: Connection Pool Exhausted
|
|
|
|
**Symptom**: `timeout acquiring connection from pool`
|
|
|
|
**Diagnosis**:
|
|
```bash
|
|
kubectl logs -n foxhunt -l app.kubernetes.io/component=trading-service | grep "connection pool"
|
|
```
|
|
|
|
**Resolution**:
|
|
```bash
|
|
# Increase pool size via environment variable
|
|
kubectl set env deployment/foxhunt-trading-service -n foxhunt DATABASE_MAX_CONNECTIONS=50
|
|
|
|
# OR scale horizontally
|
|
kubectl scale deployment/foxhunt-trading-service -n foxhunt --replicas=10
|
|
```
|
|
|
|
### Cause 3: Network Latency
|
|
|
|
**Symptom**: High latency between services
|
|
|
|
**Diagnosis**:
|
|
```bash
|
|
# Test inter-service latency
|
|
kubectl run -it --rm debug --image=nicolaka/netshoot --restart=Never -- \
|
|
ping foxhunt-postgresql.foxhunt.svc.cluster.local
|
|
|
|
# Check for packet loss
|
|
kubectl run -it --rm debug --image=nicolaka/netshoot --restart=Never -- \
|
|
mtr --report --report-cycles 10 foxhunt-postgresql.foxhunt.svc.cluster.local
|
|
```
|
|
|
|
**Resolution**:
|
|
```bash
|
|
# Verify services are in same availability zone
|
|
kubectl get pods -n foxhunt -o wide
|
|
|
|
# Use node affinity to colocate services
|
|
```
|
|
|
|
---
|
|
|
|
**End of High Latency Troubleshooting Guide**
|