18944be360
All 12 validation agents complete: - Agent 1: E2E auth testing (11/11 tests pass, 8-layer validation) - Agent 2: Load testing framework ready (4 scenarios documented) - Agent 3: Docker deployment (6/6 infra services healthy) - Agent 4: Database integration (4 migrations, 6 NOTIFY channels, RBAC) - Agent 5: TLI client integration (JWT auth, OS keyring, API Gateway) - Agent 6: Performance profiling (978ns pipeline, 3 optimization recommendations) - Agent 7: Security penetration testing (OWASP Top 10, 3 critical findings) - Agent 8: gRPC proxy testing (3 proxies, 100% test pass, 5-8μs overhead) - Agent 9: Monitoring validation (Prometheus + Grafana, 5 issues identified) - Agent 10: Rate limiting stress test (8/8 tests pass, 99% attack mitigation) - Agent 11: Production readiness (7/9 criteria, 2 P0 blockers identified) - Agent 12: Documentation audit (92% complete, A- grade, production ready) Deliverables: - 30+ validation reports created (150+ KB documentation) - All 5 Dockerfiles updated with complete workspace - Redis/PostgreSQL integration tests operational - Comprehensive performance profiling completed - Security vulnerabilities documented with remediation 🔴 CRITICAL P0 BLOCKERS IDENTIFIED: 1. Audit trail persistence (trading_engine/src/compliance/audit_trails.rs:857) - Impact: SOX/MiFID II compliance violation - Status: Events not saved to database (only printed) 2. Test suite validation timeout - Historical: 1,919/1,919 tests passing (100%) - Current: Timeout after 2 minutes - Impact: Cannot certify regression-free state ⚠️ CRITICAL SECURITY VULNERABILITIES: 1. Authentication DISABLED (services/trading_service/src/main.rs:298-302) 2. Execution engine PANICS (execution_engine.rs:661,667,674) 3. Audit trail persistence (covered above) Production Decision: CONDITIONAL GO - Must fix 2 P0 blockers before production deployment - 7/9 production criteria met (78%) - SOX: 87.5% compliant, MiFID II: 87.5% compliant - Documentation: 92% complete (4,329 production lines) Next Wave: Address P0 blockers + performance optimization
162 lines
3.2 KiB
Markdown
162 lines
3.2 KiB
Markdown
# Load Testing Cheat Sheet
|
|
|
|
## Prerequisites Check
|
|
```bash
|
|
# Docker services running?
|
|
docker ps | grep -E "(redis|postgres)" | wc -l # Should be 2+
|
|
|
|
# API Gateway running?
|
|
curl -f http://localhost:50050/health && echo "✅ Ready" || echo "❌ Not running"
|
|
|
|
# Load tests built?
|
|
ls -lah ../../target/release/load_test_runner && echo "✅ Built" || echo "❌ Not built"
|
|
```
|
|
|
|
## Quick Commands
|
|
|
|
### Run Tests
|
|
```bash
|
|
# Normal Load (1K clients, 60s)
|
|
cargo run -r --bin load_test_runner -- normal
|
|
|
|
# Spike Load (0→10K clients)
|
|
cargo run -r --bin load_test_runner -- spike
|
|
|
|
# Stress Test (find breaking point)
|
|
cargo run -r --bin load_test_runner -- stress
|
|
|
|
# All tests
|
|
cargo run -r --bin load_test_runner -- all
|
|
```
|
|
|
|
### View Results
|
|
```bash
|
|
firefox *.html # Open all reports
|
|
```
|
|
|
|
### Extract Metrics
|
|
```bash
|
|
# P99 Latency
|
|
grep -oP 'P99 Latency.*?value">\K[^<]+' normal_load_report.html
|
|
|
|
# Throughput
|
|
grep -oP 'Requests/Second.*?value">\K[^<]+' normal_load_report.html
|
|
|
|
# Error Rate
|
|
grep -oP 'Error Rate.*?value">\K[^<]+' normal_load_report.html
|
|
```
|
|
|
|
## Performance Targets
|
|
|
|
| Metric | Target | Pass/Fail |
|
|
|--------|--------|-----------|
|
|
| P99 Latency | <10μs | ✅ / ❌ |
|
|
| Throughput | >100K req/s | ✅ / ❌ |
|
|
| Error Rate | <0.1% | ✅ / ❌ |
|
|
|
|
## Troubleshooting
|
|
|
|
### Connection Refused
|
|
```bash
|
|
# Start API Gateway
|
|
cargo run -r -p api_gateway
|
|
```
|
|
|
|
### High Latency
|
|
```bash
|
|
# Check CPU
|
|
top -p $(pgrep api_gateway)
|
|
|
|
# Check DB connections
|
|
psql -h localhost -p 5433 -U postgres -c 'SELECT count(*) FROM pg_stat_activity;'
|
|
```
|
|
|
|
### High Errors
|
|
```bash
|
|
# Check logs
|
|
docker logs foxhunt-api-gateway --tail 50
|
|
|
|
# Check backend health
|
|
curl http://localhost:50051/health # Trading
|
|
curl http://localhost:50052/health # Backtesting
|
|
curl http://localhost:50053/health # ML Training
|
|
```
|
|
|
|
## 5-Minute Validation
|
|
|
|
```bash
|
|
# 1. Start services
|
|
docker-compose up -d redis postgres
|
|
|
|
# 2. Start API Gateway
|
|
cargo run -r -p api_gateway &
|
|
|
|
# 3. Wait for startup
|
|
sleep 5
|
|
|
|
# 4. Run quick test
|
|
cd services/api_gateway/load_tests
|
|
cargo run -r --bin load_test_runner -- normal --num-clients 100 --duration-secs 30
|
|
|
|
# 5. Check results
|
|
firefox normal_load_report.html
|
|
```
|
|
|
|
## Expected Results
|
|
|
|
### Good Performance ✅
|
|
```
|
|
Requests/Second: >100,000
|
|
Error Rate: <0.1%
|
|
P99 Latency: <10μs
|
|
```
|
|
|
|
### Needs Work ⚠️
|
|
```
|
|
Requests/Second: 1,000-10,000
|
|
Error Rate: 0.5%-5%
|
|
P99 Latency: 10-100ms
|
|
```
|
|
|
|
## Files Generated
|
|
|
|
```
|
|
normal_load_report.html # Main dashboard
|
|
normal_load_report.rps.svg # RPS chart
|
|
normal_load_report.latency.svg # Latency chart
|
|
normal_load_report.errors.svg # Error chart
|
|
```
|
|
|
|
## Custom Parameters
|
|
|
|
```bash
|
|
# Custom normal load
|
|
cargo run -r --bin load_test_runner -- normal \
|
|
--num-clients 500 \
|
|
--duration-secs 120
|
|
|
|
# Custom spike load
|
|
cargo run -r --bin load_test_runner -- spike \
|
|
--target-clients 5000 \
|
|
--ramp-up-secs 15
|
|
|
|
# Custom stress test
|
|
cargo run -r --bin load_test_runner -- stress \
|
|
--initial-clients 200 \
|
|
--increment 200 \
|
|
--max-p99-latency-ms 100.0
|
|
```
|
|
|
|
## Monitoring Commands
|
|
|
|
```bash
|
|
# Real-time metrics
|
|
watch -n 1 "curl -s http://localhost:50050/metrics | grep requests_total"
|
|
|
|
# Docker stats
|
|
watch -n 1 "docker stats --no-stream foxhunt-api-gateway"
|
|
|
|
# System resources
|
|
watch -n 1 "ps aux | grep api_gateway | grep -v grep"
|
|
```
|