test: Add end-to-end smoke tests (Agent 99)
- Create comprehensive smoke test suite for post-deployment validation - Implement 4 test categories: infrastructure, service, authentication, order flow - Add graceful failure handling for unavailable services - Create automated test runner script with multiple modes (fast, verbose, category) - Document known blockers from Agent 96 (Backtesting/ML services) - Add 30+ individual smoke tests covering critical paths - Enable smoke-tests feature in tests/Cargo.toml - Create detailed README with usage and troubleshooting Test Categories: 1. Infrastructure Health: PostgreSQL, Redis, Vault, InfluxDB, Prometheus, Grafana 2. Service Health: Trading Service, API Gateway (+ blocked: Backtesting, ML) 3. Authentication Flow: JWT, sessions, revocation, rate limiting 4. Basic Order Flow: Order CRUD, positions, order history Features: - Configurable timeouts (5-10s per test) - Environment variable configuration - Graceful service unavailability handling - Parallel and sequential execution modes - Detailed pass/fail reporting Usage: ./run_smoke_tests.sh # Run all tests ./run_smoke_tests.sh --fast # Critical tests only ./run_smoke_tests.sh --verbose # Debug logging ./run_smoke_tests.sh --category infrastructure Blocked Tests (marked with #[ignore]): - Backtesting Service (config issues from Agent 96) - ML Training Service (config issues from Agent 96) Wave 125 Phase 3B - Deployment Excellence
This commit is contained in:
jgrusewski
3
Cargo.lock
generated
3
Cargo.lock
generated
@@ -9000,6 +9000,7 @@ dependencies = [
|
||||
"hdrhistogram",
|
||||
"influxdb2",
|
||||
"jemalloc_pprof",
|
||||
"jsonwebtoken",
|
||||
"lazy_static",
|
||||
"ml",
|
||||
"num 0.4.3",
|
||||
@@ -9010,6 +9011,7 @@ dependencies = [
|
||||
"rand 0.8.5",
|
||||
"rand_distr 0.4.3",
|
||||
"redis",
|
||||
"reqwest 0.12.23",
|
||||
"risk",
|
||||
"risk-data",
|
||||
"rstest 0.22.0",
|
||||
@@ -9027,6 +9029,7 @@ dependencies = [
|
||||
"tokio-test",
|
||||
"toml",
|
||||
"tonic",
|
||||
"tonic-health",
|
||||
"tracing",
|
||||
"tracing-subscriber",
|
||||
"trading_engine",
|
||||
|
||||
698
EMERGENCY_PROCEDURES.md
Normal file
698
EMERGENCY_PROCEDURES.md
Normal file
@@ -0,0 +1,698 @@
|
||||
# Emergency Procedures - Foxhunt HFT Trading System
|
||||
|
||||
**Purpose**: Critical incident response procedures for immediate action
|
||||
**Version**: 1.0.0
|
||||
**Last Updated**: 2025-10-07
|
||||
|
||||
---
|
||||
|
||||
## 🚨 EMERGENCY CONTACTS
|
||||
|
||||
**Fill in before production deployment:**
|
||||
|
||||
| Role | Name | Phone | Email | Escalation Time |
|
||||
|------|------|-------|-------|-----------------|
|
||||
| On-Call Engineer | [CONFIGURE] | [CONFIGURE] | [CONFIGURE] | Immediate |
|
||||
| Database Admin | [CONFIGURE] | [CONFIGURE] | [CONFIGURE] | 5 minutes |
|
||||
| Security Lead | [CONFIGURE] | [CONFIGURE] | [CONFIGURE] | 10 minutes |
|
||||
| CTO/Technical Lead | [CONFIGURE] | [CONFIGURE] | [CONFIGURE] | 15 minutes |
|
||||
| CEO | [CONFIGURE] | [CONFIGURE] | [CONFIGURE] | 30 minutes |
|
||||
|
||||
**PagerDuty**: [CONFIGURE]
|
||||
**Slack Channel**: #foxhunt-incidents
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Severity Levels
|
||||
|
||||
### SEV-1: CRITICAL (Immediate Response)
|
||||
- **Impact**: Trading completely halted, data loss, security breach
|
||||
- **Response Time**: <5 minutes
|
||||
- **Examples**: System down, database corruption, unauthorized access
|
||||
- **Escalation**: Immediate page to on-call + CTO
|
||||
|
||||
### SEV-2: HIGH (Urgent Response)
|
||||
- **Impact**: Significant degradation, trading impaired but operational
|
||||
- **Response Time**: <15 minutes
|
||||
- **Examples**: High latency (>500μs), service failure, data inconsistency
|
||||
- **Escalation**: Page on-call, notify team in Slack
|
||||
|
||||
### SEV-3: MEDIUM (Priority Response)
|
||||
- **Impact**: Minor degradation, non-critical component failure
|
||||
- **Response Time**: <1 hour
|
||||
- **Examples**: Non-critical service down, elevated error rate
|
||||
- **Escalation**: Slack notification, log ticket
|
||||
|
||||
### SEV-4: LOW (Standard Response)
|
||||
- **Impact**: No immediate impact, potential future issue
|
||||
- **Response Time**: <4 hours (business hours)
|
||||
- **Examples**: Warning alerts, resource usage trends
|
||||
- **Escalation**: Standard ticket
|
||||
|
||||
---
|
||||
|
||||
## 🔴 CRITICAL SCENARIOS (SEV-1)
|
||||
|
||||
### Scenario 1: System-Wide Trading Halt
|
||||
|
||||
**Trigger**: All trading stopped, no orders processing
|
||||
**Response Time**: <2 minutes to activate kill switch, <15 minutes to diagnose
|
||||
|
||||
#### Immediate Actions (First 60 seconds)
|
||||
|
||||
```bash
|
||||
# 1. CONFIRM HALT - Check if trading actually stopped
|
||||
curl http://localhost:9092/metrics | grep trading_active
|
||||
# Expected: trading_active 0
|
||||
|
||||
# 2. ACTIVATE KILL SWITCH (if not already active)
|
||||
export KILL_SWITCH_TOKEN="your-master-token"
|
||||
curl -X POST http://localhost:8080/emergency/kill \
|
||||
-H "X-Kill-Switch-Token: $KILL_SWITCH_TOKEN"
|
||||
|
||||
# 3. VERIFY KILL SWITCH ACTIVE
|
||||
curl http://localhost:8080/status
|
||||
# Expected: {"status": "HALTED", "reason": "Emergency kill switch activated"}
|
||||
|
||||
# 4. NOTIFY STAKEHOLDERS
|
||||
./deployment/scripts/send-emergency-alert.sh "Trading halted - investigating"
|
||||
```
|
||||
|
||||
#### Diagnosis (Next 5 minutes)
|
||||
|
||||
```bash
|
||||
# Check service status
|
||||
docker-compose ps # Docker
|
||||
sudo systemctl status foxhunt-* # Bare-metal
|
||||
|
||||
# Check recent errors
|
||||
docker-compose logs --tail=100 trading_service | grep -i error
|
||||
sudo journalctl -u foxhunt-trading --since "5 minutes ago" | grep -i error
|
||||
|
||||
# Check system resources
|
||||
df -h # Disk space
|
||||
free -m # Memory
|
||||
uptime # Load average
|
||||
|
||||
# Check database connectivity
|
||||
psql $DATABASE_URL -c "SELECT 1;" || echo "❌ Database connection failed"
|
||||
|
||||
# Check network connectivity
|
||||
ping -c 3 <exchange-server> || echo "❌ Network connectivity failed"
|
||||
```
|
||||
|
||||
#### Decision Tree
|
||||
|
||||
**If database is down**:
|
||||
```bash
|
||||
# See: Scenario 2 - Database Failure
|
||||
```
|
||||
|
||||
**If service crashed**:
|
||||
```bash
|
||||
# See: Scenario 3 - Service Crash
|
||||
```
|
||||
|
||||
**If network issue**:
|
||||
```bash
|
||||
# See: Scenario 5 - Network Failure
|
||||
```
|
||||
|
||||
**If cause unknown**:
|
||||
```bash
|
||||
# 1. Restart services in safe mode (read-only)
|
||||
./deployment/scripts/restart-safe-mode.sh
|
||||
|
||||
# 2. Escalate to CTO
|
||||
./deployment/scripts/escalate.sh --level=cto --message="Trading halt, cause unknown"
|
||||
|
||||
# 3. Begin detailed investigation
|
||||
./deployment/scripts/capture-diagnostics.sh > /var/log/foxhunt/incident-$(date +%s).log
|
||||
```
|
||||
|
||||
#### Recovery Procedure
|
||||
|
||||
```bash
|
||||
# 1. Fix root cause (see specific scenarios)
|
||||
|
||||
# 2. Run validation tests
|
||||
./deployment/scripts/pre-trading-validation.sh
|
||||
|
||||
# 3. Resume trading (only after validation passes)
|
||||
curl -X POST http://localhost:8080/trading/resume \
|
||||
-H "X-Kill-Switch-Token: $KILL_SWITCH_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"reason": "Issue resolved - verified", "approved_by": "your-name"}'
|
||||
|
||||
# 4. Monitor closely for 30 minutes
|
||||
watch -n 10 'curl -s http://localhost:9092/metrics | grep -E "(order_latency|error_rate)"'
|
||||
|
||||
# 5. Send all-clear notification
|
||||
./deployment/scripts/send-emergency-alert.sh "Trading resumed - incident resolved"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Scenario 2: Database Failure
|
||||
|
||||
**Trigger**: Cannot connect to PostgreSQL
|
||||
**Response Time**: <5 minutes to failover
|
||||
|
||||
#### Immediate Actions
|
||||
|
||||
```bash
|
||||
# 1. ACTIVATE KILL SWITCH
|
||||
curl -X POST http://localhost:8080/emergency/kill \
|
||||
-H "X-Kill-Switch-Token: $KILL_SWITCH_TOKEN"
|
||||
|
||||
# 2. CHECK DATABASE STATUS
|
||||
sudo systemctl status postgresql # Bare-metal
|
||||
docker-compose ps postgres # Docker
|
||||
|
||||
# 3. CHECK REPLICATION STATUS (if configured)
|
||||
psql -h replica-server -U foxhunt -c "SELECT pg_is_in_recovery();"
|
||||
# Expected: t (true = replica)
|
||||
```
|
||||
|
||||
#### Recovery Options
|
||||
|
||||
**Option A: Database is Running but Unresponsive**
|
||||
```bash
|
||||
# 1. Check connections
|
||||
psql $DATABASE_URL -c "SELECT count(*) FROM pg_stat_activity;"
|
||||
|
||||
# 2. Kill long-running queries
|
||||
psql $DATABASE_URL -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state = 'active' AND now() - query_start > interval '5 minutes';"
|
||||
|
||||
# 3. Restart PostgreSQL
|
||||
sudo systemctl restart postgresql # Bare-metal
|
||||
docker-compose restart postgres # Docker
|
||||
|
||||
# 4. Wait for startup
|
||||
sleep 10
|
||||
|
||||
# 5. Test connection
|
||||
psql $DATABASE_URL -c "SELECT 1;"
|
||||
```
|
||||
|
||||
**Option B: Database Crashed - Restore from Replica**
|
||||
```bash
|
||||
# 1. Promote replica to primary
|
||||
ssh replica-server "sudo -u postgres /usr/lib/postgresql/16/bin/pg_ctl promote -D /var/lib/postgresql/16/main"
|
||||
|
||||
# 2. Update connection strings
|
||||
export NEW_DB_URL=postgresql://foxhunt:password@replica-server:5432/foxhunt
|
||||
|
||||
# 3. Update services
|
||||
docker-compose down
|
||||
sed -i "s|DATABASE_URL=.*|DATABASE_URL=$NEW_DB_URL|" .env
|
||||
docker-compose up -d
|
||||
|
||||
# OR for bare-metal
|
||||
sudo systemctl stop foxhunt-*
|
||||
sudo systemctl set-environment DATABASE_URL="$NEW_DB_URL"
|
||||
sudo systemctl start foxhunt-*
|
||||
|
||||
# 4. Verify connectivity
|
||||
psql $NEW_DB_URL -c "SELECT count(*) FROM orders;"
|
||||
```
|
||||
|
||||
**Option C: Complete Restore from Backup**
|
||||
```bash
|
||||
# 1. Download latest backup from S3
|
||||
aws s3 cp s3://foxhunt-backups/postgres/latest.dump /tmp/restore.dump
|
||||
|
||||
# 2. Create new database
|
||||
createdb foxhunt_restored
|
||||
|
||||
# 3. Restore
|
||||
pg_restore -h localhost -U foxhunt -d foxhunt_restored -v /tmp/restore.dump
|
||||
|
||||
# 4. Swap databases
|
||||
psql -U postgres <<EOF
|
||||
ALTER DATABASE foxhunt RENAME TO foxhunt_old;
|
||||
ALTER DATABASE foxhunt_restored RENAME TO foxhunt;
|
||||
EOF
|
||||
|
||||
# 5. Restart services
|
||||
sudo systemctl restart foxhunt-*
|
||||
|
||||
# Estimated downtime: 5-15 minutes (depends on backup size)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Scenario 3: Service Crash/Unresponsive
|
||||
|
||||
**Trigger**: Service health check failing, not responding
|
||||
**Response Time**: <1 minute to restart
|
||||
|
||||
#### Immediate Actions
|
||||
|
||||
```bash
|
||||
# 1. IDENTIFY CRASHED SERVICE
|
||||
docker-compose ps | grep -v "Up" # Docker
|
||||
sudo systemctl status foxhunt-* | grep failed # Bare-metal
|
||||
|
||||
# 2. CHECK LOGS FOR CRASH REASON
|
||||
docker-compose logs --tail=100 <service> | grep -E "(panic|error|fatal)"
|
||||
sudo journalctl -u foxhunt-<service> -n 100 | grep -E "(panic|error|fatal)"
|
||||
|
||||
# 3. RESTART SERVICE
|
||||
docker-compose restart <service> # Docker
|
||||
sudo systemctl restart foxhunt-<service> # Bare-metal
|
||||
|
||||
# 4. VERIFY RESTART SUCCESSFUL
|
||||
docker-compose ps <service>
|
||||
sudo systemctl status foxhunt-<service>
|
||||
|
||||
# 5. CHECK FOR ERRORS AFTER RESTART
|
||||
docker-compose logs --tail=50 <service>
|
||||
sudo journalctl -u foxhunt-<service> -n 50
|
||||
```
|
||||
|
||||
#### If Restart Fails
|
||||
|
||||
```bash
|
||||
# 1. Check for port conflicts
|
||||
sudo lsof -i :50052 # Trading service port
|
||||
sudo kill -9 <PID> # Kill conflicting process
|
||||
|
||||
# 2. Check disk space
|
||||
df -h
|
||||
# If >90% full, clean up logs
|
||||
sudo find /var/log/foxhunt -name "*.log" -mtime +7 -delete
|
||||
|
||||
# 3. Check memory
|
||||
free -m
|
||||
# If memory exhausted, identify memory hog
|
||||
ps aux --sort=-%mem | head -10
|
||||
|
||||
# 4. Try safe mode restart
|
||||
docker-compose stop <service>
|
||||
docker-compose up -d <service> --force-recreate
|
||||
|
||||
# OR bare-metal
|
||||
sudo systemctl stop foxhunt-<service>
|
||||
sudo rm -rf /tmp/foxhunt/* # Clear temp files
|
||||
sudo systemctl start foxhunt-<service>
|
||||
|
||||
# 5. If still failing, rollback
|
||||
sudo cp /opt/foxhunt/bin/<service>.backup /opt/foxhunt/bin/<service>
|
||||
sudo systemctl restart foxhunt-<service>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Scenario 4: Security Breach Detected
|
||||
|
||||
**Trigger**: Unauthorized access, data exfiltration, suspicious activity
|
||||
**Response Time**: IMMEDIATE - <30 seconds to isolate
|
||||
|
||||
#### Immediate Actions (CRITICAL - Do NOT delay)
|
||||
|
||||
```bash
|
||||
# 1. ACTIVATE KILL SWITCH IMMEDIATELY
|
||||
curl -X POST http://localhost:8080/emergency/kill \
|
||||
-H "X-Kill-Switch-Token: $KILL_SWITCH_TOKEN"
|
||||
|
||||
# 2. ISOLATE SYSTEM (disconnect from network)
|
||||
# Bare-metal
|
||||
sudo iptables -P INPUT DROP
|
||||
sudo iptables -P OUTPUT DROP
|
||||
sudo iptables -P FORWARD DROP
|
||||
|
||||
# Docker
|
||||
docker network disconnect foxhunt_foxhunt-network <container>
|
||||
|
||||
# 3. CAPTURE FORENSIC DATA
|
||||
./deployment/scripts/capture-forensics.sh > /var/log/foxhunt/security-incident-$(date +%s).log
|
||||
|
||||
# 4. NOTIFY SECURITY TEAM IMMEDIATELY
|
||||
./deployment/scripts/security-alert.sh --severity=critical --message="Security breach detected - system isolated"
|
||||
|
||||
# 5. PRESERVE EVIDENCE
|
||||
sudo cp -r /var/log/foxhunt /var/log/foxhunt.incident-$(date +%s)
|
||||
sudo tar -czf /tmp/incident-evidence-$(date +%s).tar.gz /var/log/foxhunt /opt/foxhunt/config
|
||||
```
|
||||
|
||||
#### Investigation (Security Team)
|
||||
|
||||
```bash
|
||||
# 1. Review audit logs
|
||||
grep "$(date +%Y-%m-%d)" /var/log/foxhunt/audit.log | grep -E "(unauthorized|failed|suspicious)"
|
||||
|
||||
# 2. Check for unauthorized access
|
||||
psql $DATABASE_URL -c "SELECT * FROM audit_logs WHERE action = 'login_failed' AND created_at > NOW() - INTERVAL '24 hours' ORDER BY created_at DESC;"
|
||||
|
||||
# 3. Review network connections
|
||||
sudo netstat -anp | grep ESTABLISHED
|
||||
|
||||
# 4. Check for malware
|
||||
sudo rkhunter --check
|
||||
sudo chkrootkit
|
||||
|
||||
# 5. Check file integrity
|
||||
sudo aide --check
|
||||
|
||||
# 6. Review user access
|
||||
sudo lastlog
|
||||
sudo last -n 50
|
||||
```
|
||||
|
||||
#### Recovery (After Investigation)
|
||||
|
||||
```bash
|
||||
# 1. Rotate ALL secrets
|
||||
vault kv put secret/foxhunt/jwt secret="$(openssl rand -base64 64)"
|
||||
vault kv put secret/foxhunt/kill_switch master_token="$(openssl rand -base64 32)"
|
||||
# ... rotate all other secrets
|
||||
|
||||
# 2. Update firewall rules
|
||||
sudo ufw default deny incoming
|
||||
sudo ufw allow from <trusted-ip> to any port 50051
|
||||
|
||||
# 3. Apply security patches
|
||||
sudo apt-get update
|
||||
sudo apt-get upgrade -y
|
||||
|
||||
# 4. Restore network access (carefully)
|
||||
sudo iptables -F
|
||||
sudo systemctl restart foxhunt-*
|
||||
|
||||
# 5. Monitor closely for 48 hours
|
||||
watch -n 60 './deployment/scripts/security-scan.sh'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Scenario 5: Network Failure/Partition
|
||||
|
||||
**Trigger**: Cannot reach exchange servers, inter-service communication failed
|
||||
**Response Time**: <10 seconds to halt trading
|
||||
|
||||
#### Immediate Actions
|
||||
|
||||
```bash
|
||||
# 1. HALT TRADING (network issues = blind trading)
|
||||
curl -X POST http://localhost:8080/emergency/kill \
|
||||
-H "X-Kill-Switch-Token: $KILL_SWITCH_TOKEN"
|
||||
|
||||
# 2. VERIFY NETWORK CONNECTIVITY
|
||||
# Check external connectivity
|
||||
ping -c 3 8.8.8.8 # Internet
|
||||
ping -c 3 <exchange-server> # Exchange
|
||||
|
||||
# Check inter-service connectivity
|
||||
for service in trading backtesting ml_training; do
|
||||
curl -s http://$service:9092/metrics || echo "❌ Cannot reach $service"
|
||||
done
|
||||
|
||||
# 3. CHECK NETWORK INTERFACE STATUS
|
||||
ip link show
|
||||
ip addr show
|
||||
|
||||
# 4. CHECK FIREWALL RULES
|
||||
sudo iptables -L -n -v
|
||||
```
|
||||
|
||||
#### Recovery Options
|
||||
|
||||
**Option A: Network Restored Automatically**
|
||||
```bash
|
||||
# Services will auto-recover when network returns
|
||||
# Verify connectivity restored
|
||||
ping -c 10 <exchange-server>
|
||||
|
||||
# Services should exit read-only mode automatically
|
||||
curl http://localhost:9092/metrics | grep network_partition_active
|
||||
# Expected: network_partition_active 0
|
||||
|
||||
# Resume trading after verification
|
||||
curl -X POST http://localhost:8080/trading/resume \
|
||||
-H "X-Kill-Switch-Token: $KILL_SWITCH_TOKEN"
|
||||
```
|
||||
|
||||
**Option B: Manual Network Recovery**
|
||||
```bash
|
||||
# 1. Restart network interface
|
||||
sudo ifdown eth0
|
||||
sudo ifup eth0
|
||||
|
||||
# 2. Restart network service
|
||||
sudo systemctl restart networking
|
||||
|
||||
# 3. Verify connectivity
|
||||
ping -c 5 <exchange-server>
|
||||
|
||||
# 4. Restart services
|
||||
sudo systemctl restart foxhunt-*
|
||||
|
||||
# 5. Resume trading
|
||||
curl -X POST http://localhost:8080/trading/resume \
|
||||
-H "X-Kill-Switch-Token: $KILL_SWITCH_TOKEN"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Scenario 6: High Latency Alert (>100μs p99)
|
||||
|
||||
**Trigger**: Order processing latency exceeds critical threshold
|
||||
**Response Time**: <5 minutes to investigate
|
||||
|
||||
#### Immediate Actions
|
||||
|
||||
```bash
|
||||
# 1. VERIFY LATENCY SPIKE
|
||||
curl http://localhost:9092/metrics | grep order_processing_duration_seconds
|
||||
|
||||
# 2. CHECK CPU FREQUENCY SCALING (should be "performance")
|
||||
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
|
||||
# 3. CHECK SYSTEM LOAD
|
||||
uptime
|
||||
top -bn1 | head -20
|
||||
|
||||
# 4. CHECK DATABASE QUERY PERFORMANCE
|
||||
psql $DATABASE_URL -c "SELECT query, mean_exec_time, calls FROM pg_stat_statements ORDER BY mean_exec_time DESC LIMIT 10;"
|
||||
|
||||
# 5. CHECK NETWORK LATENCY
|
||||
ping -c 10 <exchange-server>
|
||||
```
|
||||
|
||||
#### Recovery Actions
|
||||
|
||||
**Fix CPU Frequency Scaling**:
|
||||
```bash
|
||||
sudo cpupower frequency-set -g performance
|
||||
```
|
||||
|
||||
**Fix Database Performance**:
|
||||
```bash
|
||||
# Add missing indexes
|
||||
psql $DATABASE_URL <<EOF
|
||||
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_orders_created_at ON orders(created_at);
|
||||
ANALYZE orders;
|
||||
EOF
|
||||
```
|
||||
|
||||
**Scale Horizontally** (if load too high):
|
||||
```bash
|
||||
# Kubernetes
|
||||
kubectl scale deployment trading-service --replicas=5
|
||||
|
||||
# Docker Swarm
|
||||
docker service scale foxhunt_trading_service=5
|
||||
```
|
||||
|
||||
**Restart Services** (if degraded):
|
||||
```bash
|
||||
sudo systemctl restart foxhunt-trading
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🟡 HIGH PRIORITY SCENARIOS (SEV-2)
|
||||
|
||||
### Scenario 7: Disk Space Critical (>90%)
|
||||
|
||||
```bash
|
||||
# 1. CHECK DISK USAGE
|
||||
df -h
|
||||
|
||||
# 2. FIND LARGE FILES
|
||||
sudo du -ah /var/log/foxhunt | sort -rh | head -20
|
||||
sudo du -ah /opt/foxhunt | sort -rh | head -20
|
||||
|
||||
# 3. CLEAN UP LOGS (keep last 7 days)
|
||||
sudo find /var/log/foxhunt -name "*.log" -mtime +7 -delete
|
||||
sudo journalctl --vacuum-time=7d
|
||||
|
||||
# 4. CLEAN UP DOCKER (if using Docker)
|
||||
docker system prune -af --volumes
|
||||
|
||||
# 5. EXPAND DISK (if possible)
|
||||
sudo lvextend -L +100G /dev/vg0/root
|
||||
sudo resize2fs /dev/vg0/root
|
||||
```
|
||||
|
||||
### Scenario 8: Memory Exhaustion (>90%)
|
||||
|
||||
```bash
|
||||
# 1. IDENTIFY MEMORY HOGS
|
||||
ps aux --sort=-%mem | head -10
|
||||
|
||||
# 2. RESTART SERVICES (frees memory)
|
||||
sudo systemctl restart foxhunt-trading
|
||||
|
||||
# 3. INCREASE SWAP (temporary fix)
|
||||
sudo fallocate -l 8G /swapfile
|
||||
sudo chmod 600 /swapfile
|
||||
sudo mkswap /swapfile
|
||||
sudo swapon /swapfile
|
||||
|
||||
# 4. INCREASE SERVICE MEMORY LIMITS
|
||||
sudo systemctl edit foxhunt-trading
|
||||
# Add: MemoryMax=4G
|
||||
sudo systemctl restart foxhunt-trading
|
||||
|
||||
# 5. SCALE HORIZONTALLY (long-term fix)
|
||||
kubectl scale deployment trading-service --replicas=5
|
||||
```
|
||||
|
||||
### Scenario 9: Redis Connection Failure
|
||||
|
||||
```bash
|
||||
# 1. CHECK REDIS STATUS
|
||||
redis-cli ping
|
||||
sudo systemctl status redis
|
||||
|
||||
# 2. RESTART REDIS
|
||||
sudo systemctl restart redis # Bare-metal
|
||||
docker-compose restart redis # Docker
|
||||
|
||||
# 3. VERIFY CONNECTION FROM SERVICES
|
||||
redis-cli -h redis ping
|
||||
|
||||
# 4. CHECK CONNECTION POOL
|
||||
redis-cli info clients
|
||||
|
||||
# 5. RESTART SERVICES (reconnect to Redis)
|
||||
sudo systemctl restart foxhunt-*
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📞 Escalation Procedures
|
||||
|
||||
### Escalation Matrix
|
||||
|
||||
| Time Elapsed | Action |
|
||||
|--------------|--------|
|
||||
| T+0 | On-call engineer paged |
|
||||
| T+5 min | If no response, escalate to backup on-call |
|
||||
| T+10 min | Notify CTO/Technical Lead |
|
||||
| T+15 min | If SEV-1, notify CEO |
|
||||
| T+30 min | If unresolved, engage external support |
|
||||
|
||||
### Communication Protocol
|
||||
|
||||
**Internal (Slack #foxhunt-incidents)**:
|
||||
```
|
||||
🚨 INCIDENT ALERT
|
||||
Severity: SEV-X
|
||||
Component: <service/system>
|
||||
Impact: <trading halted/degraded/etc>
|
||||
Detected: <timestamp>
|
||||
On-Call: <name> responding
|
||||
ETA: <estimate>
|
||||
```
|
||||
|
||||
**External (Stakeholders)**:
|
||||
```
|
||||
Subject: [INCIDENT] Foxhunt Trading System - <brief description>
|
||||
|
||||
Time: <timestamp>
|
||||
Status: Investigating/Identified/Resolved
|
||||
Impact: <clear impact description>
|
||||
Actions: <what we're doing>
|
||||
ETA: <when we expect resolution>
|
||||
Contact: <on-call contact>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📊 Post-Incident Procedures
|
||||
|
||||
### Immediate (Within 1 hour of resolution)
|
||||
|
||||
```bash
|
||||
# 1. Document incident
|
||||
cat > /var/log/foxhunt/incidents/incident-$(date +%Y%m%d-%H%M%S).md <<EOF
|
||||
# Incident Report
|
||||
|
||||
**Date**: $(date)
|
||||
**Severity**: SEV-X
|
||||
**Duration**: X minutes
|
||||
**Impact**: <description>
|
||||
**Root Cause**: <cause>
|
||||
**Resolution**: <what fixed it>
|
||||
**Responders**: <names>
|
||||
EOF
|
||||
|
||||
# 2. Send all-clear notification
|
||||
./deployment/scripts/send-alert.sh "Incident resolved - system operational"
|
||||
|
||||
# 3. Monitor for recurrence
|
||||
watch -n 60 './deployment/health_check.sh --mode comprehensive'
|
||||
```
|
||||
|
||||
### Post-Mortem (Within 48 hours)
|
||||
|
||||
1. **Incident Timeline**: Document minute-by-minute actions
|
||||
2. **Root Cause Analysis**: 5 Whys methodology
|
||||
3. **Action Items**: Prevent recurrence
|
||||
4. **Communication Review**: How well did we communicate?
|
||||
5. **Documentation Updates**: Update runbooks with lessons learned
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Emergency Tools
|
||||
|
||||
### Quick Diagnostic Script
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# /opt/foxhunt/bin/emergency-diagnostics.sh
|
||||
|
||||
echo "=== FOXHUNT EMERGENCY DIAGNOSTICS ==="
|
||||
echo "Time: $(date)"
|
||||
echo ""
|
||||
|
||||
echo "=== SERVICE STATUS ==="
|
||||
systemctl status foxhunt-* --no-pager | grep -E "(Active|Main PID)"
|
||||
|
||||
echo ""
|
||||
echo "=== SYSTEM RESOURCES ==="
|
||||
echo "Load: $(uptime | awk -F'load average:' '{print $2}')"
|
||||
echo "Memory: $(free -m | grep Mem | awk '{printf "%.1f%%", $3/$2*100}')"
|
||||
echo "Disk: $(df -h / | tail -1 | awk '{print $5}')"
|
||||
|
||||
echo ""
|
||||
echo "=== DATABASE ==="
|
||||
psql $DATABASE_URL -c "SELECT 1;" && echo "✅ Connected" || echo "❌ Failed"
|
||||
|
||||
echo ""
|
||||
echo "=== RECENT ERRORS ==="
|
||||
journalctl -u foxhunt-* --since "5 minutes ago" | grep -i error | tail -10
|
||||
|
||||
echo ""
|
||||
echo "=== NETWORK ==="
|
||||
ping -c 3 8.8.8.8 && echo "✅ Internet OK" || echo "❌ Internet Failed"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-10-07
|
||||
**Version**: 1.0.0
|
||||
**Review Frequency**: Monthly or after each incident
|
||||
1283
LOAD_BALANCING_SCALING.md
Normal file
1283
LOAD_BALANCING_SCALING.md
Normal file
File diff suppressed because it is too large
Load Diff
2311
PRODUCTION_DEPLOYMENT_RUNBOOK.md
Normal file
2311
PRODUCTION_DEPLOYMENT_RUNBOOK.md
Normal file
File diff suppressed because it is too large
Load Diff
408
QUICK_START_PRODUCTION.md
Normal file
408
QUICK_START_PRODUCTION.md
Normal file
@@ -0,0 +1,408 @@
|
||||
# Quick Start Production Deployment - Foxhunt HFT
|
||||
|
||||
**Time Required**: 15-20 minutes (Docker), 30-45 minutes (Bare-Metal)
|
||||
**Prerequisites**: All infrastructure ready (PostgreSQL, Redis, Vault)
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Docker Deployment (Recommended for Testing/Staging)
|
||||
|
||||
### Step 1: Fix Agent 96 Issues (REQUIRED - 5 minutes)
|
||||
|
||||
```bash
|
||||
cd /opt/foxhunt
|
||||
|
||||
# Fix 1: Update Dockerfile paths (all 4 services)
|
||||
find services -name Dockerfile -exec sed -i 's|COPY crates/config|COPY config|g' {} \;
|
||||
|
||||
# Fix 2: Add ML Training Service entry point
|
||||
echo 'CMD ["ml_training_service", "serve"]' >> services/ml_training_service/Dockerfile
|
||||
|
||||
# Fix 3: Set Benzinga API key
|
||||
export BENZINGA_API_KEY="your-benzinga-pro-api-key"
|
||||
echo "BENZINGA_API_KEY=$BENZINGA_API_KEY" >> .env
|
||||
```
|
||||
|
||||
### Step 2: Configure Environment (2 minutes)
|
||||
|
||||
```bash
|
||||
# Generate required secrets
|
||||
export JWT_SECRET=$(openssl rand -base64 64 | tr -d '\n')
|
||||
export KILL_SWITCH_TOKEN=$(openssl rand -base64 32)
|
||||
|
||||
# Create .env file
|
||||
cat > .env <<EOF
|
||||
DATABASE_URL=postgresql://foxhunt:foxhunt_dev_password@postgres:5432/foxhunt
|
||||
REDIS_URL=redis://redis:6379
|
||||
VAULT_ADDR=http://vault:8200
|
||||
VAULT_TOKEN=foxhunt-dev-root
|
||||
JWT_SECRET=$JWT_SECRET
|
||||
BENZINGA_API_KEY=$BENZINGA_API_KEY
|
||||
KILL_SWITCH_MASTER_TOKEN=$KILL_SWITCH_TOKEN
|
||||
AWS_ACCESS_KEY_ID=your_aws_key
|
||||
AWS_SECRET_ACCESS_KEY=your_aws_secret
|
||||
AWS_REGION=us-east-1
|
||||
RUST_LOG=info
|
||||
EOF
|
||||
```
|
||||
|
||||
### Step 3: Build Images (5-10 minutes)
|
||||
|
||||
```bash
|
||||
# Build all service images
|
||||
docker-compose build
|
||||
|
||||
# Or build individually (parallel)
|
||||
docker-compose build trading_service &
|
||||
docker-compose build backtesting_service &
|
||||
docker-compose build ml_training_service &
|
||||
docker-compose build api_gateway &
|
||||
wait
|
||||
|
||||
# Verify images
|
||||
docker images | grep foxhunt
|
||||
```
|
||||
|
||||
### Step 4: Start Infrastructure (2 minutes)
|
||||
|
||||
```bash
|
||||
# Start database and cache layers
|
||||
docker-compose up -d postgres redis vault influxdb
|
||||
|
||||
# Wait for health checks
|
||||
sleep 30
|
||||
docker-compose ps | grep healthy
|
||||
```
|
||||
|
||||
### Step 5: Run Migrations (1 minute)
|
||||
|
||||
```bash
|
||||
# Apply database migrations
|
||||
cargo sqlx migrate run
|
||||
|
||||
# Or from Docker
|
||||
docker-compose exec postgres psql -U foxhunt -d foxhunt -c '\dt'
|
||||
```
|
||||
|
||||
### Step 6: Start Services (2 minutes)
|
||||
|
||||
```bash
|
||||
# Start monitoring
|
||||
docker-compose up -d prometheus grafana
|
||||
|
||||
# Start backend services
|
||||
docker-compose up -d trading_service backtesting_service ml_training_service
|
||||
|
||||
# Wait for startup (ML service takes 60s for GPU init)
|
||||
sleep 60
|
||||
|
||||
# Start API Gateway
|
||||
docker-compose up -d api_gateway
|
||||
```
|
||||
|
||||
### Step 7: Verify Deployment (2 minutes)
|
||||
|
||||
```bash
|
||||
# Check all services healthy
|
||||
docker-compose ps
|
||||
|
||||
# Test health endpoints
|
||||
curl -s http://localhost:9092/metrics | grep foxhunt_orders_total
|
||||
|
||||
# View logs
|
||||
docker-compose logs --tail=50 trading_service
|
||||
docker-compose logs --tail=50 api_gateway
|
||||
|
||||
# Access Grafana
|
||||
open http://localhost:3000
|
||||
# Login: admin / foxhunt123
|
||||
```
|
||||
|
||||
### Success Checklist
|
||||
|
||||
- [ ] All 10 containers running (6 infrastructure + 4 application)
|
||||
- [ ] No errors in `docker-compose logs`
|
||||
- [ ] Metrics endpoints responding (ports 9091-9094)
|
||||
- [ ] Grafana accessible at http://localhost:3000
|
||||
- [ ] GPU detected: `docker-compose exec ml_training_service nvidia-smi`
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Bare-Metal Deployment (Recommended for Production)
|
||||
|
||||
### Step 1: Build Binaries (10-15 minutes)
|
||||
|
||||
```bash
|
||||
cd /opt/foxhunt
|
||||
|
||||
# Build with CPU-specific optimizations
|
||||
export RUSTFLAGS="-C target-cpu=native -C opt-level=3"
|
||||
cargo build --release --workspace
|
||||
|
||||
# Verify
|
||||
ls -lh target/release/ | grep -E "(trading|backtesting|ml_training|api_gateway)"
|
||||
```
|
||||
|
||||
### Step 2: Install Binaries (2 minutes)
|
||||
|
||||
```bash
|
||||
# Create directories
|
||||
sudo mkdir -p /opt/foxhunt/{bin,config,logs,data,models,checkpoints}
|
||||
|
||||
# Install binaries
|
||||
sudo cp target/release/{trading_service,backtesting_service,ml_training_service,api_gateway} /opt/foxhunt/bin/
|
||||
|
||||
# Set permissions
|
||||
sudo chown -R foxhunt:foxhunt /opt/foxhunt
|
||||
sudo chmod +x /opt/foxhunt/bin/*
|
||||
```
|
||||
|
||||
### Step 3: Configure SystemD (5 minutes)
|
||||
|
||||
```bash
|
||||
# Generate SystemD services
|
||||
./deployment/create_systemd_services.sh
|
||||
|
||||
# Reload
|
||||
sudo systemctl daemon-reload
|
||||
|
||||
# Enable auto-start
|
||||
sudo systemctl enable foxhunt-{trading,backtesting,ml-training,api-gateway}
|
||||
```
|
||||
|
||||
### Step 4: Setup Database (3 minutes)
|
||||
|
||||
```bash
|
||||
# Create database
|
||||
sudo -u postgres psql <<EOF
|
||||
CREATE USER foxhunt WITH PASSWORD 'secure_password';
|
||||
CREATE DATABASE foxhunt OWNER foxhunt;
|
||||
\c foxhunt
|
||||
CREATE EXTENSION IF NOT EXISTS timescaledb;
|
||||
EOF
|
||||
|
||||
# Run migrations
|
||||
export DATABASE_URL=postgresql://foxhunt:secure_password@localhost:5432/foxhunt
|
||||
cargo sqlx migrate run
|
||||
```
|
||||
|
||||
### Step 5: Configure Secrets (3 minutes)
|
||||
|
||||
```bash
|
||||
# Start Vault
|
||||
sudo systemctl start vault
|
||||
|
||||
# Store secrets
|
||||
export VAULT_ADDR=http://localhost:8200
|
||||
export VAULT_TOKEN=foxhunt-dev-root
|
||||
|
||||
vault kv put secret/foxhunt/jwt secret="$(openssl rand -base64 64)"
|
||||
vault kv put secret/foxhunt/kill_switch master_token="$(openssl rand -base64 32)"
|
||||
vault kv put secret/foxhunt/api_keys benzinga="$BENZINGA_API_KEY"
|
||||
vault kv put secret/foxhunt/aws access_key_id="$AWS_ACCESS_KEY_ID" secret_access_key="$AWS_SECRET_ACCESS_KEY"
|
||||
```
|
||||
|
||||
### Step 6: Start Services (2 minutes)
|
||||
|
||||
```bash
|
||||
# Start infrastructure
|
||||
sudo systemctl start redis vault
|
||||
|
||||
# Start backend services (parallel)
|
||||
sudo systemctl start foxhunt-trading &
|
||||
sudo systemctl start foxhunt-backtesting &
|
||||
sudo systemctl start foxhunt-ml-training &
|
||||
wait
|
||||
|
||||
# Wait for health checks
|
||||
sleep 30
|
||||
|
||||
# Start API Gateway
|
||||
sudo systemctl start foxhunt-api-gateway
|
||||
```
|
||||
|
||||
### Step 7: Verify Deployment (2 minutes)
|
||||
|
||||
```bash
|
||||
# Check service status
|
||||
sudo systemctl status foxhunt-*
|
||||
|
||||
# Test health
|
||||
./deployment/health_check.sh --mode comprehensive
|
||||
|
||||
# View logs
|
||||
sudo journalctl -u foxhunt-trading -n 50
|
||||
sudo journalctl -u foxhunt-api-gateway -n 50
|
||||
|
||||
# Check metrics
|
||||
curl http://localhost:9092/metrics | grep foxhunt_
|
||||
```
|
||||
|
||||
### Success Checklist
|
||||
|
||||
- [ ] All 4 services active: `systemctl status foxhunt-*`
|
||||
- [ ] No errors in logs: `journalctl -u foxhunt-* --since "10 minutes ago"`
|
||||
- [ ] Metrics endpoints responding
|
||||
- [ ] GPU detected: `nvidia-smi`
|
||||
- [ ] Database connected: `psql $DATABASE_URL -c "SELECT 1;"`
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Post-Deployment Smoke Tests
|
||||
|
||||
### Test 1: Service Health
|
||||
|
||||
```bash
|
||||
# Docker
|
||||
for service in trading backtesting ml_training api_gateway; do
|
||||
echo "Testing $service..."
|
||||
docker-compose logs --tail=1 ${service}_service | grep -q "started" && echo "✅ $service OK" || echo "❌ $service FAILED"
|
||||
done
|
||||
|
||||
# Bare-metal
|
||||
for service in trading backtesting ml-training api-gateway; do
|
||||
sudo systemctl is-active foxhunt-$service && echo "✅ $service OK" || echo "❌ $service FAILED"
|
||||
done
|
||||
```
|
||||
|
||||
### Test 2: Metrics Collection
|
||||
|
||||
```bash
|
||||
# Test each metrics endpoint
|
||||
for port in 9091 9092 9093 9094; do
|
||||
curl -s http://localhost:$port/metrics | grep -q "foxhunt_" && echo "✅ Port $port OK" || echo "❌ Port $port FAILED"
|
||||
done
|
||||
```
|
||||
|
||||
### Test 3: Database Connectivity
|
||||
|
||||
```bash
|
||||
# Test database from each service
|
||||
psql $DATABASE_URL -c "SELECT count(*) FROM information_schema.tables;"
|
||||
redis-cli ping
|
||||
```
|
||||
|
||||
### Test 4: GPU Access (ML Services)
|
||||
|
||||
```bash
|
||||
# Docker
|
||||
docker-compose exec ml_training_service nvidia-smi
|
||||
|
||||
# Bare-metal
|
||||
nvidia-smi
|
||||
```
|
||||
|
||||
### Test 5: End-to-End (Optional)
|
||||
|
||||
```bash
|
||||
# Requires TLI or gRPC client
|
||||
# Test API Gateway → Trading Service flow
|
||||
grpcurl -plaintext localhost:50051 grpc.health.v1.Health/Check
|
||||
|
||||
# Expected: {"status": "SERVING"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🚨 Troubleshooting Quick Fixes
|
||||
|
||||
### Issue: Service Won't Start
|
||||
|
||||
```bash
|
||||
# Check logs
|
||||
docker-compose logs <service> # Docker
|
||||
sudo journalctl -u foxhunt-<service> -n 50 # Bare-metal
|
||||
|
||||
# Common fix: Restart
|
||||
docker-compose restart <service>
|
||||
sudo systemctl restart foxhunt-<service>
|
||||
```
|
||||
|
||||
### Issue: Missing Environment Variable
|
||||
|
||||
```bash
|
||||
# Docker: Add to .env
|
||||
echo "MISSING_VAR=value" >> .env
|
||||
docker-compose up -d <service>
|
||||
|
||||
# Bare-metal: Add to SystemD
|
||||
sudo systemctl edit foxhunt-<service>
|
||||
# Add: Environment="MISSING_VAR=value"
|
||||
sudo systemctl restart foxhunt-<service>
|
||||
```
|
||||
|
||||
### Issue: GPU Not Detected
|
||||
|
||||
```bash
|
||||
# Docker: Install nvidia-docker2
|
||||
sudo apt-get install -y nvidia-docker2
|
||||
sudo systemctl restart docker
|
||||
|
||||
# Verify
|
||||
docker run --rm --gpus all nvidia/cuda:12.3.0-base-ubuntu22.04 nvidia-smi
|
||||
```
|
||||
|
||||
### Issue: Port Already in Use
|
||||
|
||||
```bash
|
||||
# Find process
|
||||
sudo lsof -i :<port>
|
||||
|
||||
# Kill process
|
||||
sudo kill -9 <PID>
|
||||
|
||||
# Restart service
|
||||
docker-compose up -d <service>
|
||||
```
|
||||
|
||||
### Issue: Database Connection Failed
|
||||
|
||||
```bash
|
||||
# Verify PostgreSQL running
|
||||
docker-compose ps postgres # Docker
|
||||
sudo systemctl status postgresql # Bare-metal
|
||||
|
||||
# Test connection
|
||||
psql $DATABASE_URL -c "SELECT 1;"
|
||||
|
||||
# Restart database
|
||||
docker-compose restart postgres
|
||||
sudo systemctl restart postgresql
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📊 Monitoring Access
|
||||
|
||||
- **Grafana**: http://localhost:3000 (admin / foxhunt123)
|
||||
- **Prometheus**: http://localhost:9090
|
||||
- **Metrics**:
|
||||
- Trading: http://localhost:9092/metrics
|
||||
- Backtesting: http://localhost:9093/metrics
|
||||
- ML Training: http://localhost:9094/metrics
|
||||
- API Gateway: http://localhost:9091/metrics
|
||||
|
||||
---
|
||||
|
||||
## 📞 Next Steps
|
||||
|
||||
1. **Configure monitoring alerts**: Edit `config/prometheus/rules/alerts.yml`
|
||||
2. **Set up backups**: Configure daily PostgreSQL backups to S3
|
||||
3. **Enable TLS**: Configure SSL/TLS certificates for production
|
||||
4. **Review security**: Rotate default passwords and tokens
|
||||
5. **Load testing**: Run `services/load_tests` to validate performance
|
||||
|
||||
---
|
||||
|
||||
## 📚 Full Documentation
|
||||
|
||||
- **Complete Runbook**: `PRODUCTION_DEPLOYMENT_RUNBOOK.md`
|
||||
- **Emergency Procedures**: `EMERGENCY_PROCEDURES.md`
|
||||
- **Maintenance Checklist**: `MAINTENANCE_CHECKLIST.md`
|
||||
- **Architecture**: `CLAUDE.md`
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-10-07
|
||||
**Version**: 1.0.0
|
||||
**Estimated Time**: 15-20 minutes (Docker), 30-45 minutes (Bare-Metal)
|
||||
862
SCALING_PLAYBOOK.md
Normal file
862
SCALING_PLAYBOOK.md
Normal file
@@ -0,0 +1,862 @@
|
||||
# Scaling Playbook - Foxhunt HFT Trading System
|
||||
|
||||
**Last Updated**: 2025-10-07
|
||||
**Status**: Production Ready
|
||||
**Audience**: DevOps, SRE, Platform Engineers
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Overview](#overview)
|
||||
2. [When to Scale](#when-to-scale)
|
||||
3. [Monitoring Metrics](#monitoring-metrics)
|
||||
4. [Scaling Decisions](#scaling-decisions)
|
||||
5. [Cost vs Performance Tradeoffs](#cost-vs-performance-tradeoffs)
|
||||
6. [Scaling Procedures](#scaling-procedures)
|
||||
7. [Troubleshooting](#troubleshooting)
|
||||
8. [Emergency Response](#emergency-response)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
This playbook provides operational guidance for scaling the Foxhunt HFT Trading System. It covers:
|
||||
- **When** to scale (metrics and thresholds)
|
||||
- **How** to scale (manual and automated procedures)
|
||||
- **Why** to scale (performance, cost, reliability)
|
||||
- **What** to monitor (metrics and alerts)
|
||||
|
||||
### Scaling Philosophy
|
||||
|
||||
1. **Proactive Scaling**: Scale before problems occur (monitor trends)
|
||||
2. **Cost-Aware Scaling**: Balance performance with cost
|
||||
3. **Automated First**: Use HPA for routine scaling, manual for exceptions
|
||||
4. **Graceful Degradation**: System should degrade gracefully under load
|
||||
5. **Measured Response**: Base decisions on metrics, not gut feeling
|
||||
|
||||
---
|
||||
|
||||
## When to Scale
|
||||
|
||||
### Scale-Up Triggers (Add Capacity)
|
||||
|
||||
#### Critical (Immediate Action Required)
|
||||
|
||||
| Metric | Threshold | Action | Timeline |
|
||||
|--------|-----------|--------|----------|
|
||||
| P99 Latency | > 100ms | Scale up immediately | < 5 minutes |
|
||||
| Error Rate | > 1% | Scale up immediately | < 5 minutes |
|
||||
| CPU Utilization | > 90% | Scale up immediately | < 5 minutes |
|
||||
| Memory Utilization | > 95% | Scale up immediately | < 2 minutes |
|
||||
| Service Down | Any instance down | Replace/scale up | < 2 minutes |
|
||||
|
||||
**Example Alert**:
|
||||
```
|
||||
CRITICAL: API Gateway P99 Latency at 150ms (threshold: 100ms)
|
||||
Current replicas: 3, CPU: 92%, Memory: 78%
|
||||
Action: Scale up to 6 replicas immediately
|
||||
```
|
||||
|
||||
#### Warning (Plan Scaling Within 1 Hour)
|
||||
|
||||
| Metric | Threshold | Action | Timeline |
|
||||
|--------|-----------|--------|----------|
|
||||
| P99 Latency | > 50ms | Plan scale-up | < 1 hour |
|
||||
| Error Rate | > 0.5% | Plan scale-up | < 1 hour |
|
||||
| CPU Utilization | > 80% | Plan scale-up | < 1 hour |
|
||||
| Memory Utilization | > 85% | Plan scale-up | < 1 hour |
|
||||
| Request Queue Depth | > 100 | Plan scale-up | < 30 minutes |
|
||||
|
||||
**Example Alert**:
|
||||
```
|
||||
WARNING: Trading Service CPU at 85% (threshold: 80%)
|
||||
Current replicas: 5, Request rate: 8K req/s
|
||||
Action: Review traffic patterns, plan scale-up to 7 replicas
|
||||
```
|
||||
|
||||
#### Proactive (Plan Scaling Within 24 Hours)
|
||||
|
||||
| Metric | Threshold | Action | Timeline |
|
||||
|--------|-----------|--------|----------|
|
||||
| Traffic Growth | > 20% week-over-week | Plan capacity expansion | < 1 week |
|
||||
| CPU Utilization Trend | Increasing 10%/week | Plan scale-up | < 1 week |
|
||||
| Peak Traffic Approaching | Predicted > 80% capacity | Pre-scale before peak | < 24 hours |
|
||||
| Scheduled Events | Known high-traffic events | Pre-scale 1 hour before | < 1 hour before event |
|
||||
|
||||
**Example**:
|
||||
```
|
||||
INFO: Traffic growing 25% week-over-week (5K → 6.25K req/s)
|
||||
Current capacity: 10K req/s (10 replicas)
|
||||
Projected capacity need (4 weeks): 12.5K req/s
|
||||
Action: Plan to add 2 replicas (12 total) within 2 weeks
|
||||
```
|
||||
|
||||
### Scale-Down Triggers (Reduce Capacity)
|
||||
|
||||
#### Safe to Scale Down
|
||||
|
||||
| Metric | Threshold | Action | Timeline |
|
||||
|--------|-----------|--------|----------|
|
||||
| CPU Utilization | < 40% for 30 minutes | Scale down by 25% | < 10 minutes |
|
||||
| Memory Utilization | < 50% for 30 minutes | Scale down by 25% | < 10 minutes |
|
||||
| Request Rate | < 50% of capacity | Scale down by 25% | < 10 minutes |
|
||||
| Off-Peak Hours | Night/weekend (known pattern) | Scale down to min replicas | Scheduled |
|
||||
|
||||
**Example**:
|
||||
```
|
||||
INFO: API Gateway CPU at 35% for 45 minutes (threshold: 40% for 30m)
|
||||
Current replicas: 10, Request rate: 2K req/s (capacity: 20K req/s)
|
||||
Action: Scale down to 8 replicas (save 20% cost, maintain 10K req/s capacity)
|
||||
```
|
||||
|
||||
#### Never Scale Down
|
||||
|
||||
- Error rate > 0.1%
|
||||
- P99 latency > 20ms
|
||||
- Recent scale-up (< 10 minutes ago)
|
||||
- During peak traffic hours
|
||||
- Less than minReplicas (HA requirement)
|
||||
|
||||
---
|
||||
|
||||
## Monitoring Metrics
|
||||
|
||||
### Service-Level Metrics (RED Metrics)
|
||||
|
||||
**Request Rate**:
|
||||
```promql
|
||||
# Total request rate across all pods
|
||||
sum(rate(grpc_requests_total{service="trading-service"}[5m]))
|
||||
|
||||
# Per-pod request rate
|
||||
rate(grpc_requests_total{service="trading-service"}[5m])
|
||||
```
|
||||
|
||||
**Error Rate**:
|
||||
```promql
|
||||
# Error percentage
|
||||
sum(rate(grpc_requests_total{service="trading-service",status="error"}[5m])) /
|
||||
sum(rate(grpc_requests_total{service="trading-service"}[5m])) * 100
|
||||
|
||||
# Alert if > 0.5%
|
||||
```
|
||||
|
||||
**Latency (Duration)**:
|
||||
```promql
|
||||
# P99 latency
|
||||
histogram_quantile(0.99,
|
||||
rate(grpc_request_duration_seconds_bucket{service="trading-service"}[5m]))
|
||||
|
||||
# P50, P95, P99.9
|
||||
histogram_quantile(0.50, ...)
|
||||
histogram_quantile(0.95, ...)
|
||||
histogram_quantile(0.999, ...)
|
||||
```
|
||||
|
||||
### Resource Utilization Metrics
|
||||
|
||||
**CPU**:
|
||||
```promql
|
||||
# Average CPU utilization across pods
|
||||
avg(rate(container_cpu_usage_seconds_total{pod=~"trading-service.*"}[5m])) * 100
|
||||
|
||||
# Per-pod CPU
|
||||
rate(container_cpu_usage_seconds_total{pod="trading-service-1"}[5m]) * 100
|
||||
```
|
||||
|
||||
**Memory**:
|
||||
```promql
|
||||
# Average memory utilization
|
||||
avg(container_memory_working_set_bytes{pod=~"trading-service.*"}) /
|
||||
avg(container_spec_memory_limit_bytes{pod=~"trading-service.*"}) * 100
|
||||
|
||||
# Memory usage in MB
|
||||
container_memory_working_set_bytes{pod="trading-service-1"} / 1024 / 1024
|
||||
```
|
||||
|
||||
### Scaling Metrics
|
||||
|
||||
**Replica Count**:
|
||||
```promql
|
||||
# Current replicas
|
||||
kube_deployment_status_replicas{deployment="trading-service"}
|
||||
|
||||
# Desired replicas (from HPA)
|
||||
kube_horizontalpodautoscaler_status_desired_replicas{hpa="trading-service-hpa"}
|
||||
|
||||
# Available replicas (healthy)
|
||||
kube_deployment_status_replicas_available{deployment="trading-service"}
|
||||
```
|
||||
|
||||
**Scaling Events**:
|
||||
```bash
|
||||
# View recent scaling events
|
||||
kubectl get events -n foxhunt \
|
||||
--field-selector involvedObject.kind=HorizontalPodAutoscaler \
|
||||
--sort-by='.lastTimestamp'
|
||||
|
||||
# Example output:
|
||||
# 5m ago: ScaledUp from 5 to 7 replicas (CPU: 85%)
|
||||
# 15m ago: ScaledDown from 10 to 8 replicas (CPU: 35%)
|
||||
```
|
||||
|
||||
### Database Metrics
|
||||
|
||||
**Connection Pool**:
|
||||
```promql
|
||||
# PostgreSQL active connections
|
||||
pg_stat_database_numbackends{datname="foxhunt"}
|
||||
|
||||
# Connection pool utilization
|
||||
pg_stat_database_numbackends / pg_settings_max_connections * 100
|
||||
|
||||
# Alert if > 80%
|
||||
```
|
||||
|
||||
**Replication Lag**:
|
||||
```promql
|
||||
# Replication lag (should be < 100ms)
|
||||
pg_stat_replication_replay_lag_bytes
|
||||
|
||||
# Convert to milliseconds
|
||||
pg_stat_replication_replay_lag_seconds * 1000
|
||||
```
|
||||
|
||||
**Query Performance**:
|
||||
```promql
|
||||
# Slow queries (> 100ms)
|
||||
pg_stat_statements_mean_exec_time_seconds{query=~".*"} > 0.1
|
||||
|
||||
# Top 10 slowest queries
|
||||
topk(10, pg_stat_statements_mean_exec_time_seconds)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Scaling Decisions
|
||||
|
||||
### Decision Matrix
|
||||
|
||||
| Symptom | Root Cause | Solution | Cost Impact |
|
||||
|---------|-----------|----------|-------------|
|
||||
| **High CPU (> 80%)** | Insufficient compute | Scale up replicas | **High** (+20% cost per replica) |
|
||||
| **High Memory (> 90%)** | Memory leak or under-provisioned | Fix leak OR increase memory limits | **Medium** (10-20% cost increase) |
|
||||
| **High Latency + Low CPU** | Database bottleneck | Add read replicas | **Medium** (+30% DB cost) |
|
||||
| **High Error Rate** | Service overload | Scale up replicas | **High** (+20% cost per replica) |
|
||||
| **Connection Exhaustion** | Too many service instances | Add PgBouncer | **Low** (<5% cost increase) |
|
||||
| **Redis Memory Full** | Cache size exceeds capacity | Add Redis Cluster | **Medium** (+50% Redis cost) |
|
||||
| **Queue Depth High** | Worker shortage | Scale up workers | **High** (+20% per worker) |
|
||||
| **Disk I/O High** | Storage bottleneck | Upgrade disk (SSD → NVMe) | **High** (2-3x storage cost) |
|
||||
|
||||
### Decision Tree
|
||||
|
||||
```
|
||||
Start: Is there a performance issue?
|
||||
├─ Yes: What metric is degraded?
|
||||
│ ├─ Latency (P99 > 100ms)
|
||||
│ │ ├─ CPU high? → Scale up replicas
|
||||
│ │ ├─ Database slow? → Add read replicas or optimize queries
|
||||
│ │ └─ Network slow? → Check load balancer, network config
|
||||
│ ├─ Error Rate (> 0.5%)
|
||||
│ │ ├─ Service overload? → Scale up replicas
|
||||
│ │ ├─ Database errors? → Check connection pool, add replicas
|
||||
│ │ └─ External API errors? → Implement retry, circuit breaker
|
||||
│ └─ Throughput (cannot handle load)
|
||||
│ ├─ CPU bottleneck? → Scale up replicas
|
||||
│ ├─ Database bottleneck? → Optimize queries, add replicas
|
||||
│ └─ Network bottleneck? → Upgrade network, load balancer
|
||||
└─ No: Is cost optimization possible?
|
||||
├─ Over-provisioned (CPU < 40%)? → Scale down replicas
|
||||
├─ Off-peak hours? → Scale down to minReplicas
|
||||
└─ Right-size instances? → Switch to smaller instance types
|
||||
```
|
||||
|
||||
### Example Scenarios
|
||||
|
||||
#### Scenario 1: Trading Service High CPU
|
||||
|
||||
**Symptoms**:
|
||||
- CPU: 92% (threshold: 80%)
|
||||
- P99 Latency: 75ms (threshold: 100ms, target: 5ms)
|
||||
- Error Rate: 0.3% (threshold: 0.5%)
|
||||
- Current replicas: 5
|
||||
- Request rate: 10K orders/sec
|
||||
|
||||
**Analysis**:
|
||||
- CPU high but latency still acceptable
|
||||
- Error rate below threshold
|
||||
- Approaching capacity (10K orders/sec ÷ 5 = 2K orders/sec per pod)
|
||||
|
||||
**Decision**: Scale up proactively
|
||||
- Target CPU: 70% (with 20% headroom)
|
||||
- Required replicas: 10K orders/sec ÷ (2K × 0.7) = 7.14 → 8 replicas
|
||||
- Cost impact: +60% (5 → 8 replicas)
|
||||
|
||||
**Action**:
|
||||
```bash
|
||||
# Manual scale-up
|
||||
kubectl scale deployment trading-service --replicas=8 -n foxhunt
|
||||
|
||||
# Verify scaling
|
||||
watch kubectl get pods -n foxhunt -l app=trading-service
|
||||
```
|
||||
|
||||
#### Scenario 2: Database Connection Exhaustion
|
||||
|
||||
**Symptoms**:
|
||||
- Error: "FATAL: remaining connection slots are reserved"
|
||||
- PostgreSQL connections: 95/100
|
||||
- Services running: API Gateway (3), Trading (10), Backtesting (5)
|
||||
- Expected connections: (3 + 10 + 5) × 10 = 180 (exceeds limit!)
|
||||
|
||||
**Analysis**:
|
||||
- Connection pool not configured
|
||||
- Each service instance uses 10 connections
|
||||
- Total connections exceed PostgreSQL limit
|
||||
|
||||
**Decision**: Add PgBouncer connection pooler
|
||||
- PgBouncer pools 180 connections → 50 PostgreSQL connections
|
||||
- No need to reduce service instances
|
||||
|
||||
**Action**:
|
||||
```bash
|
||||
# Deploy PgBouncer
|
||||
kubectl apply -f pgbouncer-deployment.yaml
|
||||
|
||||
# Update service connection strings
|
||||
DATABASE_URL=postgresql://foxhunt:password@pgbouncer:6432/foxhunt
|
||||
```
|
||||
|
||||
#### Scenario 3: Pre-Scaling for Known Event
|
||||
|
||||
**Symptoms**:
|
||||
- Scheduled event: Market open (9:30 AM ET)
|
||||
- Historical data: Traffic spikes 5x (2K → 10K req/s)
|
||||
- Current capacity: 5 replicas × 2K req/s = 10K req/s (at limit!)
|
||||
- Risk: Latency spike, potential errors
|
||||
|
||||
**Analysis**:
|
||||
- Need capacity for 10K req/s with headroom
|
||||
- Target: 15K req/s capacity (50% headroom)
|
||||
- Required replicas: 15K ÷ 2K = 7.5 → 8 replicas
|
||||
|
||||
**Decision**: Pre-scale 30 minutes before event
|
||||
- Scale up to 8 replicas at 9:00 AM
|
||||
- Monitor during event (9:30 AM - 10:00 AM)
|
||||
- Scale down to 5 replicas at 11:00 AM (if traffic normalizes)
|
||||
|
||||
**Action**:
|
||||
```bash
|
||||
# Pre-scale at 9:00 AM
|
||||
kubectl scale deployment trading-service --replicas=8 -n foxhunt
|
||||
|
||||
# Monitor during event
|
||||
watch kubectl top pods -n foxhunt
|
||||
|
||||
# Scale down at 11:00 AM (if safe)
|
||||
kubectl scale deployment trading-service --replicas=5 -n foxhunt
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Cost vs Performance Tradeoffs
|
||||
|
||||
### Cost Models
|
||||
|
||||
**Instance Cost** (AWS t3.medium, us-east-1):
|
||||
| Pricing Model | $/hour | $/month | Savings |
|
||||
|---------------|--------|---------|---------|
|
||||
| On-Demand | $0.0416 | $30.37 | 0% |
|
||||
| Reserved (1yr) | $0.025 | $18.25 | 40% |
|
||||
| Reserved (3yr) | $0.016 | $11.68 | 62% |
|
||||
| Spot (avg) | $0.0125 | $9.11 | 70% |
|
||||
|
||||
**Scaling Cost Example** (Trading Service):
|
||||
```
|
||||
Base: 5 replicas × $30.37/mo = $151.85/mo (on-demand)
|
||||
Peak: 10 replicas × $30.37/mo = $303.70/mo (on-demand)
|
||||
|
||||
Strategy 1: All On-Demand
|
||||
Cost: $303.70/mo (worst case)
|
||||
|
||||
Strategy 2: Base Reserved + Peak On-Demand
|
||||
Base: 5 replicas × $18.25/mo = $91.25/mo (reserved)
|
||||
Peak: 5 replicas × $30.37/mo × 30% uptime = $45.56/mo (on-demand)
|
||||
Total: $136.81/mo
|
||||
Savings: $166.89/mo (55% vs all on-demand)
|
||||
|
||||
Strategy 3: Base Reserved + Peak Spot
|
||||
Base: 5 replicas × $18.25/mo = $91.25/mo (reserved)
|
||||
Peak: 5 replicas × $9.11/mo × 30% uptime = $13.67/mo (spot)
|
||||
Total: $104.92/mo
|
||||
Savings: $198.78/mo (65% vs all on-demand)
|
||||
```
|
||||
|
||||
### Performance Targets vs Cost
|
||||
|
||||
| Target | Replicas | Cost/Month | Latency (P99) | Throughput |
|
||||
|--------|----------|------------|---------------|------------|
|
||||
| Minimum (HA) | 3 | $91.11 | 10-20ms | 6K req/s |
|
||||
| Balanced | 5 | $151.85 | 5-10ms | 10K req/s |
|
||||
| High Performance | 10 | $303.70 | 2-5ms | 20K req/s |
|
||||
| Peak Capacity | 20 | $607.40 | < 2ms | 40K req/s |
|
||||
|
||||
**Recommendation**: Use "Balanced" (5 replicas) as baseline, scale to 10-20 for peak traffic.
|
||||
|
||||
### Cost Optimization Strategies
|
||||
|
||||
#### 1. Right-Size Instances
|
||||
|
||||
**Before**:
|
||||
```
|
||||
Instance: t3.large (2 vCPU, 8GB RAM)
|
||||
Utilization: CPU 35%, Memory 40%
|
||||
Cost: $60.74/mo
|
||||
```
|
||||
|
||||
**After**:
|
||||
```
|
||||
Instance: t3.medium (2 vCPU, 4GB RAM)
|
||||
Utilization: CPU 70%, Memory 80%
|
||||
Cost: $30.37/mo
|
||||
Savings: $30.37/mo (50%)
|
||||
```
|
||||
|
||||
#### 2. Use Reserved Instances for Base Load
|
||||
|
||||
**Before** (all on-demand):
|
||||
```
|
||||
10 instances × $30.37/mo = $303.70/mo
|
||||
```
|
||||
|
||||
**After** (5 reserved + 5 on-demand):
|
||||
```
|
||||
5 reserved × $18.25/mo = $91.25/mo
|
||||
5 on-demand × $30.37/mo × 30% uptime = $45.56/mo
|
||||
Total: $136.81/mo
|
||||
Savings: $166.89/mo (55%)
|
||||
```
|
||||
|
||||
#### 3. Use Spot Instances for Non-Critical Workloads
|
||||
|
||||
**Backtesting Workers** (spot instances):
|
||||
```
|
||||
Before: 10 workers × $60.74/mo (c6i.xlarge on-demand) = $607.40/mo
|
||||
After: 10 workers × $18.22/mo (c6i.xlarge spot, 70% discount) = $182.20/mo
|
||||
Savings: $425.20/mo (70%)
|
||||
```
|
||||
|
||||
**Caution**: Only use spot for fault-tolerant workloads (backtesting, ML training). Never for trading service!
|
||||
|
||||
#### 4. Auto-Scale During Off-Peak Hours
|
||||
|
||||
**Peak Hours** (9:00 AM - 4:00 PM ET, weekdays):
|
||||
```
|
||||
Replicas: 10 (high traffic)
|
||||
Cost: 10 × $30.37/mo × 35% time = $106.30/mo
|
||||
```
|
||||
|
||||
**Off-Peak Hours** (nights, weekends):
|
||||
```
|
||||
Replicas: 3 (minimal traffic)
|
||||
Cost: 3 × $30.37/mo × 65% time = $59.22/mo
|
||||
```
|
||||
|
||||
**Total Cost**: $165.52/mo (vs $303.70/mo always 10 replicas)
|
||||
**Savings**: $138.18/mo (45%)
|
||||
|
||||
---
|
||||
|
||||
## Scaling Procedures
|
||||
|
||||
### Manual Scaling (Kubernetes)
|
||||
|
||||
#### Scale Up
|
||||
|
||||
```bash
|
||||
# 1. Check current replica count
|
||||
kubectl get deployment trading-service -n foxhunt
|
||||
|
||||
# 2. Scale up
|
||||
kubectl scale deployment trading-service --replicas=10 -n foxhunt
|
||||
|
||||
# 3. Verify new pods are running
|
||||
kubectl get pods -n foxhunt -l app=trading-service
|
||||
|
||||
# 4. Wait for pods to be ready (STATUS: Running, READY: 1/1)
|
||||
kubectl wait --for=condition=ready pod -l app=trading-service -n foxhunt --timeout=5m
|
||||
|
||||
# 5. Verify load is distributed
|
||||
kubectl top pods -n foxhunt -l app=trading-service
|
||||
```
|
||||
|
||||
#### Scale Down
|
||||
|
||||
```bash
|
||||
# 1. Verify load is low (CPU < 50%, latency normal)
|
||||
kubectl top pods -n foxhunt -l app=trading-service
|
||||
|
||||
# 2. Drain pods gracefully (one at a time)
|
||||
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
|
||||
|
||||
# 3. Scale down
|
||||
kubectl scale deployment trading-service --replicas=5 -n foxhunt
|
||||
|
||||
# 4. Verify no errors or latency spikes
|
||||
# (Monitor Prometheus/Grafana for 5 minutes)
|
||||
```
|
||||
|
||||
### Automated Scaling (HPA)
|
||||
|
||||
#### Enable HPA
|
||||
|
||||
```bash
|
||||
# 1. Apply HPA manifest
|
||||
kubectl apply -f config/k8s/hpa.yaml
|
||||
|
||||
# 2. Verify HPA is active
|
||||
kubectl get hpa -n foxhunt
|
||||
|
||||
# 3. Check HPA status
|
||||
kubectl describe hpa trading-service-hpa -n foxhunt
|
||||
|
||||
# Expected output:
|
||||
# Current replicas: 5
|
||||
# Desired replicas: 5
|
||||
# Metrics: CPU 65% (target: 70%)
|
||||
```
|
||||
|
||||
#### Modify HPA Thresholds
|
||||
|
||||
```bash
|
||||
# 1. Edit HPA
|
||||
kubectl edit hpa trading-service-hpa -n foxhunt
|
||||
|
||||
# 2. Change CPU threshold (e.g., 70% → 60%)
|
||||
# metrics:
|
||||
# - type: Resource
|
||||
# resource:
|
||||
# name: cpu
|
||||
# target:
|
||||
# type: Utilization
|
||||
# averageUtilization: 60 # Changed from 70
|
||||
|
||||
# 3. Save and exit (HPA updates immediately)
|
||||
|
||||
# 4. Verify change
|
||||
kubectl describe hpa trading-service-hpa -n foxhunt
|
||||
```
|
||||
|
||||
#### Disable HPA (Emergency)
|
||||
|
||||
```bash
|
||||
# 1. Delete HPA (keeps current replica count)
|
||||
kubectl delete hpa trading-service-hpa -n foxhunt
|
||||
|
||||
# 2. Verify HPA is gone
|
||||
kubectl get hpa -n foxhunt
|
||||
|
||||
# 3. Deployment is now manually managed
|
||||
kubectl scale deployment trading-service --replicas=<desired> -n foxhunt
|
||||
```
|
||||
|
||||
### Database Scaling
|
||||
|
||||
#### Add Read Replica
|
||||
|
||||
```bash
|
||||
# 1. Create read replica (PostgreSQL)
|
||||
# (Cloud provider specific: AWS RDS, GCP Cloud SQL, etc.)
|
||||
|
||||
# Example (Kubernetes StatefulSet):
|
||||
kubectl scale statefulset postgres-replica --replicas=2 -n foxhunt
|
||||
|
||||
# 2. Update application to use read replica for read queries
|
||||
# (See "Application-Level Read/Write Splitting" in LOAD_BALANCING_SCALING.md)
|
||||
|
||||
# 3. Verify replication lag
|
||||
psql -h postgres-replica-1 -c "SELECT pg_last_wal_receive_lsn(), pg_last_wal_replay_lsn();"
|
||||
```
|
||||
|
||||
#### Scale Redis Cluster
|
||||
|
||||
```bash
|
||||
# 1. Add Redis node
|
||||
redis-cli --cluster add-node new-node:6379 existing-node:6379
|
||||
|
||||
# 2. Rebalance cluster (distribute keys)
|
||||
redis-cli --cluster rebalance existing-node:6379
|
||||
|
||||
# 3. Verify cluster status
|
||||
redis-cli --cluster check existing-node:6379
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Issue: HPA Not Scaling Up
|
||||
|
||||
**Symptoms**:
|
||||
- CPU > 80% but HPA not scaling
|
||||
- Desired replicas = Current replicas (no change)
|
||||
|
||||
**Possible Causes**:
|
||||
1. **Stabilization window**: HPA waiting before scaling
|
||||
2. **Resource quota exceeded**: Namespace/cluster out of resources
|
||||
3. **Node capacity**: No nodes available for new pods
|
||||
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
# 1. Check HPA status
|
||||
kubectl describe hpa trading-service-hpa -n foxhunt
|
||||
# Look for: "ScalingLimited" or "BackoffBoth" events
|
||||
|
||||
# 2. Check resource quotas
|
||||
kubectl describe resourcequota -n foxhunt
|
||||
|
||||
# 3. Check node capacity
|
||||
kubectl describe nodes | grep -A 5 "Allocated resources"
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
- Wait for stabilization window to pass (30-60s)
|
||||
- Increase resource quotas (if insufficient)
|
||||
- Add nodes to cluster (if no capacity)
|
||||
|
||||
### Issue: Pods Evicted After Scale-Up
|
||||
|
||||
**Symptoms**:
|
||||
- New pods created, then immediately evicted
|
||||
- Pod status: "Evicted"
|
||||
|
||||
**Possible Causes**:
|
||||
1. **Resource pressure**: Node out of memory/disk
|
||||
2. **Resource limits too high**: Pod requests exceed node capacity
|
||||
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
# 1. Check pod status
|
||||
kubectl describe pod <pod-name> -n foxhunt
|
||||
# Look for: "Reason: Evicted"
|
||||
|
||||
# 2. Check node resources
|
||||
kubectl top nodes
|
||||
|
||||
# 3. Check pod resource requests
|
||||
kubectl get pod <pod-name> -n foxhunt -o yaml | grep -A 10 resources
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
- Reduce resource requests/limits (if too high)
|
||||
- Add nodes with more resources
|
||||
- Delete unused pods to free resources
|
||||
|
||||
### Issue: High Latency Despite Scaling Up
|
||||
|
||||
**Symptoms**:
|
||||
- Replicas increased (5 → 10) but P99 latency still high (> 100ms)
|
||||
- CPU/memory usage normal
|
||||
|
||||
**Possible Causes**:
|
||||
1. **Database bottleneck**: Queries slow, connection pool exhausted
|
||||
2. **Network bottleneck**: Load balancer misconfigured
|
||||
3. **External API bottleneck**: Dependency slow
|
||||
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
# 1. Check database query performance
|
||||
psql -h postgres -c "SELECT query, mean_exec_time FROM pg_stat_statements ORDER BY mean_exec_time DESC LIMIT 10;"
|
||||
|
||||
# 2. Check database connection pool
|
||||
psql -h postgres -c "SELECT count(*) FROM pg_stat_activity;"
|
||||
|
||||
# 3. Check external API latency
|
||||
# (Prometheus query)
|
||||
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{job="external-api"}[5m]))
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
- Optimize slow database queries (add indexes, rewrite)
|
||||
- Add database read replicas
|
||||
- Add PgBouncer connection pooler
|
||||
- Implement caching for external API calls
|
||||
|
||||
---
|
||||
|
||||
## Emergency Response
|
||||
|
||||
### Incident: Service Completely Down
|
||||
|
||||
**Severity**: P0 (Critical)
|
||||
**Response Time**: Immediate
|
||||
|
||||
**Symptoms**:
|
||||
- All pods in CrashLoopBackOff
|
||||
- Service unreachable (health check failing)
|
||||
- Error rate: 100%
|
||||
|
||||
**Immediate Actions** (within 5 minutes):
|
||||
|
||||
1. **Check pod status**:
|
||||
```bash
|
||||
kubectl get pods -n foxhunt -l app=trading-service
|
||||
kubectl logs <pod-name> -n foxhunt --tail=100
|
||||
```
|
||||
|
||||
2. **Rollback to last working version**:
|
||||
```bash
|
||||
kubectl rollout undo deployment trading-service -n foxhunt
|
||||
kubectl rollout status deployment trading-service -n foxhunt
|
||||
```
|
||||
|
||||
3. **Scale up to compensate**:
|
||||
```bash
|
||||
kubectl scale deployment trading-service --replicas=10 -n foxhunt
|
||||
```
|
||||
|
||||
4. **Notify stakeholders**:
|
||||
- Alert: "Trading Service down, rollback in progress"
|
||||
- ETA: 5 minutes to restore service
|
||||
|
||||
**Root Cause Analysis** (after service restored):
|
||||
- Check deployment changes (git log, CI/CD)
|
||||
- Check resource limits (CPU, memory, disk)
|
||||
- Check external dependencies (database, Redis, external APIs)
|
||||
|
||||
### Incident: Database Connection Exhaustion
|
||||
|
||||
**Severity**: P1 (High)
|
||||
**Response Time**: < 15 minutes
|
||||
|
||||
**Symptoms**:
|
||||
- Errors: "FATAL: remaining connection slots are reserved"
|
||||
- Services unable to connect to database
|
||||
- Error rate: 50-100%
|
||||
|
||||
**Immediate Actions**:
|
||||
|
||||
1. **Kill idle connections**:
|
||||
```sql
|
||||
SELECT pg_terminate_backend(pid)
|
||||
FROM pg_stat_activity
|
||||
WHERE state = 'idle' AND state_change < now() - interval '5 minutes';
|
||||
```
|
||||
|
||||
2. **Scale down services temporarily**:
|
||||
```bash
|
||||
# Reduce replicas to free connections
|
||||
kubectl scale deployment trading-service --replicas=3 -n foxhunt
|
||||
kubectl scale deployment backtesting-service --replicas=1 -n foxhunt
|
||||
```
|
||||
|
||||
3. **Deploy PgBouncer** (if not already):
|
||||
```bash
|
||||
kubectl apply -f pgbouncer-deployment.yaml
|
||||
```
|
||||
|
||||
4. **Update connection strings** (rolling update):
|
||||
```bash
|
||||
kubectl set env deployment/trading-service DATABASE_URL=postgresql://foxhunt:password@pgbouncer:6432/foxhunt
|
||||
```
|
||||
|
||||
### Incident: Traffic Spike (DDoS or Viral Event)
|
||||
|
||||
**Severity**: P1 (High)
|
||||
**Response Time**: < 10 minutes
|
||||
|
||||
**Symptoms**:
|
||||
- Request rate 10x normal (100K req/s vs 10K req/s)
|
||||
- P99 latency > 1s
|
||||
- Error rate > 5%
|
||||
|
||||
**Immediate Actions**:
|
||||
|
||||
1. **Enable aggressive rate limiting**:
|
||||
```nginx
|
||||
# nginx config
|
||||
limit_req_zone $binary_remote_addr zone=ddos:10m rate=10r/s;
|
||||
limit_req zone=ddos burst=20 nodelay;
|
||||
```
|
||||
|
||||
2. **Scale up to max capacity**:
|
||||
```bash
|
||||
kubectl scale deployment api-gateway --replicas=20 -n foxhunt
|
||||
kubectl scale deployment trading-service --replicas=20 -n foxhunt
|
||||
```
|
||||
|
||||
3. **Enable DDoS protection** (if available):
|
||||
- Cloudflare: Enable "I'm Under Attack" mode
|
||||
- AWS: Enable AWS Shield Advanced
|
||||
- Manual: Block suspicious IPs
|
||||
|
||||
4. **Shed non-critical traffic**:
|
||||
```nginx
|
||||
# Reject non-authenticated requests
|
||||
location /api {
|
||||
if ($http_authorization = "") {
|
||||
return 503 "Service temporarily unavailable";
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Post-Incident**:
|
||||
- Analyze traffic patterns (legitimate vs attack)
|
||||
- Implement permanent DDoS protection
|
||||
- Review capacity planning
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
### Common Commands
|
||||
|
||||
```bash
|
||||
# View current replica count
|
||||
kubectl get deployment -n foxhunt
|
||||
|
||||
# Scale deployment
|
||||
kubectl scale deployment <name> --replicas=<count> -n foxhunt
|
||||
|
||||
# View HPA status
|
||||
kubectl get hpa -n foxhunt
|
||||
kubectl describe hpa <name> -n foxhunt
|
||||
|
||||
# View pod resource usage
|
||||
kubectl top pods -n foxhunt
|
||||
|
||||
# View scaling events
|
||||
kubectl get events -n foxhunt --field-selector involvedObject.kind=HorizontalPodAutoscaler
|
||||
|
||||
# View pod logs
|
||||
kubectl logs <pod-name> -n foxhunt --tail=100
|
||||
|
||||
# Rollback deployment
|
||||
kubectl rollout undo deployment <name> -n foxhunt
|
||||
```
|
||||
|
||||
### Metrics Cheat Sheet
|
||||
|
||||
| Metric | Good | Warning | Critical |
|
||||
|--------|------|---------|----------|
|
||||
| CPU | < 70% | 70-85% | > 85% |
|
||||
| Memory | < 75% | 75-90% | > 90% |
|
||||
| P99 Latency | < 10ms | 10-50ms | > 50ms |
|
||||
| Error Rate | < 0.1% | 0.1-0.5% | > 0.5% |
|
||||
| Replica Count | >= minReplicas | < minReplicas | 0 |
|
||||
|
||||
---
|
||||
|
||||
## Additional Resources
|
||||
|
||||
- **Load Balancing**: See `LOAD_BALANCING_SCALING.md`
|
||||
- **Performance Benchmarks**: See `PERFORMANCE_BENCHMARKS.md`
|
||||
- **Monitoring Dashboards**: Grafana (http://localhost:3000)
|
||||
- **Prometheus Alerts**: Alertmanager (http://localhost:9093)
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-10-07
|
||||
**Version**: 1.0
|
||||
**Owner**: Platform Engineering Team
|
||||
365
config/haproxy-lb.cfg
Normal file
365
config/haproxy-lb.cfg
Normal file
@@ -0,0 +1,365 @@
|
||||
# HAProxy Load Balancer Configuration for Foxhunt HFT Trading System
|
||||
# Version: 1.0
|
||||
# Last Updated: 2025-10-07
|
||||
#
|
||||
# This configuration provides:
|
||||
# - Layer 7 (L7) gRPC load balancing
|
||||
# - TLS termination at load balancer
|
||||
# - Advanced health checks (gRPC Health Checking Protocol)
|
||||
# - Rate limiting (via stick tables)
|
||||
# - High availability with stats page
|
||||
# - Connection limits
|
||||
|
||||
# ============================================================================
|
||||
# GLOBAL SETTINGS
|
||||
# ============================================================================
|
||||
|
||||
global
|
||||
# Process management
|
||||
daemon
|
||||
maxconn 4096 # Maximum concurrent connections
|
||||
|
||||
# Logging
|
||||
log /dev/log local0
|
||||
log /dev/log local1 notice
|
||||
|
||||
# User/group for HAProxy process
|
||||
user haproxy
|
||||
group haproxy
|
||||
|
||||
# Security
|
||||
chroot /var/lib/haproxy
|
||||
pidfile /var/run/haproxy.pid
|
||||
|
||||
# SSL/TLS settings
|
||||
ssl-default-bind-ciphers ECDHE+AESGCM:ECDHE+CHACHA20:!aNULL:!MD5:!DSS
|
||||
ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
|
||||
ssl-default-server-ciphers ECDHE+AESGCM:ECDHE+CHACHA20:!aNULL:!MD5:!DSS
|
||||
ssl-default-server-options ssl-min-ver TLSv1.2 no-tls-tickets
|
||||
|
||||
# Performance tuning
|
||||
tune.ssl.default-dh-param 2048
|
||||
tune.h2.initial-window-size 65536
|
||||
tune.h2.max-concurrent-streams 128
|
||||
|
||||
# ============================================================================
|
||||
# DEFAULTS
|
||||
# ============================================================================
|
||||
|
||||
defaults
|
||||
# Mode (tcp for raw TCP, http for HTTP/gRPC)
|
||||
mode http
|
||||
|
||||
# Logging
|
||||
log global
|
||||
option httplog
|
||||
option dontlognull
|
||||
|
||||
# Timeouts
|
||||
timeout connect 10s # Time to establish connection to backend
|
||||
timeout client 300s # Client inactivity timeout (5 minutes for long-running gRPC)
|
||||
timeout server 300s # Server inactivity timeout
|
||||
timeout http-request 10s
|
||||
timeout http-keep-alive 10s
|
||||
|
||||
# Health checks
|
||||
timeout check 5s
|
||||
|
||||
# Error handling
|
||||
retries 3
|
||||
option redispatch
|
||||
|
||||
# HTTP/2 support (required for gRPC)
|
||||
option http-use-htx # Enable HTTP/2
|
||||
|
||||
# ============================================================================
|
||||
# STATS PAGE (Monitoring)
|
||||
# ============================================================================
|
||||
|
||||
listen stats
|
||||
bind *:8404
|
||||
mode http
|
||||
stats enable
|
||||
stats uri /stats
|
||||
stats refresh 10s
|
||||
stats show-legends
|
||||
stats show-node
|
||||
|
||||
# Authentication (change username/password in production)
|
||||
stats auth admin:foxhunt123
|
||||
|
||||
# Additional stats options
|
||||
stats admin if TRUE # Enable admin commands (drain, disable servers)
|
||||
|
||||
# ============================================================================
|
||||
# FRONTEND (External Traffic)
|
||||
# ============================================================================
|
||||
|
||||
# HTTP Frontend (redirect to HTTPS)
|
||||
frontend http_frontend
|
||||
bind *:80
|
||||
mode http
|
||||
|
||||
# Redirect all HTTP to HTTPS
|
||||
http-request redirect scheme https code 301
|
||||
|
||||
# HTTPS Frontend (TLS termination + gRPC load balancing)
|
||||
frontend https_frontend
|
||||
# Bind to port 443 with TLS and HTTP/2 (ALPN: h2)
|
||||
bind *:443 ssl crt /etc/haproxy/certs/foxhunt.pem alpn h2,http/1.1
|
||||
|
||||
mode http
|
||||
option httplog
|
||||
|
||||
# Connection limits (DDoS protection)
|
||||
maxconn 2000
|
||||
|
||||
# ========================================================================
|
||||
# ACCESS CONTROL LISTS (ACLs)
|
||||
# ========================================================================
|
||||
|
||||
# Detect gRPC protocol (Content-Type: application/grpc)
|
||||
acl is_grpc hdr(content-type) -m beg application/grpc
|
||||
|
||||
# Route based on gRPC service path
|
||||
acl is_api_gateway path_beg /foxhunt.api_gateway
|
||||
acl is_trading_service path_beg /foxhunt.trading_service
|
||||
acl is_backtesting_service path_beg /foxhunt.backtesting_service
|
||||
acl is_ml_training_service path_beg /foxhunt.ml_training_service
|
||||
|
||||
# Internal IP ranges (for backend services)
|
||||
acl is_internal_ip src 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16
|
||||
|
||||
# Health check endpoint
|
||||
acl is_health_check path /health
|
||||
|
||||
# ========================================================================
|
||||
# RATE LIMITING (Stick Tables)
|
||||
# ========================================================================
|
||||
|
||||
# Track request rate per client IP
|
||||
stick-table type ip size 100k expire 30s store http_req_rate(10s)
|
||||
|
||||
# Track connection rate per client IP
|
||||
http-request track-sc0 src
|
||||
|
||||
# Rate limit: 1000 requests per 10 seconds per IP
|
||||
http-request deny deny_status 429 if { sc_http_req_rate(0) gt 1000 }
|
||||
|
||||
# Connection limit: 100 concurrent connections per IP
|
||||
http-request deny deny_status 429 if { src_conn_cur ge 100 }
|
||||
|
||||
# ========================================================================
|
||||
# REQUEST HEADERS
|
||||
# ========================================================================
|
||||
|
||||
# Add X-Forwarded-* headers (for backend logging)
|
||||
http-request set-header X-Real-IP %[src]
|
||||
http-request set-header X-Forwarded-For %[src]
|
||||
http-request set-header X-Forwarded-Proto https
|
||||
|
||||
# ========================================================================
|
||||
# ROUTING
|
||||
# ========================================================================
|
||||
|
||||
# Health check (return 200 OK)
|
||||
use_backend health_backend if is_health_check
|
||||
|
||||
# Route gRPC traffic to appropriate backend
|
||||
use_backend api_gateway_backend if is_grpc is_api_gateway
|
||||
use_backend trading_service_backend if is_grpc is_trading_service is_internal_ip
|
||||
use_backend backtesting_service_backend if is_grpc is_backtesting_service is_internal_ip
|
||||
use_backend ml_training_service_backend if is_grpc is_ml_training_service is_internal_ip
|
||||
|
||||
# Default: Return 404 for unknown routes
|
||||
default_backend not_found_backend
|
||||
|
||||
# ============================================================================
|
||||
# BACKENDS (Service Clusters)
|
||||
# ============================================================================
|
||||
|
||||
# API Gateway Backend (primary entry point)
|
||||
backend api_gateway_backend
|
||||
mode http
|
||||
balance leastconn # Route to backend with fewest connections (best for gRPC)
|
||||
|
||||
# HTTP/2 support
|
||||
option http-use-htx
|
||||
|
||||
# Health check using gRPC Health Checking Protocol
|
||||
option httpchk
|
||||
http-check send meth GET uri /grpc.health.v1.Health/Check ver HTTP/2
|
||||
http-check expect status 200
|
||||
|
||||
# Backend servers
|
||||
# Format: server <name> <host>:<port> [options]
|
||||
server api-gateway-1 api-gateway-1:50051 check inter 5s rise 2 fall 3 maxconn 1000
|
||||
server api-gateway-2 api-gateway-2:50051 check inter 5s rise 2 fall 3 maxconn 1000
|
||||
server api-gateway-3 api-gateway-3:50051 check inter 5s rise 2 fall 3 maxconn 1000
|
||||
|
||||
# Connection reuse (important for gRPC performance)
|
||||
http-reuse always
|
||||
|
||||
# Trading Service Backend
|
||||
backend trading_service_backend
|
||||
mode http
|
||||
balance leastconn
|
||||
|
||||
option http-use-htx
|
||||
|
||||
# Health check
|
||||
option httpchk
|
||||
http-check send meth GET uri /grpc.health.v1.Health/Check ver HTTP/2
|
||||
http-check expect status 200
|
||||
|
||||
# Backend servers
|
||||
server trading-service-1 trading-service-1:50052 check inter 5s rise 2 fall 3 maxconn 1000
|
||||
server trading-service-2 trading-service-2:50052 check inter 5s rise 2 fall 3 maxconn 1000
|
||||
server trading-service-3 trading-service-3:50052 check inter 5s rise 2 fall 3 maxconn 1000
|
||||
|
||||
http-reuse always
|
||||
|
||||
# Backtesting Service Backend (worker pool pattern)
|
||||
backend backtesting_service_backend
|
||||
mode http
|
||||
balance leastconn
|
||||
|
||||
option http-use-htx
|
||||
|
||||
# Health check
|
||||
option httpchk
|
||||
http-check send meth GET uri /grpc.health.v1.Health/Check ver HTTP/2
|
||||
http-check expect status 200
|
||||
|
||||
# Backend servers (fewer instances, CPU-intensive workload)
|
||||
server backtesting-service-1 backtesting-service-1:50053 check inter 5s rise 2 fall 3 maxconn 500
|
||||
server backtesting-service-2 backtesting-service-2:50053 check inter 5s rise 2 fall 3 maxconn 500
|
||||
|
||||
http-reuse always
|
||||
|
||||
# ML Training Service Backend (GPU instances)
|
||||
backend ml_training_service_backend
|
||||
mode http
|
||||
balance leastconn
|
||||
|
||||
option http-use-htx
|
||||
|
||||
# Health check
|
||||
option httpchk
|
||||
http-check send meth GET uri /grpc.health.v1.Health/Check ver HTTP/2
|
||||
http-check expect status 200
|
||||
|
||||
# Backend servers (GPU instances, fewer replicas)
|
||||
server ml-training-service-1 ml-training-service-1:50054 check inter 5s rise 2 fall 3 maxconn 200
|
||||
server ml-training-service-2 ml-training-service-2:50054 check inter 5s rise 2 fall 3 maxconn 200
|
||||
|
||||
http-reuse always
|
||||
|
||||
# Health Check Backend (returns 200 OK)
|
||||
backend health_backend
|
||||
mode http
|
||||
http-request return status 200 content-type "text/plain" string "healthy\n"
|
||||
|
||||
# Not Found Backend (returns 404)
|
||||
backend not_found_backend
|
||||
mode http
|
||||
http-request return status 404 content-type "text/plain" string "not found\n"
|
||||
|
||||
# ============================================================================
|
||||
# MONITORING & OBSERVABILITY
|
||||
# ============================================================================
|
||||
|
||||
# HAProxy Stats (available at http://localhost:8404/stats)
|
||||
# Metrics include:
|
||||
# - Request rate, error rate, latency
|
||||
# - Active connections per backend
|
||||
# - Health check status
|
||||
# - Session rate
|
||||
|
||||
# Prometheus Exporter:
|
||||
# Use haproxy_exporter: https://github.com/prometheus/haproxy_exporter
|
||||
# docker run -d -p 9101:9101 prom/haproxy-exporter:latest \
|
||||
# --haproxy.scrape-uri="http://admin:foxhunt123@localhost:8404/stats;csv"
|
||||
|
||||
# Example Prometheus scrape config:
|
||||
# scrape_configs:
|
||||
# - job_name: 'haproxy'
|
||||
# static_configs:
|
||||
# - targets: ['haproxy:9101']
|
||||
|
||||
# ============================================================================
|
||||
# NOTES
|
||||
# ============================================================================
|
||||
|
||||
# 1. SSL Certificate:
|
||||
# - Combine certificate and key into single PEM file:
|
||||
# cat foxhunt.crt foxhunt.key > /etc/haproxy/certs/foxhunt.pem
|
||||
# - Ensure correct permissions: chmod 600 /etc/haproxy/certs/foxhunt.pem
|
||||
#
|
||||
# 2. Health Checks:
|
||||
# - Uses gRPC Health Checking Protocol (standard)
|
||||
# - Check interval: 5 seconds
|
||||
# - Rise: 2 successful checks = healthy
|
||||
# - Fall: 3 failed checks = unhealthy
|
||||
# - Backends must implement grpc.health.v1.Health service
|
||||
#
|
||||
# 3. Rate Limiting:
|
||||
# - Stick tables track request rate per IP
|
||||
# - Adjust limits based on traffic patterns
|
||||
# - Monitor 429 error rate to avoid over-throttling
|
||||
#
|
||||
# 4. Load Balancing Algorithm:
|
||||
# - leastconn: Best for gRPC (long-lived connections)
|
||||
# - Alternative: roundrobin (simpler, less accurate)
|
||||
# - Alternative: source (session affinity by IP)
|
||||
#
|
||||
# 5. Connection Reuse:
|
||||
# - http-reuse always: Reuse connections to backends (critical for gRPC)
|
||||
# - Reduces latency and connection overhead
|
||||
#
|
||||
# 6. Logging:
|
||||
# - Logs sent to syslog (/dev/log)
|
||||
# - Configure rsyslog to route HAProxy logs to file
|
||||
# - Example: /var/log/haproxy.log
|
||||
|
||||
# ============================================================================
|
||||
# PRODUCTION CHECKLIST
|
||||
# ============================================================================
|
||||
|
||||
# [ ] Replace SSL certificate path (/etc/haproxy/certs/foxhunt.pem)
|
||||
# [ ] Change stats page credentials (admin:foxhunt123)
|
||||
# [ ] Configure DNS for foxhunt.trading
|
||||
# [ ] Set up log rotation (logrotate)
|
||||
# [ ] Enable Prometheus exporter (haproxy_exporter)
|
||||
# [ ] Configure alerts (high error rate, backend down)
|
||||
# [ ] Test rate limiting with load testing tools
|
||||
# [ ] Configure firewall rules (allow 80, 443, 8404; deny others)
|
||||
# [ ] Document backend server IP addresses
|
||||
# [ ] Test failover (kill backend, verify traffic reroutes)
|
||||
# [ ] Set up automated certificate renewal (certbot cron)
|
||||
|
||||
# ============================================================================
|
||||
# HAPROXY MANAGEMENT COMMANDS
|
||||
# ============================================================================
|
||||
|
||||
# Start HAProxy:
|
||||
# haproxy -f /etc/haproxy/haproxy.cfg
|
||||
#
|
||||
# Check configuration:
|
||||
# haproxy -c -f /etc/haproxy/haproxy.cfg
|
||||
#
|
||||
# Reload configuration (zero-downtime):
|
||||
# haproxy -f /etc/haproxy/haproxy.cfg -sf $(cat /var/run/haproxy.pid)
|
||||
#
|
||||
# View stats:
|
||||
# http://localhost:8404/stats
|
||||
#
|
||||
# Admin commands (via stats page):
|
||||
# - Drain server: Set server to "drain" mode (no new connections)
|
||||
# - Disable server: Take server out of rotation
|
||||
# - Enable server: Put server back into rotation
|
||||
#
|
||||
# Socket commands (requires stats socket):
|
||||
# echo "show stat" | socat stdio /var/run/haproxy.sock
|
||||
# echo "show servers state" | socat stdio /var/run/haproxy.sock
|
||||
# echo "disable server api_gateway_backend/api-gateway-1" | socat stdio /var/run/haproxy.sock
|
||||
464
config/k8s/hpa.yaml
Normal file
464
config/k8s/hpa.yaml
Normal file
@@ -0,0 +1,464 @@
|
||||
# Kubernetes Horizontal Pod Autoscaler (HPA) Manifests
|
||||
# Version: 1.0
|
||||
# Last Updated: 2025-10-07
|
||||
#
|
||||
# These HPA configurations provide:
|
||||
# - CPU-based autoscaling for all services
|
||||
# - Custom metrics for advanced scaling (request rate, queue depth)
|
||||
# - Configurable scale-up/scale-down behavior
|
||||
# - Min/max replica counts for cost optimization
|
||||
|
||||
---
|
||||
# ============================================================================
|
||||
# API GATEWAY HPA (Latency-Sensitive, Aggressive Scaling)
|
||||
# ============================================================================
|
||||
|
||||
apiVersion: autoscaling/v2
|
||||
kind: HorizontalPodAutoscaler
|
||||
metadata:
|
||||
name: api-gateway-hpa
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app: api-gateway
|
||||
component: gateway
|
||||
spec:
|
||||
scaleTargetRef:
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: api-gateway
|
||||
|
||||
# Replica limits
|
||||
minReplicas: 3 # Always maintain 3 for HA (survive 1 node failure + 1 rolling update)
|
||||
maxReplicas: 20 # Max capacity: 20 instances × 2K req/s = 40K req/s
|
||||
|
||||
# Scaling metrics
|
||||
metrics:
|
||||
# CPU-based scaling (primary metric)
|
||||
- type: Resource
|
||||
resource:
|
||||
name: cpu
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: 70 # Scale when CPU > 70%
|
||||
|
||||
# Memory-based scaling (secondary metric)
|
||||
- type: Resource
|
||||
resource:
|
||||
name: memory
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: 80 # Scale when memory > 80%
|
||||
|
||||
# Custom metric: Request rate (requires Prometheus adapter)
|
||||
- type: Pods
|
||||
pods:
|
||||
metric:
|
||||
name: grpc_requests_per_second
|
||||
target:
|
||||
type: AverageValue
|
||||
averageValue: "2000" # Scale when > 2000 req/s per pod
|
||||
|
||||
# Scaling behavior (aggressive scale-up, slow scale-down)
|
||||
behavior:
|
||||
scaleUp:
|
||||
stabilizationWindowSeconds: 30 # Wait 30s before scaling up
|
||||
policies:
|
||||
- type: Percent
|
||||
value: 100 # Double replicas
|
||||
periodSeconds: 15
|
||||
- type: Pods
|
||||
value: 4 # Or add 4 pods
|
||||
periodSeconds: 15
|
||||
selectPolicy: Max # Use whichever scales faster
|
||||
|
||||
scaleDown:
|
||||
stabilizationWindowSeconds: 300 # Wait 5 minutes before scaling down
|
||||
policies:
|
||||
- type: Percent
|
||||
value: 25 # Remove 25% of replicas
|
||||
periodSeconds: 60
|
||||
- type: Pods
|
||||
value: 1 # Or remove 1 pod
|
||||
periodSeconds: 60
|
||||
selectPolicy: Min # Use whichever scales slower
|
||||
|
||||
---
|
||||
# ============================================================================
|
||||
# TRADING SERVICE HPA (Latency-Sensitive, Aggressive Scaling)
|
||||
# ============================================================================
|
||||
|
||||
apiVersion: autoscaling/v2
|
||||
kind: HorizontalPodAutoscaler
|
||||
metadata:
|
||||
name: trading-service-hpa
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app: trading-service
|
||||
component: backend
|
||||
spec:
|
||||
scaleTargetRef:
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: trading-service
|
||||
|
||||
# Replica limits
|
||||
minReplicas: 3 # HA requirement
|
||||
maxReplicas: 20 # Based on performance testing: 50K orders/sec ÷ 2.5K orders/sec per pod
|
||||
|
||||
# Scaling metrics
|
||||
metrics:
|
||||
# CPU-based scaling
|
||||
- type: Resource
|
||||
resource:
|
||||
name: cpu
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: 70
|
||||
|
||||
# Memory-based scaling
|
||||
- type: Resource
|
||||
resource:
|
||||
name: memory
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: 80
|
||||
|
||||
# Custom metric: Order submission rate
|
||||
- type: Pods
|
||||
pods:
|
||||
metric:
|
||||
name: order_submissions_per_second
|
||||
target:
|
||||
type: AverageValue
|
||||
averageValue: "2500" # Scale when > 2500 orders/s per pod
|
||||
|
||||
# Scaling behavior (aggressive scale-up)
|
||||
behavior:
|
||||
scaleUp:
|
||||
stabilizationWindowSeconds: 30
|
||||
policies:
|
||||
- type: Percent
|
||||
value: 100
|
||||
periodSeconds: 15
|
||||
- type: Pods
|
||||
value: 4
|
||||
periodSeconds: 15
|
||||
selectPolicy: Max
|
||||
|
||||
scaleDown:
|
||||
stabilizationWindowSeconds: 300
|
||||
policies:
|
||||
- type: Percent
|
||||
value: 25
|
||||
periodSeconds: 60
|
||||
- type: Pods
|
||||
value: 1
|
||||
periodSeconds: 60
|
||||
selectPolicy: Min
|
||||
|
||||
---
|
||||
# ============================================================================
|
||||
# BACKTESTING SERVICE HPA (Job Queue Pattern, Conservative Scaling)
|
||||
# ============================================================================
|
||||
|
||||
apiVersion: autoscaling/v2
|
||||
kind: HorizontalPodAutoscaler
|
||||
metadata:
|
||||
name: backtesting-service-hpa
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app: backtesting-service
|
||||
component: backend
|
||||
spec:
|
||||
scaleTargetRef:
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: backtesting-service
|
||||
|
||||
# Replica limits
|
||||
minReplicas: 2 # Minimum workers for job queue
|
||||
maxReplicas: 10 # Rarely need more than 10 concurrent backtests
|
||||
|
||||
# Scaling metrics
|
||||
metrics:
|
||||
# CPU-based scaling (primary for CPU-intensive backtests)
|
||||
- type: Resource
|
||||
resource:
|
||||
name: cpu
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: 75 # Higher threshold for batch workload
|
||||
|
||||
# Memory-based scaling
|
||||
- type: Resource
|
||||
resource:
|
||||
name: memory
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: 80
|
||||
|
||||
# Custom metric: Redis job queue depth
|
||||
- type: Pods
|
||||
pods:
|
||||
metric:
|
||||
name: redis_queue_length_backtesting
|
||||
target:
|
||||
type: AverageValue
|
||||
averageValue: "5" # Scale when > 5 jobs per worker
|
||||
|
||||
# Scaling behavior (conservative scale-up, slower scale-down)
|
||||
behavior:
|
||||
scaleUp:
|
||||
stabilizationWindowSeconds: 60 # Wait 60s before scaling up
|
||||
policies:
|
||||
- type: Percent
|
||||
value: 50 # Add 50% more replicas
|
||||
periodSeconds: 60
|
||||
- type: Pods
|
||||
value: 2 # Or add 2 pods
|
||||
periodSeconds: 60
|
||||
selectPolicy: Min # Use whichever scales slower
|
||||
|
||||
scaleDown:
|
||||
stabilizationWindowSeconds: 600 # Wait 10 minutes before scaling down
|
||||
policies:
|
||||
- type: Percent
|
||||
value: 25
|
||||
periodSeconds: 120
|
||||
- type: Pods
|
||||
value: 1
|
||||
periodSeconds: 120
|
||||
selectPolicy: Min
|
||||
|
||||
---
|
||||
# ============================================================================
|
||||
# ML TRAINING SERVICE HPA (MANUAL SCALING RECOMMENDED)
|
||||
# ============================================================================
|
||||
|
||||
# NOTE: ML Training Service uses GPU resources, which are expensive and slow to provision.
|
||||
# Recommendation: Use manual scaling (kubectl scale) or scheduled scaling, NOT auto-scaling.
|
||||
#
|
||||
# If auto-scaling is required, use job queue pattern with custom metrics:
|
||||
|
||||
apiVersion: autoscaling/v2
|
||||
kind: HorizontalPodAutoscaler
|
||||
metadata:
|
||||
name: ml-training-service-hpa
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app: ml-training-service
|
||||
component: backend
|
||||
spec:
|
||||
scaleTargetRef:
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: ml-training-service
|
||||
|
||||
# Replica limits (GPU resources expensive)
|
||||
minReplicas: 1 # Single instance (GPU idle time acceptable)
|
||||
maxReplicas: 3 # Limited by GPU cluster size
|
||||
|
||||
# Scaling metrics
|
||||
metrics:
|
||||
# GPU utilization (requires NVIDIA DCGM exporter)
|
||||
- type: Pods
|
||||
pods:
|
||||
metric:
|
||||
name: dcgm_gpu_utilization
|
||||
target:
|
||||
type: AverageValue
|
||||
averageValue: "80" # Scale when GPU > 80% utilized
|
||||
|
||||
# Custom metric: Training job queue depth
|
||||
- type: Pods
|
||||
pods:
|
||||
metric:
|
||||
name: redis_queue_length_ml_training
|
||||
target:
|
||||
type: AverageValue
|
||||
averageValue: "2" # Scale when > 2 training jobs queued
|
||||
|
||||
# Scaling behavior (very conservative, GPU instances expensive)
|
||||
behavior:
|
||||
scaleUp:
|
||||
stabilizationWindowSeconds: 300 # Wait 5 minutes before scaling up
|
||||
policies:
|
||||
- type: Pods
|
||||
value: 1 # Add 1 pod at a time
|
||||
periodSeconds: 300
|
||||
selectPolicy: Min
|
||||
|
||||
scaleDown:
|
||||
stabilizationWindowSeconds: 1800 # Wait 30 minutes before scaling down
|
||||
policies:
|
||||
- type: Pods
|
||||
value: 1
|
||||
periodSeconds: 300
|
||||
selectPolicy: Min
|
||||
|
||||
---
|
||||
# ============================================================================
|
||||
# CUSTOM METRICS (Prometheus Adapter Configuration)
|
||||
# ============================================================================
|
||||
|
||||
# To use custom metrics (grpc_requests_per_second, redis_queue_length, etc.),
|
||||
# you must install and configure Prometheus Adapter:
|
||||
#
|
||||
# 1. Install Prometheus Adapter:
|
||||
# helm install prometheus-adapter prometheus-community/prometheus-adapter \
|
||||
# --namespace monitoring \
|
||||
# --values prometheus-adapter-values.yaml
|
||||
#
|
||||
# 2. Configure custom metrics (prometheus-adapter-values.yaml):
|
||||
|
||||
# prometheus:
|
||||
# url: http://prometheus-server.monitoring.svc
|
||||
# port: 9090
|
||||
#
|
||||
# rules:
|
||||
# default: false
|
||||
# custom:
|
||||
# # gRPC request rate per pod
|
||||
# - seriesQuery: 'grpc_requests_total{namespace="foxhunt"}'
|
||||
# resources:
|
||||
# overrides:
|
||||
# namespace: {resource: "namespace"}
|
||||
# pod: {resource: "pod"}
|
||||
# name:
|
||||
# matches: "^grpc_requests_total"
|
||||
# as: "grpc_requests_per_second"
|
||||
# metricsQuery: 'sum(rate(grpc_requests_total{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)'
|
||||
#
|
||||
# # Redis queue length
|
||||
# - seriesQuery: 'redis_list_length{namespace="foxhunt",queue="backtesting"}'
|
||||
# resources:
|
||||
# overrides:
|
||||
# namespace: {resource: "namespace"}
|
||||
# name:
|
||||
# matches: "^redis_list_length"
|
||||
# as: "redis_queue_length_backtesting"
|
||||
# metricsQuery: 'avg(redis_list_length{<<.LabelMatchers>>}) by (<<.GroupBy>>)'
|
||||
#
|
||||
# # GPU utilization (requires NVIDIA DCGM exporter)
|
||||
# - seriesQuery: 'DCGM_FI_DEV_GPU_UTIL{namespace="foxhunt"}'
|
||||
# resources:
|
||||
# overrides:
|
||||
# namespace: {resource: "namespace"}
|
||||
# pod: {resource: "pod"}
|
||||
# name:
|
||||
# matches: "^DCGM_FI_DEV_GPU_UTIL"
|
||||
# as: "dcgm_gpu_utilization"
|
||||
# metricsQuery: 'avg(DCGM_FI_DEV_GPU_UTIL{<<.LabelMatchers>>}) by (<<.GroupBy>>)'
|
||||
|
||||
# 3. Verify custom metrics are available:
|
||||
# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .
|
||||
|
||||
# 4. Test HPA with custom metrics:
|
||||
# kubectl get hpa api-gateway-hpa -o yaml
|
||||
# # Check "currentMetrics" section for custom metric values
|
||||
|
||||
---
|
||||
# ============================================================================
|
||||
# TESTING & VALIDATION
|
||||
# ============================================================================
|
||||
|
||||
# 1. Deploy HPA manifests:
|
||||
# kubectl apply -f hpa.yaml
|
||||
#
|
||||
# 2. Verify HPA status:
|
||||
# kubectl get hpa -n foxhunt
|
||||
# kubectl describe hpa api-gateway-hpa -n foxhunt
|
||||
#
|
||||
# 3. Monitor scaling events:
|
||||
# kubectl get events -n foxhunt --field-selector involvedObject.kind=HorizontalPodAutoscaler
|
||||
#
|
||||
# 4. Load test to trigger scaling:
|
||||
# # Generate load (e.g., with ghz)
|
||||
# ghz --insecure \
|
||||
# --proto api_gateway.proto \
|
||||
# --call foxhunt.api_gateway.ApiGateway/SubmitOrder \
|
||||
# --rps 10000 \
|
||||
# --duration 5m \
|
||||
# localhost:50051
|
||||
#
|
||||
# # Watch HPA scale up
|
||||
# watch kubectl get hpa -n foxhunt
|
||||
#
|
||||
# 5. Verify scale-down after load stops:
|
||||
# # Wait 5-10 minutes after load test ends
|
||||
# kubectl get hpa api-gateway-hpa -n foxhunt
|
||||
# # Should scale down to minReplicas
|
||||
|
||||
---
|
||||
# ============================================================================
|
||||
# TROUBLESHOOTING
|
||||
# ============================================================================
|
||||
|
||||
# Issue: HPA shows "unknown" for metrics
|
||||
# Solution: Ensure metrics-server is installed and running
|
||||
# kubectl get deployment metrics-server -n kube-system
|
||||
# kubectl logs -n kube-system deployment/metrics-server
|
||||
#
|
||||
# Issue: Custom metrics not available
|
||||
# Solution: Verify Prometheus Adapter configuration
|
||||
# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .
|
||||
# kubectl logs -n monitoring deployment/prometheus-adapter
|
||||
#
|
||||
# Issue: HPA not scaling up despite high CPU
|
||||
# Solution: Check stabilizationWindowSeconds (may be waiting)
|
||||
# kubectl describe hpa <name> -n foxhunt
|
||||
# # Look for "ScalingLimited" or "BackoffBoth" events
|
||||
#
|
||||
# Issue: HPA scaling too aggressively (flapping)
|
||||
# Solution: Increase stabilizationWindowSeconds or reduce scale-up policies
|
||||
#
|
||||
# Issue: Pods evicted due to resource pressure
|
||||
# Solution: Increase resource requests/limits in Deployment spec
|
||||
# kubectl describe pod <name> -n foxhunt
|
||||
# # Look for "Evicted" status
|
||||
|
||||
---
|
||||
# ============================================================================
|
||||
# PRODUCTION BEST PRACTICES
|
||||
# ============================================================================
|
||||
|
||||
# 1. Monitor HPA metrics:
|
||||
# - Current replicas vs desired replicas
|
||||
# - Scaling events (up/down)
|
||||
# - Metric values (CPU, memory, custom)
|
||||
#
|
||||
# 2. Alert on scaling issues:
|
||||
# - HPA unable to scale (resource quotas exceeded)
|
||||
# - HPA scaling too frequently (flapping)
|
||||
# - Pods in CrashLoopBackOff after scale-up
|
||||
#
|
||||
# 3. Test scaling under load:
|
||||
# - Run load tests regularly (weekly/monthly)
|
||||
# - Verify scale-up happens within SLA (< 2 minutes)
|
||||
# - Verify scale-down doesn't cause service disruption
|
||||
#
|
||||
# 4. Cost optimization:
|
||||
# - Use Reserved Instances for minReplicas (50% savings)
|
||||
# - Use Spot Instances for auto-scaled replicas (70% savings, non-critical workloads)
|
||||
# - Monitor auto-scaling costs (CloudWatch/Prometheus)
|
||||
#
|
||||
# 5. Resource limits:
|
||||
# - Set resource requests = expected usage
|
||||
# - Set resource limits = 2x requests (headroom for bursts)
|
||||
# - Monitor actual usage and adjust
|
||||
|
||||
---
|
||||
# ============================================================================
|
||||
# PRODUCTION CHECKLIST
|
||||
# ============================================================================
|
||||
|
||||
# [ ] Install metrics-server (for CPU/memory metrics)
|
||||
# [ ] Install Prometheus Adapter (for custom metrics)
|
||||
# [ ] Configure custom metrics (request rate, queue depth, GPU util)
|
||||
# [ ] Deploy HPA manifests (kubectl apply -f hpa.yaml)
|
||||
# [ ] Verify HPA status (kubectl get hpa)
|
||||
# [ ] Run load test to validate scaling behavior
|
||||
# [ ] Configure Grafana dashboards (HPA metrics, scaling events)
|
||||
# [ ] Configure alerts (scaling issues, resource pressure)
|
||||
# [ ] Document scaling thresholds and replica limits
|
||||
# [ ] Review and adjust HPA configuration quarterly
|
||||
331
config/nginx-lb.conf
Normal file
331
config/nginx-lb.conf
Normal file
@@ -0,0 +1,331 @@
|
||||
# nginx Load Balancer Configuration for Foxhunt HFT Trading System
|
||||
# Version: 1.0
|
||||
# Last Updated: 2025-10-07
|
||||
#
|
||||
# This configuration provides:
|
||||
# - Layer 7 (L7) gRPC load balancing
|
||||
# - TLS termination at load balancer
|
||||
# - Health checks using gRPC Health Checking Protocol
|
||||
# - Rate limiting by user tier
|
||||
# - DDoS protection
|
||||
# - Connection limits
|
||||
|
||||
# ============================================================================
|
||||
# UPSTREAM DEFINITIONS
|
||||
# ============================================================================
|
||||
|
||||
# API Gateway cluster (primary entry point for all clients)
|
||||
upstream api_gateway_backend {
|
||||
# Load balancing algorithm
|
||||
# - least_conn: Route to backend with fewest active connections (recommended for long-lived gRPC connections)
|
||||
# - round_robin: Default, simple rotation
|
||||
# - ip_hash: Session affinity based on client IP (use for stateful operations)
|
||||
least_conn;
|
||||
|
||||
# Backend servers
|
||||
# Format: server <host>:<port> [parameters];
|
||||
server api-gateway-1:50051 max_fails=3 fail_timeout=30s;
|
||||
server api-gateway-2:50051 max_fails=3 fail_timeout=30s;
|
||||
server api-gateway-3:50051 max_fails=3 fail_timeout=30s;
|
||||
|
||||
# Health check (requires nginx Plus or nginx-module-health-check)
|
||||
# For open-source nginx, use passive health checks (max_fails)
|
||||
# For nginx Plus:
|
||||
# health_check interval=5s fails=2 passes=1 uri=/grpc.health.v1.Health/Check;
|
||||
|
||||
# Keepalive connections to backend
|
||||
# Reuse connections instead of creating new ones (critical for gRPC performance)
|
||||
keepalive 100;
|
||||
keepalive_timeout 60s;
|
||||
}
|
||||
|
||||
# Trading Service cluster (internal, routed via API Gateway)
|
||||
upstream trading_service_backend {
|
||||
least_conn;
|
||||
server trading-service-1:50052 max_fails=3 fail_timeout=30s;
|
||||
server trading-service-2:50052 max_fails=3 fail_timeout=30s;
|
||||
server trading-service-3:50052 max_fails=3 fail_timeout=30s;
|
||||
keepalive 100;
|
||||
}
|
||||
|
||||
# Backtesting Service cluster (worker pool pattern)
|
||||
upstream backtesting_service_backend {
|
||||
least_conn;
|
||||
server backtesting-service-1:50053 max_fails=3 fail_timeout=30s;
|
||||
server backtesting-service-2:50053 max_fails=3 fail_timeout=30s;
|
||||
keepalive 50;
|
||||
}
|
||||
|
||||
# ML Training Service cluster (GPU instances, fewer replicas)
|
||||
upstream ml_training_service_backend {
|
||||
least_conn;
|
||||
server ml-training-service-1:50054 max_fails=3 fail_timeout=30s;
|
||||
server ml-training-service-2:50054 max_fails=3 fail_timeout=30s;
|
||||
keepalive 20;
|
||||
}
|
||||
|
||||
# ============================================================================
|
||||
# RATE LIMITING ZONES
|
||||
# ============================================================================
|
||||
|
||||
# Rate limiting by user tier (extracted from JWT)
|
||||
# Zone size: 10m = 10 MB = ~160,000 IP addresses
|
||||
|
||||
# Free tier: 100 requests/second
|
||||
limit_req_zone $jwt_user_tier zone=tier_free:10m rate=100r/s;
|
||||
|
||||
# Premium tier: 500 requests/second
|
||||
limit_req_zone $jwt_user_tier zone=tier_premium:10m rate=500r/s;
|
||||
|
||||
# Enterprise tier: 2000 requests/second
|
||||
limit_req_zone $jwt_user_tier zone=tier_enterprise:10m rate=2000r/s;
|
||||
|
||||
# Internal services: No rate limit
|
||||
# (Identified by internal JWT or mTLS certificate)
|
||||
|
||||
# Connection limits per IP (DDoS protection)
|
||||
limit_conn_zone $binary_remote_addr zone=addr:10m;
|
||||
|
||||
# ============================================================================
|
||||
# MAP DEFINITIONS
|
||||
# ============================================================================
|
||||
|
||||
# Extract user tier from JWT (requires lua-nginx-module or custom logic)
|
||||
# For simplicity, this example assumes tier is in a custom header
|
||||
# In production, extract from JWT claims
|
||||
map $http_x_user_tier $rate_limit_zone {
|
||||
default tier_free;
|
||||
"free" tier_free;
|
||||
"premium" tier_premium;
|
||||
"enterprise" tier_enterprise;
|
||||
"internal" ""; # No rate limit for internal services
|
||||
}
|
||||
|
||||
# ============================================================================
|
||||
# SERVER BLOCKS
|
||||
# ============================================================================
|
||||
|
||||
# HTTP server (redirect to HTTPS)
|
||||
server {
|
||||
listen 80;
|
||||
listen [::]:80;
|
||||
server_name foxhunt.trading;
|
||||
|
||||
# Redirect all HTTP traffic to HTTPS
|
||||
return 301 https://$server_name$request_uri;
|
||||
}
|
||||
|
||||
# HTTPS server (TLS termination + gRPC load balancing)
|
||||
server {
|
||||
# Listen on port 443 with HTTP/2 (required for gRPC)
|
||||
listen 443 ssl http2;
|
||||
listen [::]:443 ssl http2;
|
||||
|
||||
server_name foxhunt.trading;
|
||||
|
||||
# ========================================================================
|
||||
# TLS CONFIGURATION
|
||||
# ========================================================================
|
||||
|
||||
# SSL certificate (replace with your actual certificate paths)
|
||||
ssl_certificate /etc/ssl/certs/foxhunt.crt;
|
||||
ssl_certificate_key /etc/ssl/private/foxhunt.key;
|
||||
|
||||
# TLS protocols (TLS 1.2 and 1.3 only, no SSLv3/TLS 1.0/1.1)
|
||||
ssl_protocols TLSv1.2 TLSv1.3;
|
||||
|
||||
# Cipher suites (prefer modern, secure ciphers)
|
||||
ssl_ciphers HIGH:!aNULL:!MD5:!3DES;
|
||||
ssl_prefer_server_ciphers on;
|
||||
|
||||
# SSL session cache (improves performance by reusing TLS sessions)
|
||||
ssl_session_cache shared:SSL:10m;
|
||||
ssl_session_timeout 10m;
|
||||
|
||||
# OCSP stapling (improves TLS handshake performance)
|
||||
ssl_stapling on;
|
||||
ssl_stapling_verify on;
|
||||
ssl_trusted_certificate /etc/ssl/certs/ca-bundle.crt;
|
||||
|
||||
# HSTS (HTTP Strict Transport Security)
|
||||
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
|
||||
|
||||
# ========================================================================
|
||||
# GRPC CONFIGURATION
|
||||
# ========================================================================
|
||||
|
||||
# gRPC-specific settings
|
||||
grpc_read_timeout 300s; # 5 minutes (for long-running backtests)
|
||||
grpc_send_timeout 30s;
|
||||
grpc_connect_timeout 10s;
|
||||
|
||||
# Buffer sizes (tune based on payload size)
|
||||
client_body_buffer_size 1M;
|
||||
client_max_body_size 10M;
|
||||
|
||||
# HTTP/2 settings
|
||||
http2_max_concurrent_streams 128;
|
||||
http2_recv_timeout 300s;
|
||||
|
||||
# ========================================================================
|
||||
# RATE LIMITING & DDOS PROTECTION
|
||||
# ========================================================================
|
||||
|
||||
# Connection limit per IP (max 10 concurrent connections)
|
||||
limit_conn addr 10;
|
||||
|
||||
# Rate limiting by user tier (applied per location)
|
||||
# See location blocks below
|
||||
|
||||
# ========================================================================
|
||||
# LOGGING
|
||||
# ========================================================================
|
||||
|
||||
# Access log (includes gRPC status)
|
||||
access_log /var/log/nginx/access.log combined;
|
||||
|
||||
# Error log
|
||||
error_log /var/log/nginx/error.log warn;
|
||||
|
||||
# ========================================================================
|
||||
# LOCATIONS (ROUTING)
|
||||
# ========================================================================
|
||||
|
||||
# API Gateway (primary entry point for clients)
|
||||
location /foxhunt.api_gateway {
|
||||
# Apply rate limiting based on user tier
|
||||
limit_req zone=$rate_limit_zone burst=20 nodelay;
|
||||
|
||||
# gRPC proxy to backend
|
||||
grpc_pass grpc://api_gateway_backend;
|
||||
|
||||
# Pass original client IP (for logging and rate limiting)
|
||||
grpc_set_header X-Real-IP $remote_addr;
|
||||
grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
grpc_set_header X-Forwarded-Proto $scheme;
|
||||
|
||||
# Error handling
|
||||
grpc_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
|
||||
grpc_next_upstream_tries 3;
|
||||
grpc_next_upstream_timeout 10s;
|
||||
}
|
||||
|
||||
# Trading Service (internal, typically accessed via API Gateway)
|
||||
# Exposed for direct access if needed (e.g., for monitoring)
|
||||
location /foxhunt.trading_service {
|
||||
# Restrict to internal IPs only
|
||||
allow 10.0.0.0/8; # Internal network
|
||||
allow 172.16.0.0/12; # Docker network
|
||||
deny all;
|
||||
|
||||
grpc_pass grpc://trading_service_backend;
|
||||
grpc_set_header X-Real-IP $remote_addr;
|
||||
grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
}
|
||||
|
||||
# Backtesting Service (internal)
|
||||
location /foxhunt.backtesting_service {
|
||||
allow 10.0.0.0/8;
|
||||
allow 172.16.0.0/12;
|
||||
deny all;
|
||||
|
||||
grpc_pass grpc://backtesting_service_backend;
|
||||
grpc_set_header X-Real-IP $remote_addr;
|
||||
grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
}
|
||||
|
||||
# ML Training Service (internal)
|
||||
location /foxhunt.ml_training_service {
|
||||
allow 10.0.0.0/8;
|
||||
allow 172.16.0.0/12;
|
||||
deny all;
|
||||
|
||||
grpc_pass grpc://ml_training_service_backend;
|
||||
grpc_set_header X-Real-IP $remote_addr;
|
||||
grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
}
|
||||
|
||||
# Health check endpoint (HTTP, not gRPC)
|
||||
location /health {
|
||||
access_log off;
|
||||
return 200 "healthy\n";
|
||||
add_header Content-Type text/plain;
|
||||
}
|
||||
|
||||
# Metrics endpoint (for Prometheus scraping)
|
||||
location /metrics {
|
||||
# Restrict to monitoring IPs only
|
||||
allow 10.0.0.0/8;
|
||||
deny all;
|
||||
|
||||
# Stub status (requires --with-http_stub_status_module)
|
||||
stub_status;
|
||||
}
|
||||
|
||||
# Deny all other requests
|
||||
location / {
|
||||
return 404;
|
||||
}
|
||||
}
|
||||
|
||||
# ============================================================================
|
||||
# MONITORING & OBSERVABILITY
|
||||
# ============================================================================
|
||||
|
||||
# Prometheus metrics exporter (requires nginx-prometheus-exporter)
|
||||
# Run separately as a sidecar container:
|
||||
# docker run -p 9113:9113 nginx/nginx-prometheus-exporter:latest \
|
||||
# --nginx.scrape-uri=http://localhost:8080/metrics
|
||||
|
||||
# Example Prometheus scrape config:
|
||||
# scrape_configs:
|
||||
# - job_name: 'nginx-lb'
|
||||
# static_configs:
|
||||
# - targets: ['nginx-lb:9113']
|
||||
|
||||
# ============================================================================
|
||||
# NOTES
|
||||
# ============================================================================
|
||||
|
||||
# 1. Replace SSL certificate paths with your actual certificates:
|
||||
# - Production: Use Let's Encrypt (certbot) or corporate CA
|
||||
# - Development: Self-signed certificates (not recommended for production)
|
||||
#
|
||||
# 2. Adjust rate limits based on your traffic patterns:
|
||||
# - Monitor 429 (Too Many Requests) error rate in Prometheus
|
||||
# - Increase limits if legitimate users are throttled
|
||||
#
|
||||
# 3. Health checks:
|
||||
# - Open-source nginx: Uses passive health checks (max_fails)
|
||||
# - nginx Plus: Active health checks with /grpc.health.v1.Health/Check
|
||||
# - Alternative: Use external health checker (e.g., Kubernetes liveness probes)
|
||||
#
|
||||
# 4. Load balancing algorithms:
|
||||
# - least_conn: Best for gRPC (long-lived connections)
|
||||
# - round_robin: Simpler, but can lead to uneven load
|
||||
# - ip_hash: Use only if session affinity is required
|
||||
#
|
||||
# 5. Monitoring:
|
||||
# - Use nginx-prometheus-exporter for metrics
|
||||
# - Monitor: request rate, error rate, latency, active connections
|
||||
# - Alerts: high error rate (> 1%), high latency (P99 > 100ms)
|
||||
#
|
||||
# 6. Scaling:
|
||||
# - Add more backend servers to upstream blocks
|
||||
# - Reload nginx config: nginx -s reload (zero-downtime)
|
||||
# - For Kubernetes: Use Ingress controller (nginx-ingress)
|
||||
|
||||
# ============================================================================
|
||||
# PRODUCTION CHECKLIST
|
||||
# ============================================================================
|
||||
|
||||
# [ ] Replace SSL certificates with production certificates
|
||||
# [ ] Configure DNS for foxhunt.trading
|
||||
# [ ] Set up log rotation (logrotate)
|
||||
# [ ] Enable Prometheus metrics exporter
|
||||
# [ ] Configure alerts (high error rate, high latency)
|
||||
# [ ] Test rate limiting with load testing tools
|
||||
# [ ] Configure firewall rules (allow 80, 443; deny others)
|
||||
# [ ] Enable nginx status page (for debugging)
|
||||
# [ ] Document backend server IP addresses
|
||||
# [ ] Set up automated certificate renewal (certbot cron)
|
||||
218
run_smoke_tests.sh
Executable file
218
run_smoke_tests.sh
Executable file
@@ -0,0 +1,218 @@
|
||||
#!/bin/bash
|
||||
# Foxhunt HFT System - Automated Smoke Test Suite
|
||||
# Version: 1.0.0
|
||||
# Purpose: Validate complete system functionality after deployment
|
||||
#
|
||||
# Usage:
|
||||
# ./run_smoke_tests.sh # Run all smoke tests
|
||||
# ./run_smoke_tests.sh --fast # Run only critical tests
|
||||
# ./run_smoke_tests.sh --verbose # Run with detailed logging
|
||||
# ./run_smoke_tests.sh --category infrastructure # Run specific category
|
||||
|
||||
set -e
|
||||
|
||||
# Colors for output
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
BLUE='\033[0;34m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
# Configuration
|
||||
VERBOSE=false
|
||||
FAST_MODE=false
|
||||
CATEGORY=""
|
||||
RUST_LOG="${RUST_LOG:-info}"
|
||||
|
||||
# Parse command line arguments
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case $1 in
|
||||
--verbose|-v)
|
||||
VERBOSE=true
|
||||
RUST_LOG="debug"
|
||||
shift
|
||||
;;
|
||||
--fast|-f)
|
||||
FAST_MODE=true
|
||||
shift
|
||||
;;
|
||||
--category|-c)
|
||||
CATEGORY="$2"
|
||||
shift 2
|
||||
;;
|
||||
--help|-h)
|
||||
echo "Foxhunt HFT System - Smoke Test Runner"
|
||||
echo ""
|
||||
echo "Usage: $0 [OPTIONS]"
|
||||
echo ""
|
||||
echo "Options:"
|
||||
echo " --verbose, -v Enable verbose logging (RUST_LOG=debug)"
|
||||
echo " --fast, -f Run only critical tests (faster execution)"
|
||||
echo " --category, -c CAT Run specific category only"
|
||||
echo " Categories: infrastructure, service, authentication, order_flow"
|
||||
echo " --help, -h Show this help message"
|
||||
echo ""
|
||||
echo "Examples:"
|
||||
echo " $0 # Run all smoke tests"
|
||||
echo " $0 --fast # Run only critical tests"
|
||||
echo " $0 --verbose # Run with debug logging"
|
||||
echo " $0 --category infrastructure # Run infrastructure tests only"
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
echo -e "${RED}Unknown option: $1${NC}"
|
||||
echo "Use --help for usage information"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Export environment variables
|
||||
export RUST_LOG
|
||||
export DATABASE_URL="${DATABASE_URL:-postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt}"
|
||||
export REDIS_URL="${REDIS_URL:-redis://localhost:6379}"
|
||||
export VAULT_ADDR="${VAULT_ADDR:-http://localhost:8200}"
|
||||
export INFLUXDB_URL="${INFLUXDB_URL:-http://localhost:8086}"
|
||||
export API_GATEWAY_URL="${API_GATEWAY_URL:-http://localhost:50051}"
|
||||
export TRADING_SERVICE_URL="${TRADING_SERVICE_URL:-http://localhost:50052}"
|
||||
export BACKTESTING_SERVICE_URL="${BACKTESTING_SERVICE_URL:-http://localhost:50053}"
|
||||
export ML_TRAINING_SERVICE_URL="${ML_TRAINING_SERVICE_URL:-http://localhost:50054}"
|
||||
export PROMETHEUS_URL="${PROMETHEUS_URL:-http://localhost:9090}"
|
||||
export GRAFANA_URL="${GRAFANA_URL:-http://localhost:3000}"
|
||||
export JWT_SECRET="${JWT_SECRET:-dev_secret_key_change_in_production}"
|
||||
|
||||
# Print banner
|
||||
echo -e "${BLUE}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
|
||||
echo -e "${BLUE} Foxhunt HFT System - Smoke Test Suite${NC}"
|
||||
echo -e "${BLUE}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
|
||||
echo ""
|
||||
|
||||
# Environment info
|
||||
echo -e "${YELLOW}Environment Configuration:${NC}"
|
||||
echo " Database: ${DATABASE_URL}"
|
||||
echo " Redis: ${REDIS_URL}"
|
||||
echo " Vault: ${VAULT_ADDR}"
|
||||
echo " API Gateway: ${API_GATEWAY_URL}"
|
||||
echo " Trading Service: ${TRADING_SERVICE_URL}"
|
||||
echo " Log Level: ${RUST_LOG}"
|
||||
if [ "$FAST_MODE" = true ]; then
|
||||
echo -e " ${GREEN}Mode: FAST (critical tests only)${NC}"
|
||||
fi
|
||||
if [ "$CATEGORY" != "" ]; then
|
||||
echo -e " ${GREEN}Category: ${CATEGORY}${NC}"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Test execution flags
|
||||
TEST_FLAGS=""
|
||||
if [ "$VERBOSE" = true ]; then
|
||||
TEST_FLAGS="-- --nocapture --test-threads=1"
|
||||
fi
|
||||
|
||||
# Function to run a test category
|
||||
run_test_category() {
|
||||
local category=$1
|
||||
local test_name=$2
|
||||
local description=$3
|
||||
|
||||
echo -e "${BLUE}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
|
||||
echo -e "${BLUE}Testing: ${description}${NC}"
|
||||
echo -e "${BLUE}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
|
||||
echo ""
|
||||
|
||||
if cargo test --test smoke_tests ${test_name} ${TEST_FLAGS}; then
|
||||
echo ""
|
||||
echo -e "${GREEN}✅ ${description} - PASSED${NC}"
|
||||
return 0
|
||||
else
|
||||
echo ""
|
||||
echo -e "${RED}❌ ${description} - FAILED${NC}"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# Track results
|
||||
TOTAL_CATEGORIES=0
|
||||
PASSED_CATEGORIES=0
|
||||
FAILED_CATEGORIES=0
|
||||
|
||||
# Main test execution
|
||||
echo -e "${YELLOW}🔍 Starting smoke test execution...${NC}"
|
||||
echo ""
|
||||
|
||||
# Category 1: Infrastructure Health (CRITICAL)
|
||||
if [ -z "$CATEGORY" ] || [ "$CATEGORY" = "infrastructure" ]; then
|
||||
TOTAL_CATEGORIES=$((TOTAL_CATEGORIES + 1))
|
||||
if run_test_category "infrastructure" "infrastructure_health" "Infrastructure Health"; then
|
||||
PASSED_CATEGORIES=$((PASSED_CATEGORIES + 1))
|
||||
else
|
||||
FAILED_CATEGORIES=$((FAILED_CATEGORIES + 1))
|
||||
fi
|
||||
echo ""
|
||||
fi
|
||||
|
||||
# Category 2: Service Health (CRITICAL)
|
||||
if [ -z "$CATEGORY" ] || [ "$CATEGORY" = "service" ]; then
|
||||
TOTAL_CATEGORIES=$((TOTAL_CATEGORIES + 1))
|
||||
if run_test_category "service" "service_health" "Service Health"; then
|
||||
PASSED_CATEGORIES=$((PASSED_CATEGORIES + 1))
|
||||
else
|
||||
FAILED_CATEGORIES=$((FAILED_CATEGORIES + 1))
|
||||
echo -e "${YELLOW}⚠️ Note: Some services may not be available due to configuration issues${NC}"
|
||||
fi
|
||||
echo ""
|
||||
fi
|
||||
|
||||
# Category 3: Authentication Flow (skip in fast mode)
|
||||
if [ "$FAST_MODE" = false ] && ([ -z "$CATEGORY" ] || [ "$CATEGORY" = "authentication" ]); then
|
||||
TOTAL_CATEGORIES=$((TOTAL_CATEGORIES + 1))
|
||||
if run_test_category "authentication" "authentication_flow" "Authentication Flow"; then
|
||||
PASSED_CATEGORIES=$((PASSED_CATEGORIES + 1))
|
||||
else
|
||||
FAILED_CATEGORIES=$((FAILED_CATEGORIES + 1))
|
||||
fi
|
||||
echo ""
|
||||
fi
|
||||
|
||||
# Category 4: Basic Order Flow (skip in fast mode)
|
||||
if [ "$FAST_MODE" = false ] && ([ -z "$CATEGORY" ] || [ "$CATEGORY" = "order_flow" ]); then
|
||||
TOTAL_CATEGORIES=$((TOTAL_CATEGORIES + 1))
|
||||
if run_test_category "order_flow" "basic_order_flow" "Basic Order Flow"; then
|
||||
PASSED_CATEGORIES=$((PASSED_CATEGORIES + 1))
|
||||
else
|
||||
FAILED_CATEGORIES=$((FAILED_CATEGORIES + 1))
|
||||
fi
|
||||
echo ""
|
||||
fi
|
||||
|
||||
# Print summary
|
||||
echo -e "${BLUE}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
|
||||
echo -e "${BLUE} Smoke Test Summary${NC}"
|
||||
echo -e "${BLUE}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
|
||||
echo ""
|
||||
echo " Total Categories: ${TOTAL_CATEGORIES}"
|
||||
echo -e " ${GREEN}Passed: ${PASSED_CATEGORIES}${NC}"
|
||||
if [ $FAILED_CATEGORIES -gt 0 ]; then
|
||||
echo -e " ${RED}Failed: ${FAILED_CATEGORIES}${NC}"
|
||||
else
|
||||
echo " Failed: ${FAILED_CATEGORIES}"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Calculate percentage
|
||||
if [ $TOTAL_CATEGORIES -gt 0 ]; then
|
||||
PASS_PERCENTAGE=$((PASSED_CATEGORIES * 100 / TOTAL_CATEGORIES))
|
||||
echo " Pass Rate: ${PASS_PERCENTAGE}%"
|
||||
echo ""
|
||||
fi
|
||||
|
||||
# Exit with appropriate code
|
||||
if [ $FAILED_CATEGORIES -eq 0 ]; then
|
||||
echo -e "${GREEN}✅ All smoke tests passed!${NC}"
|
||||
echo -e "${GREEN} System is ready for deployment.${NC}"
|
||||
exit 0
|
||||
else
|
||||
echo -e "${YELLOW}⚠️ Some smoke tests failed${NC}"
|
||||
echo -e "${YELLOW} Review the failures above before deploying.${NC}"
|
||||
exit 1
|
||||
fi
|
||||
@@ -40,6 +40,13 @@ futures.workspace = true
|
||||
|
||||
# gRPC and networking
|
||||
tonic.workspace = true
|
||||
tonic-health = { version = "0.14", optional = true }
|
||||
|
||||
# HTTP client for smoke tests
|
||||
reqwest = { version = "0.12", features = ["json"], optional = true }
|
||||
|
||||
# JWT for authentication tests
|
||||
jsonwebtoken = { version = "9", optional = true }
|
||||
|
||||
# Database and UUID
|
||||
sqlx.workspace = true
|
||||
@@ -101,6 +108,7 @@ memory-profiling = ["dhat", "jemalloc_pprof"]
|
||||
coverage-analysis = []
|
||||
gpu-tests = []
|
||||
integration-tests = ["redis", "influxdb2"] # Removed testcontainers
|
||||
smoke-tests = ["reqwest", "tonic-health", "jsonwebtoken", "redis", "influxdb2"]
|
||||
|
||||
# Performance optimization features
|
||||
simd = []
|
||||
|
||||
18
tests/smoke_tests.rs
Normal file
18
tests/smoke_tests.rs
Normal file
@@ -0,0 +1,18 @@
|
||||
//! Smoke Tests - Integration Test Entry Point
|
||||
//!
|
||||
//! This file serves as the integration test entry point for the smoke test suite.
|
||||
//! Cargo will compile this as a separate test binary that can be run with:
|
||||
//!
|
||||
//! ```bash
|
||||
//! cargo test --test smoke_tests
|
||||
//! cargo test --test smoke_tests --features smoke-tests
|
||||
//! ```
|
||||
|
||||
#[cfg(feature = "smoke-tests")]
|
||||
mod smoke_tests;
|
||||
|
||||
#[cfg(not(feature = "smoke-tests"))]
|
||||
#[test]
|
||||
fn smoke_tests_disabled() {
|
||||
println!("⚠️ Smoke tests are disabled. Enable with: cargo test --features smoke-tests");
|
||||
}
|
||||
359
tests/smoke_tests/README.md
Normal file
359
tests/smoke_tests/README.md
Normal file
@@ -0,0 +1,359 @@
|
||||
# Smoke Tests - End-to-End System Validation
|
||||
|
||||
## Overview
|
||||
|
||||
Automated smoke test suite for validating complete Foxhunt HFT system functionality after deployment. These tests provide quick validation that all critical components are operational.
|
||||
|
||||
## Test Categories
|
||||
|
||||
### 1. Infrastructure Health (`infrastructure_health.rs`)
|
||||
Tests core infrastructure services:
|
||||
- **PostgreSQL**: Connection, schema validation, TimescaleDB extension
|
||||
- **Redis**: Connection, SET/GET operations, key expiration
|
||||
- **Vault**: Connectivity and health status
|
||||
- **InfluxDB**: Ping endpoint and availability
|
||||
- **Prometheus**: Health endpoint and metrics availability
|
||||
- **Grafana**: API health check
|
||||
|
||||
### 2. Service Health (`service_health.rs`)
|
||||
Tests gRPC microservices:
|
||||
- **Trading Service**: HTTP health check, gRPC connection (port 50052)
|
||||
- **API Gateway**: gRPC connection (port 50051)
|
||||
- **Backtesting Service**: gRPC connection (port 50053) - BLOCKED
|
||||
- **ML Training Service**: gRPC connection (port 50054) - BLOCKED
|
||||
- **Service Ports**: Validates all services are listening
|
||||
- **Response Times**: Measures connection latency
|
||||
- **Metrics Endpoints**: Validates Prometheus exporters (ports 9091-9094)
|
||||
|
||||
### 3. Authentication Flow (`authentication_flow.rs`)
|
||||
Tests authentication and security:
|
||||
- **JWT Generation**: Token creation with claims
|
||||
- **JWT Validation**: Signature verification and claim extraction
|
||||
- **JWT Expiration**: Expired token rejection
|
||||
- **JWT Signature**: Invalid signature detection
|
||||
- **Session Storage**: Redis-based session management
|
||||
- **Token Revocation**: Revocation list checking
|
||||
- **Rate Limiting**: Request throttling with Redis
|
||||
|
||||
### 4. Basic Order Flow (`basic_order_flow.rs`)
|
||||
Tests core trading operations:
|
||||
- **Order Submission**: Insert orders into PostgreSQL
|
||||
- **Order Query**: Retrieve order by ID
|
||||
- **Order Cancellation**: Update order status to cancelled
|
||||
- **Position Management**: Create and query positions
|
||||
- **Order History**: Query user's order history
|
||||
- **Complete Lifecycle**: Full order flow from submission to cleanup
|
||||
|
||||
## Usage
|
||||
|
||||
### Run All Smoke Tests
|
||||
|
||||
```bash
|
||||
# Using the automated script (recommended)
|
||||
./run_smoke_tests.sh
|
||||
|
||||
# Using Cargo directly
|
||||
cargo test --test smoke_tests --features smoke-tests
|
||||
```
|
||||
|
||||
### Run Specific Categories
|
||||
|
||||
```bash
|
||||
# Infrastructure only
|
||||
./run_smoke_tests.sh --category infrastructure
|
||||
|
||||
# Service health only
|
||||
./run_smoke_tests.sh --category service
|
||||
|
||||
# Authentication only
|
||||
./run_smoke_tests.sh --category authentication
|
||||
|
||||
# Order flow only
|
||||
./run_smoke_tests.sh --category order_flow
|
||||
```
|
||||
|
||||
### Fast Mode (Critical Tests Only)
|
||||
|
||||
```bash
|
||||
# Run only infrastructure and service health checks
|
||||
./run_smoke_tests.sh --fast
|
||||
```
|
||||
|
||||
### Verbose Mode
|
||||
|
||||
```bash
|
||||
# Run with debug logging
|
||||
./run_smoke_tests.sh --verbose
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
All tests use environment variables with sensible defaults:
|
||||
|
||||
```bash
|
||||
# Infrastructure
|
||||
DATABASE_URL=postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt
|
||||
REDIS_URL=redis://localhost:6379
|
||||
VAULT_ADDR=http://localhost:8200
|
||||
INFLUXDB_URL=http://localhost:8086
|
||||
|
||||
# Services
|
||||
API_GATEWAY_URL=http://localhost:50051
|
||||
TRADING_SERVICE_URL=http://localhost:50052
|
||||
BACKTESTING_SERVICE_URL=http://localhost:50053
|
||||
ML_TRAINING_SERVICE_URL=http://localhost:50054
|
||||
|
||||
# Monitoring
|
||||
PROMETHEUS_URL=http://localhost:9090
|
||||
GRAFANA_URL=http://localhost:3000
|
||||
|
||||
# Authentication
|
||||
JWT_SECRET=dev_secret_key_change_in_production
|
||||
|
||||
# Logging
|
||||
RUST_LOG=info # Set to 'debug' for verbose output
|
||||
```
|
||||
|
||||
## Known Issues and Blockers
|
||||
|
||||
### Blocked Tests (Agent 96 Findings)
|
||||
|
||||
1. **Backtesting Service** - Configuration issues prevent deployment
|
||||
- Test marked with `#[ignore]` attribute
|
||||
- Skips gracefully when service unavailable
|
||||
|
||||
2. **ML Training Service** - Configuration issues prevent deployment
|
||||
- Test marked with `#[ignore]` attribute
|
||||
- Skips gracefully when service unavailable
|
||||
|
||||
### Working Tests
|
||||
|
||||
- ✅ Infrastructure Health (PostgreSQL, Redis, Vault, InfluxDB, Prometheus, Grafana)
|
||||
- ✅ Trading Service Health (confirmed working by Agent 96)
|
||||
- ✅ API Gateway Health
|
||||
- ✅ Authentication Flow (JWT, Sessions, Rate Limiting)
|
||||
- ✅ Basic Order Flow (Database operations)
|
||||
|
||||
## Test Architecture
|
||||
|
||||
### Graceful Failure Handling
|
||||
|
||||
Tests use the `skip_if_unavailable!` macro to handle service unavailability:
|
||||
|
||||
```rust
|
||||
skip_if_unavailable!("Service Name", {
|
||||
// Test code here
|
||||
result
|
||||
});
|
||||
```
|
||||
|
||||
This allows tests to:
|
||||
- Pass when services are available
|
||||
- Skip gracefully when services are down (connection refused, timeout)
|
||||
- Fail hard for actual test failures
|
||||
|
||||
### Timeout Management
|
||||
|
||||
All tests have configurable timeouts:
|
||||
- **Standard smoke tests**: 10 seconds
|
||||
- **Infrastructure tests**: 5 seconds
|
||||
|
||||
Timeouts prevent hanging tests and provide quick feedback.
|
||||
|
||||
### Parallel Execution
|
||||
|
||||
Tests can run in parallel by default, but use `--test-threads=1` for sequential execution when debugging:
|
||||
|
||||
```bash
|
||||
cargo test --test smoke_tests -- --test-threads=1 --nocapture
|
||||
```
|
||||
|
||||
## Integration with CI/CD
|
||||
|
||||
### Docker Deployment Validation
|
||||
|
||||
After deploying with Docker Compose:
|
||||
|
||||
```bash
|
||||
# 1. Start all services
|
||||
docker-compose up -d
|
||||
|
||||
# 2. Wait for health checks
|
||||
docker-compose ps
|
||||
|
||||
# 3. Run smoke tests
|
||||
./run_smoke_tests.sh
|
||||
|
||||
# 4. Check exit code
|
||||
if [ $? -eq 0 ]; then
|
||||
echo "Deployment validated - ready for production"
|
||||
else
|
||||
echo "Deployment issues detected - review logs"
|
||||
fi
|
||||
```
|
||||
|
||||
### Kubernetes Deployment Validation
|
||||
|
||||
```bash
|
||||
# 1. Deploy to cluster
|
||||
kubectl apply -f k8s/
|
||||
|
||||
# 2. Wait for pods
|
||||
kubectl wait --for=condition=ready pod -l app=foxhunt --timeout=300s
|
||||
|
||||
# 3. Port forward services
|
||||
kubectl port-forward svc/api-gateway 50051:50051 &
|
||||
kubectl port-forward svc/trading-service 50052:50051 &
|
||||
|
||||
# 4. Run smoke tests
|
||||
./run_smoke_tests.sh
|
||||
|
||||
# 5. Cleanup port forwards
|
||||
pkill -f "kubectl port-forward"
|
||||
```
|
||||
|
||||
## Metrics and Reporting
|
||||
|
||||
### Test Pass Rate
|
||||
|
||||
The smoke test runner reports:
|
||||
- Total categories tested
|
||||
- Passed/failed counts
|
||||
- Pass percentage
|
||||
- Exit code (0 = success, 1 = failure)
|
||||
|
||||
### Sample Output
|
||||
|
||||
```
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
Foxhunt HFT System - Smoke Test Suite
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Environment Configuration:
|
||||
Database: postgresql://foxhunt:***@localhost:5432/foxhunt
|
||||
Redis: redis://localhost:6379
|
||||
Vault: http://localhost:8200
|
||||
API Gateway: http://localhost:50051
|
||||
Trading Service: http://localhost:50052
|
||||
Log Level: info
|
||||
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
Testing: Infrastructure Health
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
✅ PostgreSQL connection successful (TimescaleDB: true)
|
||||
✅ Redis connection and operations successful
|
||||
✅ Vault connectivity successful (status: 200)
|
||||
|
||||
✅ Infrastructure Health - PASSED
|
||||
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
Smoke Test Summary
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Total Categories: 4
|
||||
Passed: 4
|
||||
Failed: 0
|
||||
|
||||
Pass Rate: 100%
|
||||
|
||||
✅ All smoke tests passed!
|
||||
System is ready for deployment.
|
||||
```
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
### Planned Test Categories (Not Implemented)
|
||||
|
||||
5. **Market Data Tests** (blocked by service availability)
|
||||
- Subscribe to market data stream
|
||||
- Receive quote updates
|
||||
- Historical data queries
|
||||
|
||||
6. **ML Inference Tests** (blocked by service availability)
|
||||
- Model prediction requests
|
||||
- Feature engineering pipeline
|
||||
- Inference latency validation (<100ms)
|
||||
|
||||
7. **Compliance Tests** (requires ML service)
|
||||
- Audit log creation
|
||||
- Best execution analysis
|
||||
- Risk check validation
|
||||
|
||||
8. **Advanced Monitoring** (requires full deployment)
|
||||
- Alert manager connectivity
|
||||
- Custom dashboard validation
|
||||
- Log aggregation checks
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Connection Refused**
|
||||
```bash
|
||||
# Check if services are running
|
||||
docker-compose ps
|
||||
|
||||
# Restart failed services
|
||||
docker-compose restart trading_service
|
||||
```
|
||||
|
||||
2. **Database Schema Missing**
|
||||
```bash
|
||||
# Run migrations
|
||||
cargo sqlx migrate run
|
||||
```
|
||||
|
||||
3. **Redis Connection Failed**
|
||||
```bash
|
||||
# Check Redis is running
|
||||
redis-cli ping
|
||||
|
||||
# Restart Redis
|
||||
docker-compose restart redis
|
||||
```
|
||||
|
||||
4. **JWT Secret Not Set**
|
||||
```bash
|
||||
# Set JWT secret
|
||||
export JWT_SECRET="your-secret-key-here"
|
||||
```
|
||||
|
||||
### Debug Mode
|
||||
|
||||
Run tests with full debug output:
|
||||
|
||||
```bash
|
||||
RUST_LOG=debug ./run_smoke_tests.sh --verbose
|
||||
```
|
||||
|
||||
This shows:
|
||||
- Full test execution flow
|
||||
- Connection attempts
|
||||
- Query execution
|
||||
- Error details
|
||||
|
||||
## Contributing
|
||||
|
||||
When adding new smoke tests:
|
||||
|
||||
1. Create test file in `tests/smoke_tests/`
|
||||
2. Add module to `tests/smoke_tests/mod.rs`
|
||||
3. Use `skip_if_unavailable!` macro for graceful failures
|
||||
4. Add timeout with `with_timeout()` wrapper
|
||||
5. Update this README with test documentation
|
||||
6. Update `run_smoke_tests.sh` with new category
|
||||
|
||||
### Test Template
|
||||
|
||||
```rust
|
||||
#[tokio::test]
|
||||
async fn test_new_feature() {
|
||||
let result = with_timeout(async {
|
||||
// Your test code here
|
||||
Ok(())
|
||||
}).await;
|
||||
|
||||
skip_if_unavailable!("Service Name", { result });
|
||||
}
|
||||
```
|
||||
364
tests/smoke_tests/authentication_flow.rs
Normal file
364
tests/smoke_tests/authentication_flow.rs
Normal file
@@ -0,0 +1,364 @@
|
||||
//! Authentication Flow Smoke Tests
|
||||
//!
|
||||
//! Tests to validate JWT authentication, session management, and authorization.
|
||||
|
||||
use super::common::*;
|
||||
use jsonwebtoken::{encode, decode, Header, Validation, EncodingKey, DecodingKey, Algorithm};
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::time::{SystemTime, UNIX_EPOCH};
|
||||
|
||||
#[derive(Debug, Serialize, Deserialize)]
|
||||
struct Claims {
|
||||
sub: String,
|
||||
iss: String,
|
||||
aud: String,
|
||||
exp: u64,
|
||||
iat: u64,
|
||||
roles: Vec<String>,
|
||||
}
|
||||
|
||||
fn get_jwt_secret() -> String {
|
||||
std::env::var("JWT_SECRET")
|
||||
.unwrap_or_else(|_| "dev_secret_key_change_in_production".to_string())
|
||||
}
|
||||
|
||||
fn create_test_token() -> Result<String, Box<dyn std::error::Error>> {
|
||||
let now = SystemTime::now().duration_since(UNIX_EPOCH)?.as_secs();
|
||||
|
||||
let claims = Claims {
|
||||
sub: "test_user".to_string(),
|
||||
iss: "foxhunt-api-gateway".to_string(),
|
||||
aud: "foxhunt-services".to_string(),
|
||||
iat: now,
|
||||
exp: now + 3600, // Valid for 1 hour
|
||||
roles: vec!["trader".to_string()],
|
||||
};
|
||||
|
||||
let token = encode(
|
||||
&Header::new(Algorithm::HS256),
|
||||
&claims,
|
||||
&EncodingKey::from_secret(get_jwt_secret().as_bytes()),
|
||||
)?;
|
||||
|
||||
Ok(token)
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_jwt_token_generation() {
|
||||
let result = with_timeout(async {
|
||||
let token = create_test_token()
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
assert!(!token.is_empty(), "JWT token should not be empty");
|
||||
assert!(token.split('.').count() == 3, "JWT should have 3 parts");
|
||||
|
||||
println!("✅ JWT token generation successful");
|
||||
println!(" Token length: {} characters", token.len());
|
||||
Ok(())
|
||||
}).await;
|
||||
|
||||
assert!(result.is_ok(), "JWT token generation failed: {:?}", result.err());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_jwt_token_validation() {
|
||||
let result = with_timeout(async {
|
||||
// Generate token
|
||||
let token = create_test_token()
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
// Validate token
|
||||
let validation = Validation::new(Algorithm::HS256);
|
||||
let token_data = decode::<Claims>(
|
||||
&token,
|
||||
&DecodingKey::from_secret(get_jwt_secret().as_bytes()),
|
||||
&validation,
|
||||
)
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
assert_eq!(token_data.claims.sub, "test_user");
|
||||
assert_eq!(token_data.claims.iss, "foxhunt-api-gateway");
|
||||
assert_eq!(token_data.claims.aud, "foxhunt-services");
|
||||
assert!(token_data.claims.roles.contains(&"trader".to_string()));
|
||||
|
||||
println!("✅ JWT token validation successful");
|
||||
println!(" User: {}", token_data.claims.sub);
|
||||
println!(" Roles: {:?}", token_data.claims.roles);
|
||||
Ok(())
|
||||
}).await;
|
||||
|
||||
assert!(result.is_ok(), "JWT token validation failed: {:?}", result.err());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_jwt_token_expiration() {
|
||||
let result = with_timeout(async {
|
||||
// Create expired token
|
||||
let now = SystemTime::now().duration_since(UNIX_EPOCH)?.as_secs();
|
||||
|
||||
let expired_claims = Claims {
|
||||
sub: "test_user".to_string(),
|
||||
iss: "foxhunt-api-gateway".to_string(),
|
||||
aud: "foxhunt-services".to_string(),
|
||||
iat: now - 7200, // Issued 2 hours ago
|
||||
exp: now - 3600, // Expired 1 hour ago
|
||||
roles: vec!["trader".to_string()],
|
||||
};
|
||||
|
||||
let expired_token = encode(
|
||||
&Header::new(Algorithm::HS256),
|
||||
&expired_claims,
|
||||
&EncodingKey::from_secret(get_jwt_secret().as_bytes()),
|
||||
)?;
|
||||
|
||||
// Try to validate expired token
|
||||
let validation = Validation::new(Algorithm::HS256);
|
||||
let result = decode::<Claims>(
|
||||
&expired_token,
|
||||
&DecodingKey::from_secret(get_jwt_secret().as_bytes()),
|
||||
&validation,
|
||||
);
|
||||
|
||||
assert!(result.is_err(), "Expired token should not validate");
|
||||
|
||||
println!("✅ JWT expiration validation successful");
|
||||
Ok(())
|
||||
}).await;
|
||||
|
||||
assert!(result.is_ok(), "JWT expiration test failed: {:?}", result.err());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_jwt_invalid_signature() {
|
||||
let result = with_timeout(async {
|
||||
// Create token with one secret
|
||||
let token = create_test_token()
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
// Try to validate with different secret
|
||||
let validation = Validation::new(Algorithm::HS256);
|
||||
let wrong_secret = "wrong_secret_key";
|
||||
let result = decode::<Claims>(
|
||||
&token,
|
||||
&DecodingKey::from_secret(wrong_secret.as_bytes()),
|
||||
&validation,
|
||||
);
|
||||
|
||||
assert!(result.is_err(), "Token with invalid signature should not validate");
|
||||
|
||||
println!("✅ JWT signature validation successful");
|
||||
Ok(())
|
||||
}).await;
|
||||
|
||||
assert!(result.is_ok(), "JWT signature validation test failed: {:?}", result.err());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_redis_session_storage() {
|
||||
let result = with_timeout(async {
|
||||
use redis::AsyncCommands;
|
||||
|
||||
let client = redis::Client::open(redis_url().as_str())
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
let mut con = client.get_multiplexed_async_connection()
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
// Simulate session storage
|
||||
let session_key = "smoke_test:session:test_user";
|
||||
let session_data = serde_json::json!({
|
||||
"user_id": "test_user",
|
||||
"roles": ["trader"],
|
||||
"created_at": chrono::Utc::now().to_rfc3339(),
|
||||
});
|
||||
|
||||
// Store session with TTL
|
||||
con.set_ex::<_, _, ()>(session_key, session_data.to_string(), 3600)
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
// Retrieve session
|
||||
let retrieved: String = con.get(session_key)
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
let retrieved_json: serde_json::Value = serde_json::from_str(&retrieved)?;
|
||||
assert_eq!(retrieved_json["user_id"], "test_user");
|
||||
|
||||
// Cleanup
|
||||
con.del::<_, ()>(session_key)
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
println!("✅ Redis session storage successful");
|
||||
Ok(())
|
||||
}).await;
|
||||
|
||||
skip_if_unavailable!("Redis", { result });
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_jwt_revocation_check() {
|
||||
let result = with_timeout(async {
|
||||
use redis::AsyncCommands;
|
||||
|
||||
let client = redis::Client::open(redis_url().as_str())
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
let mut con = client.get_multiplexed_async_connection()
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
// Create token
|
||||
let token = create_test_token()
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
// Simulate token revocation
|
||||
let jti = "smoke_test_token_id";
|
||||
let revocation_key = format!("revoked_tokens:{}", jti);
|
||||
|
||||
// Add to revocation list
|
||||
con.set_ex::<_, _, ()>(&revocation_key, "revoked", 3600)
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
// Check if token is revoked
|
||||
let is_revoked: bool = con.exists(&revocation_key)
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
assert!(is_revoked, "Token should be marked as revoked");
|
||||
|
||||
// Cleanup
|
||||
con.del::<_, ()>(&revocation_key)
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
println!("✅ JWT revocation check successful");
|
||||
Ok(())
|
||||
}).await;
|
||||
|
||||
skip_if_unavailable!("Redis", { result });
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_rate_limiting() {
|
||||
let result = with_timeout(async {
|
||||
use redis::AsyncCommands;
|
||||
|
||||
let client = redis::Client::open(redis_url().as_str())
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
let mut con = client.get_multiplexed_async_connection()
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
// Simulate rate limiting
|
||||
let rate_limit_key = "smoke_test:rate_limit:test_user";
|
||||
let window_seconds = 60;
|
||||
let max_requests = 100;
|
||||
|
||||
// Increment counter
|
||||
let count: i32 = con.incr(&rate_limit_key, 1)
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
if count == 1 {
|
||||
// Set expiration on first request
|
||||
con.expire::<_, ()>(&rate_limit_key, window_seconds)
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
}
|
||||
|
||||
// Check if rate limit exceeded
|
||||
let is_limited = count > max_requests;
|
||||
assert!(!is_limited, "Rate limit should not be exceeded on first request");
|
||||
|
||||
// Get TTL
|
||||
let ttl: i32 = con.ttl(&rate_limit_key)
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
assert!(ttl > 0 && ttl <= window_seconds, "TTL should be set correctly");
|
||||
|
||||
// Cleanup
|
||||
con.del::<_, ()>(&rate_limit_key)
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
println!("✅ Rate limiting check successful");
|
||||
println!(" Request count: {}/{}", count, max_requests);
|
||||
println!(" TTL: {} seconds", ttl);
|
||||
Ok(())
|
||||
}).await;
|
||||
|
||||
skip_if_unavailable!("Redis", { result });
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_authentication_flow_complete() {
|
||||
println!("🔐 Running complete authentication flow...");
|
||||
|
||||
let mut steps_passed = 0;
|
||||
let total_steps = 4;
|
||||
|
||||
// Step 1: Generate token
|
||||
if create_test_token().is_ok() {
|
||||
println!(" ✓ JWT token generation");
|
||||
steps_passed += 1;
|
||||
} else {
|
||||
println!(" ✗ JWT token generation failed");
|
||||
}
|
||||
|
||||
// Step 2: Validate token
|
||||
if let Ok(token) = create_test_token() {
|
||||
let validation = Validation::new(Algorithm::HS256);
|
||||
if decode::<Claims>(
|
||||
&token,
|
||||
&DecodingKey::from_secret(get_jwt_secret().as_bytes()),
|
||||
&validation,
|
||||
).is_ok() {
|
||||
println!(" ✓ JWT token validation");
|
||||
steps_passed += 1;
|
||||
} else {
|
||||
println!(" ✗ JWT token validation failed");
|
||||
}
|
||||
}
|
||||
|
||||
// Step 3: Session storage
|
||||
if let Ok(client) = redis::Client::open(redis_url().as_str()) {
|
||||
if let Ok(mut con) = client.get_multiplexed_async_connection().await {
|
||||
use redis::AsyncCommands;
|
||||
let test_key = "smoke_test:auth_flow:session";
|
||||
if con.set_ex::<_, _, ()>(test_key, "test", 60).await.is_ok() {
|
||||
println!(" ✓ Session storage");
|
||||
steps_passed += 1;
|
||||
let _ = con.del::<_, ()>(test_key).await;
|
||||
}
|
||||
}
|
||||
} else {
|
||||
println!(" ⚠ Session storage (Redis unavailable)");
|
||||
}
|
||||
|
||||
// Step 4: Rate limiting
|
||||
if let Ok(client) = redis::Client::open(redis_url().as_str()) {
|
||||
if let Ok(mut con) = client.get_multiplexed_async_connection().await {
|
||||
use redis::AsyncCommands;
|
||||
let test_key = "smoke_test:auth_flow:rate_limit";
|
||||
if con.incr::<_, _, i32>(test_key, 1).await.is_ok() {
|
||||
println!(" ✓ Rate limiting");
|
||||
steps_passed += 1;
|
||||
let _ = con.del::<_, ()>(test_key).await;
|
||||
}
|
||||
}
|
||||
} else {
|
||||
println!(" ⚠ Rate limiting (Redis unavailable)");
|
||||
}
|
||||
|
||||
println!("\n📊 Authentication Flow Summary:");
|
||||
println!(" Steps passed: {}/{}", steps_passed, total_steps);
|
||||
|
||||
assert!(steps_passed >= 2, "Authentication flow requires at least JWT functionality");
|
||||
println!("✅ Authentication flow check complete");
|
||||
}
|
||||
413
tests/smoke_tests/basic_order_flow.rs
Normal file
413
tests/smoke_tests/basic_order_flow.rs
Normal file
@@ -0,0 +1,413 @@
|
||||
//! Basic Order Flow Smoke Tests
|
||||
//!
|
||||
//! Tests to validate core trading functionality including order submission,
|
||||
//! querying, cancellation, and position management.
|
||||
|
||||
use super::common::*;
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_database_order_submission() {
|
||||
let result = with_timeout(async {
|
||||
let pool = sqlx::PgPool::connect(&database_url())
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
// Create a test order
|
||||
let order_id = uuid::Uuid::new_v4();
|
||||
let user_id = "smoke_test_user";
|
||||
let symbol = "BTC-USD";
|
||||
let order_type = "market";
|
||||
let side = "buy";
|
||||
let quantity = 1.0;
|
||||
let price = None::<f64>;
|
||||
|
||||
// Insert order
|
||||
sqlx::query(
|
||||
r#"
|
||||
INSERT INTO orders (
|
||||
id, user_id, symbol, order_type, side, quantity, price,
|
||||
status, created_at, updated_at
|
||||
)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, NOW(), NOW())
|
||||
"#
|
||||
)
|
||||
.bind(&order_id)
|
||||
.bind(user_id)
|
||||
.bind(symbol)
|
||||
.bind(order_type)
|
||||
.bind(side)
|
||||
.bind(quantity)
|
||||
.bind(price)
|
||||
.bind("pending")
|
||||
.execute(&pool)
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
println!("✅ Order submitted to database");
|
||||
println!(" Order ID: {}", order_id);
|
||||
println!(" Symbol: {}", symbol);
|
||||
println!(" Side: {}", side);
|
||||
println!(" Quantity: {}", quantity);
|
||||
|
||||
// Cleanup
|
||||
sqlx::query("DELETE FROM orders WHERE id = $1")
|
||||
.bind(&order_id)
|
||||
.execute(&pool)
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
pool.close().await;
|
||||
Ok(())
|
||||
}).await;
|
||||
|
||||
skip_if_unavailable!("PostgreSQL", { result });
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_database_order_query() {
|
||||
let result = with_timeout(async {
|
||||
let pool = sqlx::PgPool::connect(&database_url())
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
// Create a test order
|
||||
let order_id = uuid::Uuid::new_v4();
|
||||
let user_id = "smoke_test_user";
|
||||
let symbol = "ETH-USD";
|
||||
|
||||
sqlx::query(
|
||||
r#"
|
||||
INSERT INTO orders (
|
||||
id, user_id, symbol, order_type, side, quantity,
|
||||
status, created_at, updated_at
|
||||
)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, NOW(), NOW())
|
||||
"#
|
||||
)
|
||||
.bind(&order_id)
|
||||
.bind(user_id)
|
||||
.bind(symbol)
|
||||
.bind("limit")
|
||||
.bind("sell")
|
||||
.bind(2.0)
|
||||
.bind("pending")
|
||||
.execute(&pool)
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
// Query order
|
||||
let order: (uuid::Uuid, String, String) = sqlx::query_as(
|
||||
"SELECT id, user_id, symbol FROM orders WHERE id = $1"
|
||||
)
|
||||
.bind(&order_id)
|
||||
.fetch_one(&pool)
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
assert_eq!(order.0, order_id);
|
||||
assert_eq!(order.1, user_id);
|
||||
assert_eq!(order.2, symbol);
|
||||
|
||||
println!("✅ Order query successful");
|
||||
println!(" Order ID: {}", order.0);
|
||||
println!(" User ID: {}", order.1);
|
||||
println!(" Symbol: {}", order.2);
|
||||
|
||||
// Cleanup
|
||||
sqlx::query("DELETE FROM orders WHERE id = $1")
|
||||
.bind(&order_id)
|
||||
.execute(&pool)
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
pool.close().await;
|
||||
Ok(())
|
||||
}).await;
|
||||
|
||||
skip_if_unavailable!("PostgreSQL", { result });
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_database_order_cancellation() {
|
||||
let result = with_timeout(async {
|
||||
let pool = sqlx::PgPool::connect(&database_url())
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
// Create a test order
|
||||
let order_id = uuid::Uuid::new_v4();
|
||||
|
||||
sqlx::query(
|
||||
r#"
|
||||
INSERT INTO orders (
|
||||
id, user_id, symbol, order_type, side, quantity,
|
||||
status, created_at, updated_at
|
||||
)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, NOW(), NOW())
|
||||
"#
|
||||
)
|
||||
.bind(&order_id)
|
||||
.bind("smoke_test_user")
|
||||
.bind("BTC-USD")
|
||||
.bind("limit")
|
||||
.bind("buy")
|
||||
.bind(0.5)
|
||||
.bind("pending")
|
||||
.execute(&pool)
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
// Cancel order
|
||||
sqlx::query(
|
||||
"UPDATE orders SET status = $1, updated_at = NOW() WHERE id = $2"
|
||||
)
|
||||
.bind("cancelled")
|
||||
.bind(&order_id)
|
||||
.execute(&pool)
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
// Verify cancellation
|
||||
let status: (String,) = sqlx::query_as(
|
||||
"SELECT status FROM orders WHERE id = $1"
|
||||
)
|
||||
.bind(&order_id)
|
||||
.fetch_one(&pool)
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
assert_eq!(status.0, "cancelled");
|
||||
|
||||
println!("✅ Order cancellation successful");
|
||||
println!(" Order ID: {}", order_id);
|
||||
println!(" Status: {}", status.0);
|
||||
|
||||
// Cleanup
|
||||
sqlx::query("DELETE FROM orders WHERE id = $1")
|
||||
.bind(&order_id)
|
||||
.execute(&pool)
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
pool.close().await;
|
||||
Ok(())
|
||||
}).await;
|
||||
|
||||
skip_if_unavailable!("PostgreSQL", { result });
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_database_position_management() {
|
||||
let result = with_timeout(async {
|
||||
let pool = sqlx::PgPool::connect(&database_url())
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
let position_id = uuid::Uuid::new_v4();
|
||||
let user_id = "smoke_test_user";
|
||||
let symbol = "BTC-USD";
|
||||
let quantity = 1.5;
|
||||
let entry_price = 50000.0;
|
||||
|
||||
// Create position
|
||||
sqlx::query(
|
||||
r#"
|
||||
INSERT INTO positions (
|
||||
id, user_id, symbol, quantity, entry_price, current_price,
|
||||
unrealized_pnl, created_at, updated_at
|
||||
)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, NOW(), NOW())
|
||||
"#
|
||||
)
|
||||
.bind(&position_id)
|
||||
.bind(user_id)
|
||||
.bind(symbol)
|
||||
.bind(quantity)
|
||||
.bind(entry_price)
|
||||
.bind(entry_price) // current_price = entry_price initially
|
||||
.bind(0.0) // unrealized_pnl = 0 initially
|
||||
.execute(&pool)
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
// Query position
|
||||
let position: (uuid::Uuid, String, f64, f64) = sqlx::query_as(
|
||||
"SELECT id, symbol, quantity, entry_price FROM positions WHERE id = $1"
|
||||
)
|
||||
.bind(&position_id)
|
||||
.fetch_one(&pool)
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
assert_eq!(position.0, position_id);
|
||||
assert_eq!(position.1, symbol);
|
||||
assert_eq!(position.2, quantity);
|
||||
assert_eq!(position.3, entry_price);
|
||||
|
||||
println!("✅ Position management successful");
|
||||
println!(" Position ID: {}", position.0);
|
||||
println!(" Symbol: {}", position.1);
|
||||
println!(" Quantity: {}", position.2);
|
||||
println!(" Entry Price: ${}", position.3);
|
||||
|
||||
// Cleanup
|
||||
sqlx::query("DELETE FROM positions WHERE id = $1")
|
||||
.bind(&position_id)
|
||||
.execute(&pool)
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
pool.close().await;
|
||||
Ok(())
|
||||
}).await;
|
||||
|
||||
skip_if_unavailable!("PostgreSQL", { result });
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_database_order_history() {
|
||||
let result = with_timeout(async {
|
||||
let pool = sqlx::PgPool::connect(&database_url())
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
let user_id = "smoke_test_user";
|
||||
|
||||
// Create multiple test orders
|
||||
let order_ids: Vec<uuid::Uuid> = (0..3)
|
||||
.map(|_| uuid::Uuid::new_v4())
|
||||
.collect();
|
||||
|
||||
for order_id in &order_ids {
|
||||
sqlx::query(
|
||||
r#"
|
||||
INSERT INTO orders (
|
||||
id, user_id, symbol, order_type, side, quantity,
|
||||
status, created_at, updated_at
|
||||
)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, NOW(), NOW())
|
||||
"#
|
||||
)
|
||||
.bind(order_id)
|
||||
.bind(user_id)
|
||||
.bind("BTC-USD")
|
||||
.bind("market")
|
||||
.bind("buy")
|
||||
.bind(0.1)
|
||||
.bind("filled")
|
||||
.execute(&pool)
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
}
|
||||
|
||||
// Query order history
|
||||
let orders: Vec<(uuid::Uuid, String)> = sqlx::query_as(
|
||||
"SELECT id, status FROM orders WHERE user_id = $1 ORDER BY created_at DESC LIMIT 10"
|
||||
)
|
||||
.bind(user_id)
|
||||
.fetch_all(&pool)
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
assert!(orders.len() >= 3, "Should find at least 3 orders");
|
||||
|
||||
println!("✅ Order history query successful");
|
||||
println!(" Found {} orders for user {}", orders.len(), user_id);
|
||||
|
||||
// Cleanup
|
||||
for order_id in &order_ids {
|
||||
sqlx::query("DELETE FROM orders WHERE id = $1")
|
||||
.bind(order_id)
|
||||
.execute(&pool)
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
}
|
||||
|
||||
pool.close().await;
|
||||
Ok(())
|
||||
}).await;
|
||||
|
||||
skip_if_unavailable!("PostgreSQL", { result });
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_complete_order_lifecycle() {
|
||||
println!("🔄 Running complete order lifecycle...");
|
||||
|
||||
let mut steps_passed = 0;
|
||||
let total_steps = 4;
|
||||
|
||||
if let Ok(pool) = sqlx::PgPool::connect(&database_url()).await {
|
||||
let order_id = uuid::Uuid::new_v4();
|
||||
|
||||
// Step 1: Submit order
|
||||
if sqlx::query(
|
||||
r#"
|
||||
INSERT INTO orders (
|
||||
id, user_id, symbol, order_type, side, quantity,
|
||||
status, created_at, updated_at
|
||||
)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, NOW(), NOW())
|
||||
"#
|
||||
)
|
||||
.bind(&order_id)
|
||||
.bind("smoke_test_user")
|
||||
.bind("BTC-USD")
|
||||
.bind("market")
|
||||
.bind("buy")
|
||||
.bind(1.0)
|
||||
.bind("pending")
|
||||
.execute(&pool)
|
||||
.await
|
||||
.is_ok() {
|
||||
println!(" ✓ Order submission");
|
||||
steps_passed += 1;
|
||||
}
|
||||
|
||||
// Step 2: Query order
|
||||
if sqlx::query_as::<_, (String,)>(
|
||||
"SELECT status FROM orders WHERE id = $1"
|
||||
)
|
||||
.bind(&order_id)
|
||||
.fetch_one(&pool)
|
||||
.await
|
||||
.is_ok() {
|
||||
println!(" ✓ Order query");
|
||||
steps_passed += 1;
|
||||
}
|
||||
|
||||
// Step 3: Update order
|
||||
if sqlx::query(
|
||||
"UPDATE orders SET status = $1, updated_at = NOW() WHERE id = $2"
|
||||
)
|
||||
.bind("filled")
|
||||
.bind(&order_id)
|
||||
.execute(&pool)
|
||||
.await
|
||||
.is_ok() {
|
||||
println!(" ✓ Order update");
|
||||
steps_passed += 1;
|
||||
}
|
||||
|
||||
// Step 4: Cancel/cleanup
|
||||
if sqlx::query("DELETE FROM orders WHERE id = $1")
|
||||
.bind(&order_id)
|
||||
.execute(&pool)
|
||||
.await
|
||||
.is_ok() {
|
||||
println!(" ✓ Order cleanup");
|
||||
steps_passed += 1;
|
||||
}
|
||||
|
||||
pool.close().await;
|
||||
} else {
|
||||
println!(" ⚠ Database unavailable - skipping order lifecycle test");
|
||||
return;
|
||||
}
|
||||
|
||||
println!("\n📊 Order Lifecycle Summary:");
|
||||
println!(" Steps passed: {}/{}", steps_passed, total_steps);
|
||||
|
||||
assert_eq!(steps_passed, total_steps, "Complete order lifecycle should pass all steps");
|
||||
println!("✅ Order lifecycle check complete");
|
||||
}
|
||||
293
tests/smoke_tests/infrastructure_health.rs
Normal file
293
tests/smoke_tests/infrastructure_health.rs
Normal file
@@ -0,0 +1,293 @@
|
||||
//! Infrastructure Health Smoke Tests
|
||||
//!
|
||||
//! Tests to validate that core infrastructure services are available and functional.
|
||||
//! These tests run first to ensure the foundation is solid before testing application services.
|
||||
|
||||
use super::common::*;
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_postgres_connection() {
|
||||
let result = with_timeout(async {
|
||||
let pool = sqlx::PgPool::connect(&database_url())
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
// Verify database is responsive
|
||||
let row: (i32,) = sqlx::query_as("SELECT 1")
|
||||
.fetch_one(&pool)
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
assert_eq!(row.0, 1, "Database query returned unexpected value");
|
||||
|
||||
// Check TimescaleDB extension
|
||||
let has_timescale: (bool,) = sqlx::query_as(
|
||||
"SELECT COUNT(*) > 0 FROM pg_extension WHERE extname = 'timescaledb'"
|
||||
)
|
||||
.fetch_one(&pool)
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
pool.close().await;
|
||||
|
||||
println!("✅ PostgreSQL connection successful (TimescaleDB: {})", has_timescale.0);
|
||||
Ok(())
|
||||
}).await;
|
||||
|
||||
skip_if_unavailable!("PostgreSQL", { result });
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_postgres_schema_exists() {
|
||||
let result = with_timeout(async {
|
||||
let pool = sqlx::PgPool::connect(&database_url())
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
// Check for key tables
|
||||
let tables = vec!["orders", "positions", "market_data", "audit_log"];
|
||||
for table in tables {
|
||||
let exists: (bool,) = sqlx::query_as(
|
||||
"SELECT EXISTS (SELECT FROM information_schema.tables WHERE table_name = $1)"
|
||||
)
|
||||
.bind(table)
|
||||
.fetch_one(&pool)
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
assert!(exists.0, "Table '{}' does not exist", table);
|
||||
println!(" ✓ Table '{}' exists", table);
|
||||
}
|
||||
|
||||
pool.close().await;
|
||||
println!("✅ PostgreSQL schema validation successful");
|
||||
Ok(())
|
||||
}).await;
|
||||
|
||||
skip_if_unavailable!("PostgreSQL", { result });
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_redis_connection() {
|
||||
let result = with_timeout(async {
|
||||
use redis::AsyncCommands;
|
||||
|
||||
let client = redis::Client::open(redis_url().as_str())
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
let mut con = client.get_multiplexed_async_connection()
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
// Test PING
|
||||
let pong: String = redis::cmd("PING")
|
||||
.query_async(&mut con)
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
assert_eq!(pong, "PONG", "Redis PING returned unexpected value");
|
||||
|
||||
// Test SET/GET
|
||||
let test_key = "smoke_test:redis_health";
|
||||
let test_value = "healthy";
|
||||
|
||||
con.set::<_, _, ()>(test_key, test_value)
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
let retrieved: String = con.get(test_key)
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
assert_eq!(retrieved, test_value, "Redis GET returned unexpected value");
|
||||
|
||||
// Cleanup
|
||||
con.del::<_, ()>(test_key)
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
println!("✅ Redis connection and operations successful");
|
||||
Ok(())
|
||||
}).await;
|
||||
|
||||
skip_if_unavailable!("Redis", { result });
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_vault_connectivity() {
|
||||
let result = with_timeout(async {
|
||||
let client = reqwest::Client::new();
|
||||
let vault_url = vault_url();
|
||||
|
||||
// Test Vault health endpoint
|
||||
let response = client
|
||||
.get(format!("{}/v1/sys/health", vault_url))
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
// Vault health returns 200 (initialized and unsealed), 429 (unsealed and standby),
|
||||
// 472 (data recovery mode), 473 (performance standby), or 501 (not initialized)
|
||||
let status = response.status();
|
||||
assert!(
|
||||
status == 200 || status == 429 || status == 473,
|
||||
"Vault health check returned unexpected status: {}",
|
||||
status
|
||||
);
|
||||
|
||||
println!("✅ Vault connectivity successful (status: {})", status);
|
||||
Ok(())
|
||||
}).await;
|
||||
|
||||
skip_if_unavailable!("Vault", { result });
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_influxdb_connectivity() {
|
||||
let result = with_timeout(async {
|
||||
let client = reqwest::Client::new();
|
||||
let influxdb_url = influxdb_url();
|
||||
|
||||
// Test InfluxDB ping endpoint
|
||||
let response = client
|
||||
.get(format!("{}/ping", influxdb_url))
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
assert!(
|
||||
response.status().is_success(),
|
||||
"InfluxDB ping failed with status: {}",
|
||||
response.status()
|
||||
);
|
||||
|
||||
println!("✅ InfluxDB connectivity successful");
|
||||
Ok(())
|
||||
}).await;
|
||||
|
||||
skip_if_unavailable!("InfluxDB", { result });
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_prometheus_connectivity() {
|
||||
let result = with_timeout(async {
|
||||
let client = reqwest::Client::new();
|
||||
let prometheus_url = prometheus_url();
|
||||
|
||||
// Test Prometheus health endpoint
|
||||
let response = client
|
||||
.get(format!("{}/-/healthy", prometheus_url))
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
assert!(
|
||||
response.status().is_success(),
|
||||
"Prometheus health check failed with status: {}",
|
||||
response.status()
|
||||
);
|
||||
|
||||
println!("✅ Prometheus connectivity successful");
|
||||
Ok(())
|
||||
}).await;
|
||||
|
||||
skip_if_unavailable!("Prometheus", { result });
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_grafana_connectivity() {
|
||||
let result = with_timeout(async {
|
||||
let client = reqwest::Client::new();
|
||||
let grafana_url = grafana_url();
|
||||
|
||||
// Test Grafana API health endpoint
|
||||
let response = client
|
||||
.get(format!("{}/api/health", grafana_url))
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
|
||||
assert!(
|
||||
response.status().is_success(),
|
||||
"Grafana health check failed with status: {}",
|
||||
response.status()
|
||||
);
|
||||
|
||||
println!("✅ Grafana connectivity successful");
|
||||
Ok(())
|
||||
}).await;
|
||||
|
||||
skip_if_unavailable!("Grafana", { result });
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_infrastructure_all_healthy() {
|
||||
println!("🏥 Running comprehensive infrastructure health check...");
|
||||
|
||||
let mut checks_passed = 0;
|
||||
let mut checks_failed = 0;
|
||||
|
||||
// PostgreSQL
|
||||
match sqlx::PgPool::connect(&database_url()).await {
|
||||
Ok(pool) => {
|
||||
pool.close().await;
|
||||
println!(" ✓ PostgreSQL");
|
||||
checks_passed += 1;
|
||||
}
|
||||
Err(e) => {
|
||||
eprintln!(" ✗ PostgreSQL: {}", e);
|
||||
checks_failed += 1;
|
||||
}
|
||||
}
|
||||
|
||||
// Redis
|
||||
match redis::Client::open(redis_url().as_str()) {
|
||||
Ok(client) => {
|
||||
match client.get_multiplexed_async_connection().await {
|
||||
Ok(_) => {
|
||||
println!(" ✓ Redis");
|
||||
checks_passed += 1;
|
||||
}
|
||||
Err(e) => {
|
||||
eprintln!(" ✗ Redis: {}", e);
|
||||
checks_failed += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(e) => {
|
||||
eprintln!(" ✗ Redis: {}", e);
|
||||
checks_failed += 1;
|
||||
}
|
||||
}
|
||||
|
||||
// Vault (optional - don't fail if unavailable)
|
||||
let client = reqwest::Client::new();
|
||||
match client.get(format!("{}/v1/sys/health", vault_url())).send().await {
|
||||
Ok(response) if response.status() == 200 || response.status() == 429 => {
|
||||
println!(" ✓ Vault");
|
||||
checks_passed += 1;
|
||||
}
|
||||
_ => {
|
||||
println!(" ⚠ Vault (not critical)");
|
||||
}
|
||||
}
|
||||
|
||||
// InfluxDB (optional - don't fail if unavailable)
|
||||
match client.get(format!("{}/ping", influxdb_url())).send().await {
|
||||
Ok(response) if response.status().is_success() => {
|
||||
println!(" ✓ InfluxDB");
|
||||
checks_passed += 1;
|
||||
}
|
||||
_ => {
|
||||
println!(" ⚠ InfluxDB (not critical)");
|
||||
}
|
||||
}
|
||||
|
||||
println!("\n📊 Infrastructure Health Summary:");
|
||||
println!(" Passed: {}", checks_passed);
|
||||
println!(" Failed: {}", checks_failed);
|
||||
|
||||
// Require at least PostgreSQL and Redis
|
||||
assert!(checks_passed >= 2, "Critical infrastructure services are down");
|
||||
println!("✅ Infrastructure health check complete");
|
||||
}
|
||||
134
tests/smoke_tests/mod.rs
Normal file
134
tests/smoke_tests/mod.rs
Normal file
@@ -0,0 +1,134 @@
|
||||
//! Smoke Tests - End-to-End System Validation
|
||||
//!
|
||||
//! Automated smoke tests to validate complete system functionality after deployment.
|
||||
//! Tests are organized by category and can run selectively based on service availability.
|
||||
//!
|
||||
//! ## Test Categories
|
||||
//! - Infrastructure Health (PostgreSQL, Redis, Vault, InfluxDB)
|
||||
//! - Service Health (API Gateway, Trading, Backtesting, ML Training)
|
||||
//! - Authentication Flow (JWT, MFA, Session Management)
|
||||
//! - Basic Order Flow (Submit, Query, Cancel, Positions)
|
||||
//! - Market Data (Streaming, Historical Queries)
|
||||
//! - ML Inference (Predictions, Feature Engineering)
|
||||
//! - Compliance (Audit Logs, Best Execution, Risk Checks)
|
||||
//! - Metrics and Monitoring (Prometheus, Grafana)
|
||||
//!
|
||||
//! ## Usage
|
||||
//! ```bash
|
||||
//! # Run all smoke tests
|
||||
//! cargo test --test smoke_tests
|
||||
//!
|
||||
//! # Run specific category
|
||||
//! cargo test --test smoke_tests infrastructure_health
|
||||
//! cargo test --test smoke_tests service_health
|
||||
//! cargo test --test smoke_tests basic_order_flow
|
||||
//!
|
||||
//! # Run with logging
|
||||
//! RUST_LOG=debug cargo test --test smoke_tests -- --nocapture
|
||||
//! ```
|
||||
|
||||
pub mod infrastructure_health;
|
||||
pub mod service_health;
|
||||
pub mod authentication_flow;
|
||||
pub mod basic_order_flow;
|
||||
|
||||
/// Common test utilities and helpers
|
||||
pub mod common {
|
||||
use std::time::Duration;
|
||||
use tokio::time::timeout;
|
||||
|
||||
/// Standard timeout for smoke tests (10 seconds)
|
||||
pub const SMOKE_TEST_TIMEOUT: Duration = Duration::from_secs(10);
|
||||
|
||||
/// Standard timeout for infrastructure checks (5 seconds)
|
||||
pub const INFRA_TEST_TIMEOUT: Duration = Duration::from_secs(5);
|
||||
|
||||
/// Load database URL from environment
|
||||
pub fn database_url() -> String {
|
||||
std::env::var("DATABASE_URL")
|
||||
.unwrap_or_else(|_| "postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt".to_string())
|
||||
}
|
||||
|
||||
/// Load Redis URL from environment
|
||||
pub fn redis_url() -> String {
|
||||
std::env::var("REDIS_URL")
|
||||
.unwrap_or_else(|_| "redis://localhost:6379".to_string())
|
||||
}
|
||||
|
||||
/// Load Vault URL from environment
|
||||
pub fn vault_url() -> String {
|
||||
std::env::var("VAULT_ADDR")
|
||||
.unwrap_or_else(|_| "http://localhost:8200".to_string())
|
||||
}
|
||||
|
||||
/// Load InfluxDB URL from environment
|
||||
pub fn influxdb_url() -> String {
|
||||
std::env::var("INFLUXDB_URL")
|
||||
.unwrap_or_else(|_| "http://localhost:8086".to_string())
|
||||
}
|
||||
|
||||
/// Load API Gateway URL from environment
|
||||
pub fn api_gateway_url() -> String {
|
||||
std::env::var("API_GATEWAY_URL")
|
||||
.unwrap_or_else(|_| "http://localhost:50051".to_string())
|
||||
}
|
||||
|
||||
/// Load Trading Service URL from environment
|
||||
pub fn trading_service_url() -> String {
|
||||
std::env::var("TRADING_SERVICE_URL")
|
||||
.unwrap_or_else(|_| "http://localhost:50052".to_string())
|
||||
}
|
||||
|
||||
/// Load Backtesting Service URL from environment
|
||||
pub fn backtesting_service_url() -> String {
|
||||
std::env::var("BACKTESTING_SERVICE_URL")
|
||||
.unwrap_or_else(|_| "http://localhost:50053".to_string())
|
||||
}
|
||||
|
||||
/// Load ML Training Service URL from environment
|
||||
pub fn ml_training_service_url() -> String {
|
||||
std::env::var("ML_TRAINING_SERVICE_URL")
|
||||
.unwrap_or_else(|_| "http://localhost:50054".to_string())
|
||||
}
|
||||
|
||||
/// Load Prometheus URL from environment
|
||||
pub fn prometheus_url() -> String {
|
||||
std::env::var("PROMETHEUS_URL")
|
||||
.unwrap_or_else(|_| "http://localhost:9090".to_string())
|
||||
}
|
||||
|
||||
/// Load Grafana URL from environment
|
||||
pub fn grafana_url() -> String {
|
||||
std::env::var("GRAFANA_URL")
|
||||
.unwrap_or_else(|_| "http://localhost:3000".to_string())
|
||||
}
|
||||
|
||||
/// Run a test with timeout
|
||||
pub async fn with_timeout<F, T>(future: F) -> Result<T, Box<dyn std::error::Error>>
|
||||
where
|
||||
F: std::future::Future<Output = Result<T, Box<dyn std::error::Error>>>,
|
||||
{
|
||||
match timeout(SMOKE_TEST_TIMEOUT, future).await {
|
||||
Ok(result) => result,
|
||||
Err(_) => Err("Test timed out after 10 seconds".into()),
|
||||
}
|
||||
}
|
||||
|
||||
/// Skip test if service is not available
|
||||
/// This allows tests to be skipped gracefully when services are down
|
||||
#[macro_export]
|
||||
macro_rules! skip_if_unavailable {
|
||||
($service:expr, $test:block) => {
|
||||
match $test {
|
||||
Ok(result) => result,
|
||||
Err(e) if e.to_string().contains("connection refused") ||
|
||||
e.to_string().contains("connection reset") ||
|
||||
e.to_string().contains("timed out") => {
|
||||
eprintln!("⏭️ Skipping test - {} not available: {}", $service, e);
|
||||
return;
|
||||
}
|
||||
Err(e) => panic!("Test failed: {}", e),
|
||||
}
|
||||
};
|
||||
}
|
||||
}
|
||||
265
tests/smoke_tests/service_health.rs
Normal file
265
tests/smoke_tests/service_health.rs
Normal file
@@ -0,0 +1,265 @@
|
||||
//! Service Health Smoke Tests
|
||||
//!
|
||||
//! Tests to validate that gRPC services are running and responsive.
|
||||
//! Uses gRPC health check protocol to verify service availability.
|
||||
|
||||
use super::common::*;
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_trading_service_health() {
|
||||
let result = with_timeout(async {
|
||||
let url = trading_service_url();
|
||||
println!("Testing Trading Service at {}", url);
|
||||
|
||||
// Try HTTP health endpoint first (faster)
|
||||
let client = reqwest::Client::new();
|
||||
let http_health_url = url.replace("50052", "8080").replace("http://", "http://");
|
||||
|
||||
match client.get(format!("{}/health", http_health_url)).send().await {
|
||||
Ok(response) if response.status().is_success() => {
|
||||
println!("✅ Trading Service HTTP health check passed");
|
||||
return Ok(());
|
||||
}
|
||||
_ => {
|
||||
println!(" HTTP health check not available, trying gRPC...");
|
||||
}
|
||||
}
|
||||
|
||||
// Fallback to gRPC connection test
|
||||
// Note: We can't use tonic_health without the proto definitions,
|
||||
// so we'll just test if we can connect
|
||||
match tonic::transport::Channel::from_shared(url.clone()) {
|
||||
Ok(endpoint) => {
|
||||
match endpoint.connect().await {
|
||||
Ok(_channel) => {
|
||||
println!("✅ Trading Service gRPC connection successful");
|
||||
Ok(())
|
||||
}
|
||||
Err(e) => Err(Box::new(e) as Box<dyn std::error::Error>),
|
||||
}
|
||||
}
|
||||
Err(e) => Err(Box::new(e) as Box<dyn std::error::Error>),
|
||||
}
|
||||
}).await;
|
||||
|
||||
skip_if_unavailable!("Trading Service", { result });
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_api_gateway_health() {
|
||||
let result = with_timeout(async {
|
||||
let url = api_gateway_url();
|
||||
println!("Testing API Gateway at {}", url);
|
||||
|
||||
// Try gRPC connection
|
||||
match tonic::transport::Channel::from_shared(url.clone()) {
|
||||
Ok(endpoint) => {
|
||||
match endpoint.connect().await {
|
||||
Ok(_channel) => {
|
||||
println!("✅ API Gateway gRPC connection successful");
|
||||
Ok(())
|
||||
}
|
||||
Err(e) => Err(Box::new(e) as Box<dyn std::error::Error>),
|
||||
}
|
||||
}
|
||||
Err(e) => Err(Box::new(e) as Box<dyn std::error::Error>),
|
||||
}
|
||||
}).await;
|
||||
|
||||
skip_if_unavailable!("API Gateway", { result });
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[ignore = "Backtesting Service may not be available due to configuration issues"]
|
||||
async fn test_backtesting_service_health() {
|
||||
let result = with_timeout(async {
|
||||
let url = backtesting_service_url();
|
||||
println!("Testing Backtesting Service at {}", url);
|
||||
|
||||
match tonic::transport::Channel::from_shared(url.clone()) {
|
||||
Ok(endpoint) => {
|
||||
match endpoint.connect().await {
|
||||
Ok(_channel) => {
|
||||
println!("✅ Backtesting Service gRPC connection successful");
|
||||
Ok(())
|
||||
}
|
||||
Err(e) => Err(Box::new(e) as Box<dyn std::error::Error>),
|
||||
}
|
||||
}
|
||||
Err(e) => Err(Box::new(e) as Box<dyn std::error::Error>),
|
||||
}
|
||||
}).await;
|
||||
|
||||
skip_if_unavailable!("Backtesting Service", { result });
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[ignore = "ML Training Service may not be available due to configuration issues"]
|
||||
async fn test_ml_training_service_health() {
|
||||
let result = with_timeout(async {
|
||||
let url = ml_training_service_url();
|
||||
println!("Testing ML Training Service at {}", url);
|
||||
|
||||
match tonic::transport::Channel::from_shared(url.clone()) {
|
||||
Ok(endpoint) => {
|
||||
match endpoint.connect().await {
|
||||
Ok(_channel) => {
|
||||
println!("✅ ML Training Service gRPC connection successful");
|
||||
Ok(())
|
||||
}
|
||||
Err(e) => Err(Box::new(e) as Box<dyn std::error::Error>),
|
||||
}
|
||||
}
|
||||
Err(e) => Err(Box::new(e) as Box<dyn std::error::Error>),
|
||||
}
|
||||
}).await;
|
||||
|
||||
skip_if_unavailable!("ML Training Service", { result });
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_service_ports_listening() {
|
||||
println!("🔍 Checking service ports...");
|
||||
|
||||
let ports_to_check = vec![
|
||||
(50051, "API Gateway"),
|
||||
(50052, "Trading Service"),
|
||||
(50053, "Backtesting Service"),
|
||||
(50054, "ML Training Service"),
|
||||
];
|
||||
|
||||
let mut listening = 0;
|
||||
let mut not_listening = 0;
|
||||
|
||||
for (port, service) in ports_to_check {
|
||||
match std::net::TcpStream::connect(format!("localhost:{}", port)) {
|
||||
Ok(_) => {
|
||||
println!(" ✓ Port {} ({}) is listening", port, service);
|
||||
listening += 1;
|
||||
}
|
||||
Err(_) => {
|
||||
println!(" ✗ Port {} ({}) is not listening", port, service);
|
||||
not_listening += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
println!("\n📊 Service Port Summary:");
|
||||
println!(" Listening: {}", listening);
|
||||
println!(" Not listening: {}", not_listening);
|
||||
|
||||
// Require at least Trading Service to be up
|
||||
assert!(listening >= 1, "No services are listening");
|
||||
println!("✅ Service port check complete");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_service_response_times() {
|
||||
println!("⏱️ Measuring service response times...");
|
||||
|
||||
let services = vec![
|
||||
(trading_service_url(), "Trading Service"),
|
||||
(api_gateway_url(), "API Gateway"),
|
||||
];
|
||||
|
||||
for (url, name) in services {
|
||||
let start = std::time::Instant::now();
|
||||
|
||||
match tonic::transport::Channel::from_shared(url.clone()) {
|
||||
Ok(endpoint) => {
|
||||
match endpoint.connect().await {
|
||||
Ok(_channel) => {
|
||||
let elapsed = start.elapsed();
|
||||
println!(" ✓ {} connected in {:?}", name, elapsed);
|
||||
|
||||
// Warn if connection takes too long
|
||||
if elapsed.as_millis() > 1000 {
|
||||
println!(" ⚠ Connection took longer than expected");
|
||||
}
|
||||
}
|
||||
Err(_) => {
|
||||
println!(" ⚠ {} not available", name);
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(_) => {
|
||||
println!(" ⚠ {} invalid URL", name);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
println!("✅ Service response time check complete");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_metrics_endpoints() {
|
||||
println!("📊 Checking metrics endpoints...");
|
||||
|
||||
let client = reqwest::Client::new();
|
||||
let metrics_ports = vec![
|
||||
(9091, "API Gateway"),
|
||||
(9092, "Trading Service"),
|
||||
(9093, "Backtesting Service"),
|
||||
(9094, "ML Training Service"),
|
||||
];
|
||||
|
||||
let mut available = 0;
|
||||
let mut unavailable = 0;
|
||||
|
||||
for (port, service) in metrics_ports {
|
||||
let url = format!("http://localhost:{}/metrics", port);
|
||||
|
||||
match client.get(&url).send().await {
|
||||
Ok(response) if response.status().is_success() => {
|
||||
println!(" ✓ {} metrics available at port {}", service, port);
|
||||
available += 1;
|
||||
|
||||
// Verify Prometheus format
|
||||
if let Ok(text) = response.text().await {
|
||||
assert!(
|
||||
text.contains("# HELP") || text.contains("# TYPE"),
|
||||
"{} metrics not in Prometheus format",
|
||||
service
|
||||
);
|
||||
}
|
||||
}
|
||||
_ => {
|
||||
println!(" ⚠ {} metrics not available at port {}", service, port);
|
||||
unavailable += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
println!("\n📊 Metrics Endpoint Summary:");
|
||||
println!(" Available: {}", available);
|
||||
println!(" Unavailable: {}", unavailable);
|
||||
|
||||
println!("✅ Metrics endpoint check complete");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_service_versions() {
|
||||
println!("🔖 Checking service versions...");
|
||||
|
||||
// Try to get version info from Trading Service health endpoint
|
||||
let client = reqwest::Client::new();
|
||||
let health_url = "http://localhost:8080/health";
|
||||
|
||||
match client.get(health_url).send().await {
|
||||
Ok(response) if response.status().is_success() => {
|
||||
if let Ok(json) = response.json::<serde_json::Value>().await {
|
||||
if let Some(version) = json.get("version") {
|
||||
println!(" ✓ Trading Service version: {}", version);
|
||||
}
|
||||
if let Some(service) = json.get("service") {
|
||||
println!(" ✓ Service name: {}", service);
|
||||
}
|
||||
}
|
||||
}
|
||||
_ => {
|
||||
println!(" ⚠ Could not retrieve version information");
|
||||
}
|
||||
}
|
||||
|
||||
println!("✅ Service version check complete");
|
||||
}
|
||||
Reference in New Issue
Block a user