foxhunt/STRESS_TEST_QUICK_REFERENCE.md

Stress Test Quick Reference

Last Updated: 2025-10-15 (Agent 21)
Status: ✅ 14/14 tests passing
Duration: 62.78 seconds

---

Quick Commands

Run All Stress Tests

cargo test -p stress_tests --test chaos_testing -- --test-threads=1

Run Specific Test

cargo test -p stress_tests --test chaos_testing test_database_connection_pool_exhaustion -- --nocapture

Run with Full Output

cargo test -p stress_tests --test chaos_testing -- --test-threads=1 --nocapture

---

Test Suite Overview

#	Test	Duration	What It Tests
1	cascade_failure	~6s	Multi-component failure cascade
2	circuit_breaker_behavior	~1s	Circuit breaker activation
3	data_consistency_during_failure	~3s	Data integrity during outages
4	database_connection_loss	~4s	DB reconnection logic
5	database_connection_pool_exhaustion	~1s	Pool under heavy load
6	extreme_network_latency	~13s	5s latency spike handling
7	full_system_resource_exhaustion	~4s	Simultaneous multi-resource failure
8	graceful_degradation	~1s	Operating without cache
9	memory_pressure	~1s	Redis 50% fill handling
10	network_partition	~5s	2s network split recovery
11	redis_cache_failure	~1s	Cache flush recovery
12	redis_cache_failure_cascade	~5s	Multi-stage Redis cascade
13	redis_connection_pool_exhaustion	~1s	Redis pool under load
14	uptime_sla_compliance	~18s	7 scenarios, 99.9% SLA

Total: ~63 seconds

---

Expected Results

Recovery Times (Target: <30s)

• Database Connection Loss: 4.01s ✅
• Redis Cache Failure: 1.02s ✅
• Network Partition: 5.00s ✅
• Cascade Failure: 6.02s ✅
• Full Resource Exhaustion: 4.02s ✅

Success Rates

• Overall: 100% (14/14 tests)
• Database Resilience: 100% (4/4 tests)
• Redis Resilience: 100% (5/5 tests)
• Network Resilience: 100% (3/3 tests)
• System-Wide: 100% (2/2 tests)

Pool Exhaustion Benchmarks

• Database: 100/100 concurrent queries completed ✅
• Redis: 50/50 concurrent operations completed ✅

---

Prerequisites

Docker Services Required

docker-compose ps postgres redis  # Must show "Up (healthy)"

Service URLs

• PostgreSQL: postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt
• Redis: redis://localhost:6379

If Tests Fail Due to Infrastructure

# Restart services
docker-compose restart postgres redis

# Check health
docker-compose ps

# Verify connectivity
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "SELECT 1;"
redis-cli -h localhost -p 6379 PING

---

Troubleshooting

Test Hangs or Times Out

Cause: Service not responding
Fix:

docker-compose restart postgres redis
docker-compose ps  # Verify healthy

Connection Refused Errors

Cause: Service not started
Fix:

docker-compose up -d postgres redis
sleep 5  # Wait for startup

Pool Exhaustion Test Fails

Expected: 90%+ completion rate (graceful handling)
If <90%: Check PostgreSQL connection pool settings in config/database.toml

Redis Memory Pressure Cleanup

If stress keys remain after test:

redis-cli -h localhost -p 6379 KEYS "stress_test_key_*" | xargs redis-cli -h localhost -p 6379 DEL

---

Test Categories

Database Resilience (4 tests)

1. test_database_connection_loss - 3s outage recovery
2. test_database_connection_pool_exhaustion - 100 concurrent queries
3. test_data_consistency_during_failure - Data integrity validation
4. (Included in cascade) - Slow query handling

Redis Cache Resilience (5 tests)

1. test_redis_cache_failure - FLUSHALL recovery
2. test_memory_pressure - 50% fill handling
3. test_redis_connection_pool_exhaustion - 50 concurrent ops
4. test_redis_cache_failure_cascade - Multi-stage cascade
5. (Included in graceful_degradation) - Operating without cache

Network Resilience (3 tests)

1. test_network_partition - 2s network split
2. test_extreme_network_latency - 5s latency spike
3. test_circuit_breaker_behavior - 3 failure threshold

System-Wide Resilience (2 tests)

1. test_cascade_failure - Redis + DB + Network simultaneous
2. test_full_system_resource_exhaustion - All resources stressed

---

Performance Expectations

Mean Recovery Time: 2.58s

P99 Recovery Time: 6.02s

Circuit Breaker Activation: Extreme scenarios only

Data Consistency: 100% maintained

Graceful Degradation: Confirmed in cache failures

---

Integration with CI/CD

GitHub Actions Workflow

- name: Run Stress Tests
  run: |
    docker-compose up -d postgres redis
    sleep 10  # Wait for services
    cargo test -p stress_tests --test chaos_testing -- --test-threads=1

Expected CI Duration

• Compilation: ~60s
• Test Execution: ~63s
• Total: ~2 minutes

---

Related Documentation

• Full Report: WAVE_3_AGENT_21_STRESS_TEST_VERIFICATION.md
• Agent 18 Implementation: Search git history for Agent 18 commits
• Fault Injectors: services/stress_tests/src/fault_injector.rs
• Test Implementation: services/stress_tests/tests/chaos_testing.rs

---

Key Metrics at a Glance

✅ 14/14 Tests Passing (100%)
✅ 62.78s Total Duration (under 3-min target)
✅ 2.58s Mean Recovery Time (92% faster than target)
✅ 100% Data Consistency
✅ 100% Pool Handling (DB: 100/100, Redis: 50/50)
✅ Circuit Breaker: Correctly activates for extreme conditions
✅ 99.9% Uptime SLA: Validated across 7 scenarios

---

Last Verified: 2025-10-15 by Agent 21
Status: Production Ready ✅