foxhunt/docs/WAVE79_AGENT9_CHECKLIST.md

Wave 79 Agent 9: Quick Reference Checklist

For next agent to retry load testing

---

Prerequisites Checklist

Before Running Load Tests

• GetOrderStatus issue resolved (Priority 1)

  • gRPC reflection enabled on trading_service
  • Service rebuilt and restarted
  • Tested with single request: grpcurl localhost:50050 list trading.TradingService
  • Confirmed GetOrderStatus is registered

• TLS certificates available (Priority 2)

  • CA cert exists: certs/production/ca/ca-cert.pem
  • Client cert exists: certs/production/foxhunt-cert.pem
  • Client key exists: certs/production/foxhunt-key.pem
  • Certificates valid: openssl x509 -in certs/production/foxhunt-cert.pem -noout -dates

• Services running

  • Trading Service on port 50050
  • Backtesting Service on port 50052
  • ML Training Service on port 50053
  • Check: netstat -tlnp | grep -E ":(50050|50052|50053)"

• Load testing tool installed

  • ghz available: which ghz
  • If not: go install github.com/bojand/ghz/cmd/ghz@latest

---

Quick Verification Tests

1. Test gRPC Reflection (Should Work First)

# Test if service is accessible
grpcurl -plaintext localhost:50050 list

# Expected: List of services or "server does not support reflection"
# If no reflection, skip to next test

2. Test Basic Connectivity with TLS

ghz --cacert certs/production/ca/ca-cert.pem \
    --cert certs/production/foxhunt-cert.pem \
    --key certs/production/foxhunt-key.pem \
    --proto services/trading_service/proto/trading.proto \
    --call trading.TradingService.GetOrderStatus \
    --connections 1 --concurrency 1 --total 1 \
    --data '{"order_id": "test"}' \
    localhost:50050

# Expected: 1 successful request, NOT "Unimplemented"
# If "Unimplemented": STOP, GetOrderStatus issue not fixed
# If "Unavailable": Check TLS certificates
# If "Unauthenticated": Add JWT token to metadata

3. Test Low-Rate Success

ghz --cacert certs/production/ca/ca-cert.pem \
    --cert certs/production/foxhunt-cert.pem \
    --key certs/production/foxhunt-key.pem \
    --proto services/trading_service/proto/trading.proto \
    --call trading.TradingService.GetOrderStatus \
    --connections 10 --concurrency 10 --rps 100 --duration 10s \
    --data '{"order_id": "test"}' \
    localhost:50050

# Expected: >90% success rate, ~100 req/s
# If failures: Check error messages

---

Run Full Load Test Suite

Option 1: Using Script (Recommended)

# Run all 4 tests automatically
./scripts/load_test_wave79.sh

# Results will be in: load_test_results/wave79/

Option 2: Manual Execution

# Test 1: Normal Load (1K concurrent, 60s)
ghz --cacert certs/production/ca/ca-cert.pem \
    --cert certs/production/foxhunt-cert.pem \
    --key certs/production/foxhunt-key.pem \
    --proto services/trading_service/proto/trading.proto \
    --call trading.TradingService.GetOrderStatus \
    --duration 60s --rps 100000 --connections 1000 --concurrency 1000 \
    --data '{"order_id": "test"}' \
    localhost:50050 > load_test_results/wave79/test1_normal_load.json

# Test 2: Spike Load (10K concurrent, 30s)
ghz --cacert certs/production/ca/ca-cert.pem \
    --cert certs/production/foxhunt-cert.pem \
    --key certs/production/foxhunt-key.pem \
    --proto services/trading_service/proto/trading.proto \
    --call trading.TradingService.GetOrderStatus \
    --duration 30s --rps 200000 --connections 10000 --concurrency 10000 \
    --data '{"order_id": "test"}' \
    localhost:50050 > load_test_results/wave79/test2_spike_load.json

# Test 3: Stress Test (10K concurrent, 60s)
ghz --cacert certs/production/ca/ca-cert.pem \
    --cert certs/production/foxhunt-cert.pem \
    --key certs/production/foxhunt-key.pem \
    --proto services/trading_service/proto/trading.proto \
    --call trading.TradingService.GetOrderStatus \
    --duration 60s --rps 200000 --connections 10000 --concurrency 10000 \
    --data '{"order_id": "test"}' \
    localhost:50050 > load_test_results/wave79/test3_stress_test.json

# Test 4: Sustained Load (5K concurrent, 5 minutes)
ghz --cacert certs/production/ca/ca-cert.pem \
    --cert certs/production/foxhunt-cert.pem \
    --key certs/production/foxhunt-key.pem \
    --proto services/trading_service/proto/trading.proto \
    --call trading.TradingService.GetOrderStatus \
    --duration 300s --rps 150000 --connections 5000 --concurrency 5000 \
    --data '{"order_id": "test"}' \
    localhost:50050 > load_test_results/wave79/test4_sustained_load.json

---

Success Criteria

Throughput

• Test 1: ≥100,000 req/s
• Test 2: ≥150,000 req/s (spike handling)
• Test 3: ≥200,000 req/s (stress test)
• Test 4: ≥150,000 req/s sustained (5 min)

Error Rate

• All tests: <0.1% error rate
• No "Unimplemented" errors
• No "Unavailable" errors
• No TLS handshake failures

HTTP/2 Health

• Check service logs for warnings:

tail -100 /tmp/trading_service.log | grep -i "stream.*error\|stream.*exhausted"
# Expected: No matches

Latency

• P99 latency <10μs (10,000 ns) for auth pipeline
• P50 latency <1μs (1,000 ns)
• Check in ghz output: "Latency distribution"

---

Troubleshooting

"Unimplemented" Errors (99.9%+)

1. Check if service has latest code:

   strings target/release/trading_service | grep GetOrderStatus
   

2. Enable reflection and test:

   // Add to services/trading_service/src/main.rs
   use tonic_reflection::server::Builder;
   let reflection = Builder::configure()
       .register_encoded_file_descriptor_set(...)
       .build()?;
   Server::builder().add_service(reflection)...
   

3. Rebuild and restart:

   cargo build --release --bin trading_service
   pkill trading_service
   ./target/release/trading_service &
   

"Unavailable" / Connection Reset

• TLS mismatch: Verify using TLS flags (not --insecure)
• Port wrong: Check port with netstat -tlnp | grep 50050
• Service crashed: Check ps aux | grep trading_service
• Rate limited: Lower RPS and retry

"Unauthenticated" Errors

• Need JWT token: Add to metadata:

  --metadata '{"authorization":"Bearer <valid-jwt>"}'
  

• Generate test token: See docs/WAVE79_AGENT9_INVESTIGATION.md

High Latency (>10μs)

• Check CPU usage: top -p $(pidof trading_service)
• Check for rate limiting in logs
• Verify HTTP/2 settings in service logs
• Test with lower concurrency

---

Documentation to Update

After successful test run:

1. WAVE79_AGENT9_LOAD_TEST_RESULTS.md

   • Update all test results
   • Replace "N/A" with actual measurements
   • Update status codes from 🔴 to ✅

2. WAVE79_AGENT9_SUMMARY.md

   • Change status from "BLOCKED" to "COMPLETE"
   • Update comparison table with Wave 78
   • Add final performance metrics

3. Create new document:

   • WAVE79_LOAD_TEST_FINAL.md with comparison to Wave 78
   • Include all 4 test results
   • Performance analysis
   • Recommendations for Wave 80

---

Quick Commands Reference

# Check services running
ps aux | grep -E "trading_service|backtesting_service|ml_training_service" | grep -v grep

# Check ports listening
netstat -tlnp | grep -E ":(50050|50052|50053)"

# View service logs
tail -f /tmp/trading_service.log
tail -f /tmp/ml_training_service.log
tail -f logs/backtesting_service.log

# Test single request
ghz --cacert certs/production/ca/ca-cert.pem \
    --cert certs/production/foxhunt-cert.pem \
    --key certs/production/foxhunt-key.pem \
    --proto services/trading_service/proto/trading.proto \
    --call trading.TradingService.GetOrderStatus \
    --total 1 --data '{"order_id": "test"}' \
    localhost:50050

# Run full test suite
./scripts/load_test_wave79.sh

# Analyze results
grep "Requests/sec:" load_test_results/wave79/*.json
grep "Status code distribution:" -A 5 load_test_results/wave79/*.json

---

Wave 78 Baseline (For Comparison)

Throughput: 211,000 req/s
Error Rate: Unknown (likely <0.1%)
HTTP/2: Had warnings (stream exhaustion)
TLS: Unknown error rate

Target: Maintain ≥200K req/s with <0.1% errors and no HTTP/2 warnings

---

Contact / Escalation

If issues persist:

1. Review docs/WAVE79_AGENT9_INVESTIGATION.md for detailed analysis
2. Check auth interceptor: services/trading_service/src/auth_interceptor.rs
3. Enable debug logging: RUST_LOG=debug ./target/release/trading_service
4. Escalate to team for GetOrderStatus root cause investigation