foxhunt/docs/WAVE79_AGENT9_CHECKLIST.md

# Wave 79 Agent 9: Quick Reference Checklist

**For next agent to retry load testing**

---

## Prerequisites Checklist

### Before Running Load Tests

- [ ] **GetOrderStatus issue resolved** (Priority 1)
  - [ ] gRPC reflection enabled on trading_service
  - [ ] Service rebuilt and restarted
  - [ ] Tested with single request: `grpcurl localhost:50050 list trading.TradingService`
  - [ ] Confirmed GetOrderStatus is registered

- [ ] **TLS certificates available** (Priority 2)
  - [ ] CA cert exists: `certs/production/ca/ca-cert.pem`
  - [ ] Client cert exists: `certs/production/foxhunt-cert.pem`
  - [ ] Client key exists: `certs/production/foxhunt-key.pem`
  - [ ] Certificates valid: `openssl x509 -in certs/production/foxhunt-cert.pem -noout -dates`

- [ ] **Services running**
  - [ ] Trading Service on port 50050
  - [ ] Backtesting Service on port 50052
  - [ ] ML Training Service on port 50053
  - [ ] Check: `netstat -tlnp | grep -E ":(50050|50052|50053)"`

- [ ] **Load testing tool installed**
  - [ ] ghz available: `which ghz`
  - [ ] If not: `go install github.com/bojand/ghz/cmd/ghz@latest`

---

## Quick Verification Tests

### 1. Test gRPC Reflection (Should Work First)

```bash
# Test if service is accessible
grpcurl -plaintext localhost:50050 list

# Expected: List of services or "server does not support reflection"
# If no reflection, skip to next test
```

### 2. Test Basic Connectivity with TLS

```bash
ghz --cacert certs/production/ca/ca-cert.pem \
    --cert certs/production/foxhunt-cert.pem \
    --key certs/production/foxhunt-key.pem \
    --proto services/trading_service/proto/trading.proto \
    --call trading.TradingService.GetOrderStatus \
    --connections 1 --concurrency 1 --total 1 \
    --data '{"order_id": "test"}' \
    localhost:50050

# Expected: 1 successful request, NOT "Unimplemented"
# If "Unimplemented": STOP, GetOrderStatus issue not fixed
# If "Unavailable": Check TLS certificates
# If "Unauthenticated": Add JWT token to metadata
```

### 3. Test Low-Rate Success

```bash
ghz --cacert certs/production/ca/ca-cert.pem \
    --cert certs/production/foxhunt-cert.pem \
    --key certs/production/foxhunt-key.pem \
    --proto services/trading_service/proto/trading.proto \
    --call trading.TradingService.GetOrderStatus \
    --connections 10 --concurrency 10 --rps 100 --duration 10s \
    --data '{"order_id": "test"}' \
    localhost:50050

# Expected: >90% success rate, ~100 req/s
# If failures: Check error messages
```

---

## Run Full Load Test Suite

### Option 1: Using Script (Recommended)

```bash
# Run all 4 tests automatically
./scripts/load_test_wave79.sh

# Results will be in: load_test_results/wave79/
```

### Option 2: Manual Execution

```bash
# Test 1: Normal Load (1K concurrent, 60s)
ghz --cacert certs/production/ca/ca-cert.pem \
    --cert certs/production/foxhunt-cert.pem \
    --key certs/production/foxhunt-key.pem \
    --proto services/trading_service/proto/trading.proto \
    --call trading.TradingService.GetOrderStatus \
    --duration 60s --rps 100000 --connections 1000 --concurrency 1000 \
    --data '{"order_id": "test"}' \
    localhost:50050 > load_test_results/wave79/test1_normal_load.json

# Test 2: Spike Load (10K concurrent, 30s)
ghz --cacert certs/production/ca/ca-cert.pem \
    --cert certs/production/foxhunt-cert.pem \
    --key certs/production/foxhunt-key.pem \
    --proto services/trading_service/proto/trading.proto \
    --call trading.TradingService.GetOrderStatus \
    --duration 30s --rps 200000 --connections 10000 --concurrency 10000 \
    --data '{"order_id": "test"}' \
    localhost:50050 > load_test_results/wave79/test2_spike_load.json

# Test 3: Stress Test (10K concurrent, 60s)
ghz --cacert certs/production/ca/ca-cert.pem \
    --cert certs/production/foxhunt-cert.pem \
    --key certs/production/foxhunt-key.pem \
    --proto services/trading_service/proto/trading.proto \
    --call trading.TradingService.GetOrderStatus \
    --duration 60s --rps 200000 --connections 10000 --concurrency 10000 \
    --data '{"order_id": "test"}' \
    localhost:50050 > load_test_results/wave79/test3_stress_test.json

# Test 4: Sustained Load (5K concurrent, 5 minutes)
ghz --cacert certs/production/ca/ca-cert.pem \
    --cert certs/production/foxhunt-cert.pem \
    --key certs/production/foxhunt-key.pem \
    --proto services/trading_service/proto/trading.proto \
    --call trading.TradingService.GetOrderStatus \
    --duration 300s --rps 150000 --connections 5000 --concurrency 5000 \
    --data '{"order_id": "test"}' \
    localhost:50050 > load_test_results/wave79/test4_sustained_load.json
```

---

## Success Criteria

### Throughput
- [ ] Test 1: ≥100,000 req/s
- [ ] Test 2: ≥150,000 req/s (spike handling)
- [ ] Test 3: ≥200,000 req/s (stress test)
- [ ] Test 4: ≥150,000 req/s sustained (5 min)

### Error Rate
- [ ] All tests: <0.1% error rate
- [ ] No "Unimplemented" errors
- [ ] No "Unavailable" errors
- [ ] No TLS handshake failures

### HTTP/2 Health
- [ ] Check service logs for warnings:
```bash
tail -100 /tmp/trading_service.log | grep -i "stream.*error\|stream.*exhausted"
# Expected: No matches
```

### Latency
- [ ] P99 latency <10μs (10,000 ns) for auth pipeline
- [ ] P50 latency <1μs (1,000 ns)
- [ ] Check in ghz output: "Latency distribution"

---

## Troubleshooting

### "Unimplemented" Errors (99.9%+)
1. Check if service has latest code:
   ```bash
   strings target/release/trading_service | grep GetOrderStatus
   ```

2. Enable reflection and test:
   ```rust
   // Add to services/trading_service/src/main.rs
   use tonic_reflection::server::Builder;
   let reflection = Builder::configure()
       .register_encoded_file_descriptor_set(...)
       .build()?;
   Server::builder().add_service(reflection)...
   ```

3. Rebuild and restart:
   ```bash
   cargo build --release --bin trading_service
   pkill trading_service
   ./target/release/trading_service &
   ```

### "Unavailable" / Connection Reset
- **TLS mismatch**: Verify using TLS flags (not `--insecure`)
- **Port wrong**: Check port with `netstat -tlnp | grep 50050`
- **Service crashed**: Check `ps aux | grep trading_service`
- **Rate limited**: Lower RPS and retry

### "Unauthenticated" Errors
- **Need JWT token**: Add to metadata:
  ```bash
  --metadata '{"authorization":"Bearer <valid-jwt>"}'
  ```
- **Generate test token**: See `docs/WAVE79_AGENT9_INVESTIGATION.md`

### High Latency (>10μs)
- Check CPU usage: `top -p $(pidof trading_service)`
- Check for rate limiting in logs
- Verify HTTP/2 settings in service logs
- Test with lower concurrency

---

## Documentation to Update

After successful test run:

1. **WAVE79_AGENT9_LOAD_TEST_RESULTS.md**
   - Update all test results
   - Replace "N/A" with actual measurements
   - Update status codes from 🔴 to ✅

2. **WAVE79_AGENT9_SUMMARY.md**
   - Change status from "BLOCKED" to "COMPLETE"
   - Update comparison table with Wave 78
   - Add final performance metrics

3. **Create new document**:
   - `WAVE79_LOAD_TEST_FINAL.md` with comparison to Wave 78
   - Include all 4 test results
   - Performance analysis
   - Recommendations for Wave 80

---

## Quick Commands Reference

```bash
# Check services running
ps aux | grep -E "trading_service|backtesting_service|ml_training_service" | grep -v grep

# Check ports listening
netstat -tlnp | grep -E ":(50050|50052|50053)"

# View service logs
tail -f /tmp/trading_service.log
tail -f /tmp/ml_training_service.log
tail -f logs/backtesting_service.log

# Test single request
ghz --cacert certs/production/ca/ca-cert.pem \
    --cert certs/production/foxhunt-cert.pem \
    --key certs/production/foxhunt-key.pem \
    --proto services/trading_service/proto/trading.proto \
    --call trading.TradingService.GetOrderStatus \
    --total 1 --data '{"order_id": "test"}' \
    localhost:50050

# Run full test suite
./scripts/load_test_wave79.sh

# Analyze results
grep "Requests/sec:" load_test_results/wave79/*.json
grep "Status code distribution:" -A 5 load_test_results/wave79/*.json
```

---

## Wave 78 Baseline (For Comparison)

```
Throughput: 211,000 req/s
Error Rate: Unknown (likely <0.1%)
HTTP/2: Had warnings (stream exhaustion)
TLS: Unknown error rate
```

**Target**: Maintain ≥200K req/s with <0.1% errors and no HTTP/2 warnings

---

## Contact / Escalation

If issues persist:
1. Review `docs/WAVE79_AGENT9_INVESTIGATION.md` for detailed analysis
2. Check auth interceptor: `services/trading_service/src/auth_interceptor.rs`
3. Enable debug logging: `RUST_LOG=debug ./target/release/trading_service`
4. Escalate to team for GetOrderStatus root cause investigation