foxhunt/load-tests/IMMEDIATE_ACTIONS.md

# Immediate Actions - Load Test Follow-up

Based on Wave 78 Agent 5 load testing results (2025-10-03)

---

## Priority 1: HTTP/2 Stream Limit (15 minutes)

**Issue**: API Gateway hits 1,024 concurrent stream limit under heavy load

**Fix**:
```rust
// File: services/api_gateway/src/main.rs
// Add to server builder (around line 100):

Server::builder()
    .max_concurrent_streams(Some(10_000))  // Increase from default 1024
    .initial_connection_window_size(Some(10 * 1024 * 1024))  // 10MB
    .initial_stream_window_size(Some(1 * 1024 * 1024))  // 1MB
    .tcp_nodelay(true)  // Already enabled
    // ... rest of config
```

**Impact**: Eliminates "locally-reset streams reached limit" warnings

**Test**:
```bash
# Verify fix
ghz --insecure --proto services/api_gateway/proto/config_service.proto \
  --call foxhunt.config.ConfigurationService.ListConfigs \
  -d '{}' -c 10000 -z 30s localhost:50050

# Should see NO stream limit warnings in logs
tail -f logs/api_gateway.log | grep "locally-reset"
```

---

## Priority 2: Regenerate TLS Certificates with SAN (30 minutes)

**Issue**: Current certificates use legacy Common Name field, preventing direct backend service testing

**Fix**:
```bash
cd /home/jgrusewski/Work/foxhunt/certs/production

# Create script: generate_certs_with_san.sh
cat > generate_certs_with_san.sh << 'EOF'
#!/bin/bash
set -e

# Generate CA
openssl genrsa -out ca/ca-key.pem 4096
openssl req -new -x509 -days 3650 -key ca/ca-key.pem -out ca/ca-cert.pem \
  -subj "/CN=Foxhunt CA"

# Generate server certificate with SAN
openssl genrsa -out foxhunt-key.pem 4096
openssl req -new -key foxhunt-key.pem -out foxhunt-csr.pem \
  -subj "/CN=localhost"

# Create SAN config
cat > san.cnf << 'SANEOF'
[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req

[req_distinguished_name]

[v3_req]
subjectAltName = @alt_names

[alt_names]
DNS.1 = localhost
DNS.2 = api-gateway
DNS.3 = trading-service
DNS.4 = backtesting-service
DNS.5 = ml-training-service
IP.1 = 127.0.0.1
IP.2 = ::1
SANEOF

# Sign certificate with SAN
openssl x509 -req -days 365 -in foxhunt-csr.pem \
  -CA ca/ca-cert.pem -CAkey ca/ca-key.pem -CAcreateserial \
  -out foxhunt-cert.pem -extensions v3_req -extfile san.cnf

echo "Certificates generated with SAN support"
openssl x509 -in foxhunt-cert.pem -text -noout | grep -A1 "Subject Alternative Name"
EOF

chmod +x generate_certs_with_san.sh
./generate_certs_with_san.sh

# Backup old certs
mv foxhunt-cert.pem foxhunt-cert.pem.old
mv foxhunt-key.pem foxhunt-key.pem.old

# Restart services to pick up new certificates
pkill -HUP trading_service backtesting_service ml_training_service api_gateway
```

**Test**:
```bash
# Test direct connection with new certificates
ghz --proto services/ml_training_service/proto/ml_training.proto \
  --call ml_training.MLTrainingService.HealthCheck \
  --cacert certs/production/ca/ca-cert.pem \
  --cert certs/production/foxhunt-cert.pem \
  --key certs/production/foxhunt-key.pem \
  -d '{}' -c 100 -z 10s localhost:50053

# Should see successful connections, not certificate errors
```

---

## Priority 3: Add Performance Metrics Endpoint (1 hour)

**Issue**: No easy way to monitor performance during load tests

**Fix**:
```rust
// File: services/api_gateway/Cargo.toml
// Add dependency:
[dependencies]
prometheus = "0.13"
prometheus-hyper = "0.1"

// File: services/api_gateway/src/metrics.rs (new file)
use prometheus::{Encoder, TextEncoder, Registry, Counter, Histogram};
use std::sync::Arc;

pub struct Metrics {
    pub registry: Arc<Registry>,
    pub request_counter: Counter,
    pub latency_histogram: Histogram,
}

impl Metrics {
    pub fn new() -> Self {
        let registry = Arc::new(Registry::new());

        let request_counter = Counter::new("grpc_requests_total", "Total gRPC requests")
            .expect("Counter creation failed");
        let latency_histogram = Histogram::new("grpc_request_duration_seconds", "Request latency")
            .expect("Histogram creation failed");

        registry.register(Box::new(request_counter.clone())).unwrap();
        registry.register(Box::new(latency_histogram.clone())).unwrap();

        Self { registry, request_counter, latency_histogram }
    }
}

// File: services/api_gateway/src/main.rs
// Add metrics HTTP endpoint on separate port (9090)
```

**Test**:
```bash
# Query metrics during load test
curl http://localhost:9090/metrics

# Expected output:
# grpc_requests_total 1234567
# grpc_request_duration_seconds_bucket{le="0.001"} 1000000
# ...
```

---

## Priority 4: Document Load Testing Procedures (30 minutes)

**Create**: `docs/LOAD_TESTING_GUIDE.md`

**Contents**:
1. Prerequisites (ghz, certificates, running services)
2. Test scenarios (normal, spike, stress)
3. Monitoring commands (logs, metrics, resources)
4. Expected results and thresholds
5. Troubleshooting common issues

---

## Quick Win: Add to CI/CD (15 minutes)

**Create**: `.github/workflows/performance-tests.yml`

```yaml
name: Performance Tests

on:
  push:
    branches: [main]
  pull_request:

jobs:
  load-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Install ghz
        run: |
          curl -sL https://github.com/bojand/ghz/releases/download/v0.120.0/ghz-linux-x86_64.tar.gz | tar xz
          sudo mv ghz /usr/local/bin/

      - name: Start services
        run: |
          docker-compose up -d
          sleep 10

      - name: Run load tests
        run: |
          ghz --insecure \
            --proto services/api_gateway/proto/config_service.proto \
            --call foxhunt.config.ConfigurationService.ListConfigs \
            -d '{}' -c 1000 -z 30s localhost:50050 \
            > load-test-results.txt

      - name: Check thresholds
        run: |
          # Fail if throughput < 20K req/s
          RPS=$(grep "Requests/sec:" load-test-results.txt | awk '{print $2}')
          if (( $(echo "$RPS < 20000" | bc -l) )); then
            echo "FAIL: Throughput $RPS < 20K req/s"
            exit 1
          fi
```

---

## Timeline

| Priority | Task | Time | Owner |
|----------|------|------|-------|
| P1 | HTTP/2 stream limit fix | 15 min | Backend team |
| P2 | Regenerate TLS certificates | 30 min | DevOps |
| P3 | Add metrics endpoint | 1 hour | Backend team |
| P4 | Documentation | 30 min | Tech writer |
| - | CI/CD integration | 15 min | DevOps |

**Total effort**: ~2.5 hours
**Expected completion**: Same day

---

## Validation Checklist

After implementing fixes:

- [ ] HTTP/2 stream limit warnings eliminated at 10K concurrency
- [ ] Direct backend service testing works with new certificates
- [ ] Metrics endpoint accessible at `:9090/metrics`
- [ ] Load testing guide in `docs/` directory
- [ ] CI/CD pipeline passes with performance thresholds
- [ ] All services restart cleanly with new certificates
- [ ] Throughput still >200K req/s after changes

---

**Questions?** Contact Wave 78 Agent 5 or review full report in `docs/WAVE78_AGENT5_LOAD_TEST_RESULTS.md`