foxhunt/WAVE_157_CERTIFICATE_FIX_REPORT.md

# Wave 157: Certificate Paths and E2E Test Report

**Date**: 2025-10-13
**Status**: ✅ DIRECT TLS CONNECTIVITY WORKING | ⚠️ API GATEWAY BLOCKED BY CERTIFICATE SANs

---

## Executive Summary

### Successes ✅
1. **Certificate paths fixed** in E2E test (hardcoded `/tmp/foxhunt/certs` → repository-relative paths)
2. **Direct TLS connectivity test PASSING** (test_ml_training_tls_connectivity)
   - TLS handshake: ✅ Successful
   - mTLS authentication: ✅ Successful
   - Health check RPC: ✅ Successful (7.86ms latency)
   - Certificate chain: ✅ Valid

### Blockers ⚠️
1. **API Gateway proxy test FAILING** (test_ml_training_tls_via_api_gateway)
   - **Root cause**: Certificate missing `ml_training_service` in SANs
   - **Impact**: API Gateway cannot establish TLS connection to ML Training Service
   - **Error**: "transport error" during TLS handshake

---

## Technical Details

### 1. Certificate Location Analysis

**Repository structure**:
```
/home/jgrusewski/Work/foxhunt/
├── certs/
│   ├── ca/
│   │   ├── ca-cert.pem         ✅ Valid (2017 bytes)
│   │   └── ca-key.pem
│   ├── server-cert.pem         ✅ Valid (signed by CA)
│   ├── server-key.pem
│   ├── client-cert.pem         ✅ Valid
│   └── client-key.pem
```

**Certificate verification**:
```bash
$ openssl verify -CAfile certs/ca/ca-cert.pem certs/server-cert.pem
certs/server-cert.pem: OK

$ openssl x509 -in certs/server-cert.pem -noout -subject -ext subjectAltName
subject=C = US, ST = NY, L = NewYork, O = Foxhunt, OU = HFT, CN = foxhunt-services
X509v3 Subject Alternative Name:
    DNS:foxhunt-services, DNS:backtesting_service, DNS:localhost, IP Address:127.0.0.1
```

**CRITICAL FINDING**: Certificate SANs missing `ml_training_service`!

### 2. Docker Container Paths

**Volume mounts** (docker-compose.yml):
```yaml
volumes:
  - ./certs:/tmp/foxhunt/certs:ro  # ✅ Correct
```

**Inside containers**: `/tmp/foxhunt/certs/*`
**On host machine**: `/home/jgrusewski/Work/foxhunt/certs/*`

**Verdict**: Docker volume mounts are correct. No changes needed.

### 3. E2E Test Fixes Applied

**File**: `tests/e2e/tests/ml_training_tls_test.rs`

**Before** (lines 37-44):
```rust
let ca_cert_path = std::env::var("ML_TRAINING_TLS_CA_CERT")
    .unwrap_or_else(|_| "/tmp/foxhunt/certs/ca/ca-cert.pem".to_string());

let client_cert_path = std::env::var("ML_TRAINING_TLS_CLIENT_CERT")
    .unwrap_or_else(|_| "/tmp/foxhunt/certs/client-cert.pem".to_string());

let client_key_path = std::env::var("ML_TRAINING_TLS_CLIENT_KEY")
    .unwrap_or_else(|_| "/tmp/foxhunt/certs/client-key.pem".to_string());
```

**After** (lines 37-51):
```rust
// Default to repository certs directory (for host-based E2E tests)
// Note: Docker containers use /tmp/foxhunt/certs via volume mount,
// but E2E tests run on the host and need the repository path
let default_ca_cert = format!("{}/certs/ca/ca-cert.pem", env!("CARGO_MANIFEST_DIR").replace("/tests/e2e", ""));
let default_client_cert = format!("{}/certs/client-cert.pem", env!("CARGO_MANIFEST_DIR").replace("/tests/e2e", ""));
let default_client_key = format!("{}/certs/client-key.pem", env!("CARGO_MANIFEST_DIR").replace("/tests/e2e", ""));

let ca_cert_path = std::env::var("ML_TRAINING_TLS_CA_CERT")
    .unwrap_or(default_ca_cert);

let client_cert_path = std::env::var("ML_TRAINING_TLS_CLIENT_CERT")
    .unwrap_or(default_client_cert);

let client_key_path = std::env::var("ML_TRAINING_TLS_CLIENT_KEY")
    .unwrap_or(default_client_key);
```

**Cargo.toml** fix:
```toml
# Added TLS features to tonic
tonic = { version = "0.14", features = ["transport", "tls-ring", "tls-webpki-roots"] }
```

### 4. Test Results

#### Test 1: Direct TLS Connectivity ✅ PASSING

```bash
$ cargo test --test ml_training_tls_test test_ml_training_tls_connectivity -- --nocapture

running 1 test
INFO Starting ML Training TLS connectivity test
INFO ML Service URL: https://localhost:50054
INFO CA Cert: /home/jgrusewski/Work/foxhunt/certs/ca/ca-cert.pem
INFO Client Cert: /home/jgrusewski/Work/foxhunt/certs/client-cert.pem
INFO Client Key: /home/jgrusewski/Work/foxhunt/certs/client-key.pem
INFO Reading TLS certificates...
INFO All TLS certificates loaded
INFO TLS SNI hostname: localhost
INFO TLS configuration created with mTLS
INFO TLS connection established successfully!
INFO gRPC client created
INFO Testing health check RPC...
INFO Health check RPC succeeded!
INFO    Healthy: true
INFO    Message: ML Training Service is healthy
INFO    Latency: 7.860421ms
INFO ML Training TLS connectivity test PASSED!
INFO    ✅ TLS handshake successful
INFO    ✅ mTLS authentication successful
INFO    ✅ gRPC RPC call successful
INFO    ✅ End-to-end latency: 7.860421ms
test test_ml_training_tls_connectivity ... ok
```

**Analysis**: Direct connection from host to ML Training Service works because:
- Client uses `localhost` as TLS SNI hostname
- Certificate includes `localhost` in SANs ✅
- All certificates load correctly from repository paths ✅

#### Test 2: API Gateway Proxy ❌ FAILING

```bash
$ cargo test --test ml_training_tls_test test_ml_training_tls_via_api_gateway -- --nocapture

running 1 test
INFO Starting ML Training TLS connectivity test via API Gateway
INFO API Gateway URL: http://localhost:50051
INFO Connected to API Gateway
INFO gRPC client created for API Gateway
INFO Testing health check RPC via API Gateway...
Error: Failed to call HealthCheck RPC via API Gateway

Caused by:
    status: 'Operation is not implemented or not supported', metadata: {...}

test test_ml_training_tls_via_api_gateway ... FAILED
```

**API Gateway logs**:
```
INFO Attempting to initialize ML Training Service proxy...
INFO Setting up ML Training Service client for https://ml_training_service:50053
INFO Configuring TLS with mTLS (client certificates)
INFO   CA cert: /tmp/foxhunt/certs/ca/ca-cert.pem
INFO   Client cert: /tmp/foxhunt/certs/client-cert.pem
INFO   Client key: /tmp/foxhunt/certs/client-key.pem
INFO Reading CA certificate...
INFO CA certificate loaded (2017 bytes)
INFO Reading client certificate...
ERROR Failed to connect to ML Training Service: transport error
ERROR ⚠ ML Training service initialization failed!
WARN ⚠ ML Training service unavailable: ML Training Service connection failed: transport error
WARN    ML training endpoints will return 503 Service Unavailable
INFO    - ML Training Service: https://ml_training_service:50053 (✗ UNAVAILABLE)
```

**Analysis**: API Gateway connection fails because:
- API Gateway tries to connect to `https://ml_training_service:50053`
- TLS SNI hostname is `ml_training_service`
- Certificate **DOES NOT** include `ml_training_service` in SANs ❌
- TLS handshake fails with "transport error"

### 5. Port Configuration

**Docker network** (internal):
```yaml
ml_training_service:
  ports:
    - "50054:50053"  # External 50054 → Internal 50053
```

**API Gateway configuration**:
```yaml
environment:
  - ML_TRAINING_SERVICE_URL=https://ml_training_service:50053  ✅ Correct
```

**Verdict**: Port configuration is correct. No changes needed.

---

## Root Cause Analysis

### Certificate SANs Missing ML Training Service

**Current certificate SANs**:
```
DNS:foxhunt-services
DNS:backtesting_service     ✅ Backtesting works
DNS:localhost               ✅ Direct host connection works
IP Address:127.0.0.1
```

**Required SANs** (for API Gateway proxy):
```
DNS:foxhunt-services
DNS:backtesting_service
DNS:ml_training_service     ❌ MISSING!
DNS:localhost
IP Address:127.0.0.1
```

**Why this matters**:
1. API Gateway uses Docker service name `ml_training_service` as hostname
2. TLS client verifies hostname against certificate SANs
3. `ml_training_service` not in SANs → TLS handshake fails
4. Service appears as "UNAVAILABLE" in API Gateway

---

## Solution: Regenerate Certificates with Correct SANs

### Option A: Quick Fix (Add ml_training_service to existing cert)

**Steps**:
1. Update certificate configuration file (if using openssl config)
2. Add `DNS:ml_training_service` to Subject Alternative Names
3. Regenerate server certificate (keep CA unchanged)
4. Restart services to pick up new certificate

**Certificate generation command** (example):
```bash
# Create extensions config
cat > server-extensions.cnf << EOF
[v3_req]
subjectAltName = @alt_names

[alt_names]
DNS.1 = foxhunt-services
DNS.2 = backtesting_service
DNS.3 = ml_training_service
DNS.4 = localhost
IP.1 = 127.0.0.1
EOF

# Generate new CSR and certificate
openssl req -new -key certs/server-key.pem -out certs/server.csr -subj "/C=US/ST=NY/L=NewYork/O=Foxhunt/OU=HFT/CN=foxhunt-services"

openssl x509 -req -in certs/server.csr \
  -CA certs/ca/ca-cert.pem \
  -CAkey certs/ca/ca-key.pem \
  -CAcreateserial \
  -out certs/server-cert.pem \
  -days 365 \
  -sha256 \
  -extfile server-extensions.cnf \
  -extensions v3_req

# Verify SANs
openssl x509 -in certs/server-cert.pem -noout -ext subjectAltName

# Restart services
docker-compose restart ml_training_service api_gateway
```

**Time estimate**: 15-30 minutes

### Option B: Comprehensive Certificate Audit (Recommended)

**Steps**:
1. Audit ALL services and their certificate requirements
2. Create unified certificate with all service names
3. Document certificate generation process
4. Create script for future certificate rotation

**Services to include**:
- `foxhunt-services` (generic)
- `api_gateway` (port 50051)
- `trading_service` (port 50052)
- `backtesting_service` (port 50053)
- `ml_training_service` (port 50054)
- `localhost` (host access)
- `127.0.0.1` (IP access)

**Time estimate**: 45-60 minutes

---

## Validation Plan

### After Certificate Regeneration

1. **Verify certificate SANs**:
   ```bash
   openssl x509 -in certs/server-cert.pem -noout -ext subjectAltName
   # Should show: DNS:ml_training_service
   ```

2. **Restart services**:
   ```bash
   docker-compose restart ml_training_service api_gateway
   ```

3. **Check API Gateway logs**:
   ```bash
   docker logs foxhunt-api-gateway 2>&1 | grep "ML Training"
   # Should show: ✓ AVAILABLE
   ```

4. **Run E2E tests**:
   ```bash
   cd tests/e2e
   source ../../.env
   cargo test --test ml_training_tls_test -- --nocapture
   # Both tests should pass
   ```

5. **Run full E2E suite**:
   ```bash
   ./scripts/run_e2e_tests.sh
   ```

---

## Files Modified

### 1. tests/e2e/tests/ml_training_tls_test.rs
- **Lines changed**: 15 lines (37-51)
- **Purpose**: Fix certificate path defaults from `/tmp/foxhunt/certs` to repository paths
- **Impact**: E2E tests now find certificates correctly on host machine

### 2. tests/e2e/Cargo.toml
- **Lines changed**: 1 line (13)
- **Purpose**: Add TLS features to tonic dependency
- **Before**: `tonic = "0.14"`
- **After**: `tonic = { version = "0.14", features = ["transport", "tls-ring", "tls-webpki-roots"] }`
- **Impact**: Enable TLS support for E2E tests

---

## Recommendations

### Immediate Actions (Wave 158)

1. **Regenerate server certificate** with `ml_training_service` SAN (Option A: 15-30 min)
2. **Restart services** to pick up new certificate
3. **Validate API Gateway** connects successfully
4. **Run E2E tests** to confirm both tests pass

### Short-term (Next 2 weeks)

1. **Document certificate generation** process in docs/security/
2. **Create certificate rotation script** for future updates
3. **Add certificate validation** to CI/CD pipeline
4. **Monitor certificate expiration** dates (current: expires in <1 year?)

### Long-term (Next quarter)

1. **Implement cert-manager** or similar for automated rotation
2. **Migrate to Kubernetes** secrets for certificate management
3. **Add certificate monitoring** alerts (30/60/90 days before expiration)
4. **Consider wildcard certificate** for `*.foxhunt.local` or similar

---

## Performance Impact

### Direct TLS Connection (Working)
- **Latency**: 7.86ms end-to-end
- **Overhead**: ~7ms (TLS handshake + gRPC + Health check)
- **Within target**: ✅ Yes (<100ms)

### API Gateway Proxy (Currently Failing)
- **Expected latency**: 15-25ms (additional proxy hop)
- **Current status**: N/A (blocked by certificate issue)

---

## Security Considerations

### Certificate Validation
- ✅ CA chain validation working
- ✅ mTLS authentication working
- ✅ Certificate not expired
- ❌ Hostname validation failing for `ml_training_service`

### Risk Assessment
- **Severity**: Medium (blocks API Gateway proxy)
- **Impact**: ML Training Service unavailable via API Gateway
- **Workaround**: Direct connection to ML Training Service (port 50054) works
- **Data exposure**: None (TLS still encrypts when it works)
- **Recommendation**: Fix before production deployment

---

## Testing Summary

| Test | Status | Latency | Notes |
|------|--------|---------|-------|
| Direct TLS connectivity | ✅ PASS | 7.86ms | Uses `localhost` SAN |
| API Gateway proxy | ❌ FAIL | N/A | Missing `ml_training_service` SAN |
| Certificate chain validation | ✅ PASS | - | `openssl verify` OK |
| Port configuration | ✅ CORRECT | - | No changes needed |
| Docker volume mounts | ✅ CORRECT | - | No changes needed |

---

## Conclusion

### What Worked ✅
1. Fixed E2E test certificate paths (repository-relative, not `/tmp/foxhunt/certs`)
2. Added TLS features to tonic in E2E tests
3. Validated direct TLS connectivity to ML Training Service
4. Confirmed port configuration is correct
5. Confirmed Docker volume mounts are correct

### What's Blocked ⚠️
1. API Gateway → ML Training Service TLS connection
2. **Root cause**: Certificate missing `ml_training_service` in SANs
3. **Solution**: Regenerate certificate with additional SAN
4. **Time estimate**: 15-30 minutes (Option A) or 45-60 minutes (Option B)

### Next Steps
1. Regenerate server certificate with `ml_training_service` SAN
2. Restart ML Training Service and API Gateway
3. Validate API Gateway logs show "✓ AVAILABLE"
4. Re-run E2E tests (both should pass)
5. Update CLAUDE.md with Wave 157 completion status

---

**Report prepared by**: Claude (Wave 157)
**Date**: 2025-10-13
**Time invested**: ~45 minutes (investigation, fixes, testing, documentation)