foxhunt/WAVE_157_CERTIFICATE_FIX_REPORT.md

Wave 157: Certificate Paths and E2E Test Report

Date: 2025-10-13 Status: ✅ DIRECT TLS CONNECTIVITY WORKING | ⚠️ API GATEWAY BLOCKED BY CERTIFICATE SANs

---

Executive Summary

Successes ✅

1. Certificate paths fixed in E2E test (hardcoded /tmp/foxhunt/certs → repository-relative paths)
2. Direct TLS connectivity test PASSING (test_ml_training_tls_connectivity)
   • TLS handshake: ✅ Successful
   • mTLS authentication: ✅ Successful
   • Health check RPC: ✅ Successful (7.86ms latency)
   • Certificate chain: ✅ Valid

Blockers ⚠️

1. API Gateway proxy test FAILING (test_ml_training_tls_via_api_gateway)
   • Root cause: Certificate missing ml_training_service in SANs
   • Impact: API Gateway cannot establish TLS connection to ML Training Service
   • Error: "transport error" during TLS handshake

---

Technical Details

1. Certificate Location Analysis

Repository structure:

/home/jgrusewski/Work/foxhunt/
├── certs/
│   ├── ca/
│   │   ├── ca-cert.pem         ✅ Valid (2017 bytes)
│   │   └── ca-key.pem
│   ├── server-cert.pem         ✅ Valid (signed by CA)
│   ├── server-key.pem
│   ├── client-cert.pem         ✅ Valid
│   └── client-key.pem

Certificate verification:

$ openssl verify -CAfile certs/ca/ca-cert.pem certs/server-cert.pem
certs/server-cert.pem: OK

$ openssl x509 -in certs/server-cert.pem -noout -subject -ext subjectAltName
subject=C = US, ST = NY, L = NewYork, O = Foxhunt, OU = HFT, CN = foxhunt-services
X509v3 Subject Alternative Name:
    DNS:foxhunt-services, DNS:backtesting_service, DNS:localhost, IP Address:127.0.0.1

CRITICAL FINDING: Certificate SANs missing ml_training_service!

2. Docker Container Paths

Volume mounts (docker-compose.yml):

volumes:
  - ./certs:/tmp/foxhunt/certs:ro  # ✅ Correct

Inside containers: /tmp/foxhunt/certs/* On host machine: /home/jgrusewski/Work/foxhunt/certs/*

Verdict: Docker volume mounts are correct. No changes needed.

3. E2E Test Fixes Applied

File: tests/e2e/tests/ml_training_tls_test.rs

Before (lines 37-44):

let ca_cert_path = std::env::var("ML_TRAINING_TLS_CA_CERT")
    .unwrap_or_else(|_| "/tmp/foxhunt/certs/ca/ca-cert.pem".to_string());

let client_cert_path = std::env::var("ML_TRAINING_TLS_CLIENT_CERT")
    .unwrap_or_else(|_| "/tmp/foxhunt/certs/client-cert.pem".to_string());

let client_key_path = std::env::var("ML_TRAINING_TLS_CLIENT_KEY")
    .unwrap_or_else(|_| "/tmp/foxhunt/certs/client-key.pem".to_string());

After (lines 37-51):

// Default to repository certs directory (for host-based E2E tests)
// Note: Docker containers use /tmp/foxhunt/certs via volume mount,
// but E2E tests run on the host and need the repository path
let default_ca_cert = format!("{}/certs/ca/ca-cert.pem", env!("CARGO_MANIFEST_DIR").replace("/tests/e2e", ""));
let default_client_cert = format!("{}/certs/client-cert.pem", env!("CARGO_MANIFEST_DIR").replace("/tests/e2e", ""));
let default_client_key = format!("{}/certs/client-key.pem", env!("CARGO_MANIFEST_DIR").replace("/tests/e2e", ""));

let ca_cert_path = std::env::var("ML_TRAINING_TLS_CA_CERT")
    .unwrap_or(default_ca_cert);

let client_cert_path = std::env::var("ML_TRAINING_TLS_CLIENT_CERT")
    .unwrap_or(default_client_cert);

let client_key_path = std::env::var("ML_TRAINING_TLS_CLIENT_KEY")
    .unwrap_or(default_client_key);

Cargo.toml fix:

# Added TLS features to tonic
tonic = { version = "0.14", features = ["transport", "tls-ring", "tls-webpki-roots"] }

4. Test Results

Test 1: Direct TLS Connectivity ✅ PASSING

$ cargo test --test ml_training_tls_test test_ml_training_tls_connectivity -- --nocapture

running 1 test
INFO Starting ML Training TLS connectivity test
INFO ML Service URL: https://localhost:50054
INFO CA Cert: /home/jgrusewski/Work/foxhunt/certs/ca/ca-cert.pem
INFO Client Cert: /home/jgrusewski/Work/foxhunt/certs/client-cert.pem
INFO Client Key: /home/jgrusewski/Work/foxhunt/certs/client-key.pem
INFO Reading TLS certificates...
INFO All TLS certificates loaded
INFO TLS SNI hostname: localhost
INFO TLS configuration created with mTLS
INFO TLS connection established successfully!
INFO gRPC client created
INFO Testing health check RPC...
INFO Health check RPC succeeded!
INFO    Healthy: true
INFO    Message: ML Training Service is healthy
INFO    Latency: 7.860421ms
INFO ML Training TLS connectivity test PASSED!
INFO    ✅ TLS handshake successful
INFO    ✅ mTLS authentication successful
INFO    ✅ gRPC RPC call successful
INFO    ✅ End-to-end latency: 7.860421ms
test test_ml_training_tls_connectivity ... ok

Analysis: Direct connection from host to ML Training Service works because:

• Client uses localhost as TLS SNI hostname
• Certificate includes localhost in SANs ✅
• All certificates load correctly from repository paths ✅

Test 2: API Gateway Proxy ❌ FAILING

$ cargo test --test ml_training_tls_test test_ml_training_tls_via_api_gateway -- --nocapture

running 1 test
INFO Starting ML Training TLS connectivity test via API Gateway
INFO API Gateway URL: http://localhost:50051
INFO Connected to API Gateway
INFO gRPC client created for API Gateway
INFO Testing health check RPC via API Gateway...
Error: Failed to call HealthCheck RPC via API Gateway

Caused by:
    status: 'Operation is not implemented or not supported', metadata: {...}

test test_ml_training_tls_via_api_gateway ... FAILED

API Gateway logs:

INFO Attempting to initialize ML Training Service proxy...
INFO Setting up ML Training Service client for https://ml_training_service:50053
INFO Configuring TLS with mTLS (client certificates)
INFO   CA cert: /tmp/foxhunt/certs/ca/ca-cert.pem
INFO   Client cert: /tmp/foxhunt/certs/client-cert.pem
INFO   Client key: /tmp/foxhunt/certs/client-key.pem
INFO Reading CA certificate...
INFO CA certificate loaded (2017 bytes)
INFO Reading client certificate...
ERROR Failed to connect to ML Training Service: transport error
ERROR ⚠ ML Training service initialization failed!
WARN ⚠ ML Training service unavailable: ML Training Service connection failed: transport error
WARN    ML training endpoints will return 503 Service Unavailable
INFO    - ML Training Service: https://ml_training_service:50053 (✗ UNAVAILABLE)

Analysis: API Gateway connection fails because:

• API Gateway tries to connect to https://ml_training_service:50053
• TLS SNI hostname is ml_training_service
• Certificate DOES NOT include ml_training_service in SANs ❌
• TLS handshake fails with "transport error"

5. Port Configuration

Docker network (internal):

ml_training_service:
  ports:
    - "50054:50053"  # External 50054 → Internal 50053

API Gateway configuration:

environment:
  - ML_TRAINING_SERVICE_URL=https://ml_training_service:50053  ✅ Correct

Verdict: Port configuration is correct. No changes needed.

---

Root Cause Analysis

Certificate SANs Missing ML Training Service

Current certificate SANs:

DNS:foxhunt-services
DNS:backtesting_service     ✅ Backtesting works
DNS:localhost               ✅ Direct host connection works
IP Address:127.0.0.1

Required SANs (for API Gateway proxy):

DNS:foxhunt-services
DNS:backtesting_service
DNS:ml_training_service     ❌ MISSING!
DNS:localhost
IP Address:127.0.0.1

Why this matters:

1. API Gateway uses Docker service name ml_training_service as hostname
2. TLS client verifies hostname against certificate SANs
3. ml_training_service not in SANs → TLS handshake fails
4. Service appears as "UNAVAILABLE" in API Gateway

---

Solution: Regenerate Certificates with Correct SANs

Option A: Quick Fix (Add ml_training_service to existing cert)

Steps:

1. Update certificate configuration file (if using openssl config)
2. Add DNS:ml_training_service to Subject Alternative Names
3. Regenerate server certificate (keep CA unchanged)
4. Restart services to pick up new certificate

Certificate generation command (example):

# Create extensions config
cat > server-extensions.cnf << EOF
[v3_req]
subjectAltName = @alt_names

[alt_names]
DNS.1 = foxhunt-services
DNS.2 = backtesting_service
DNS.3 = ml_training_service
DNS.4 = localhost
IP.1 = 127.0.0.1
EOF

# Generate new CSR and certificate
openssl req -new -key certs/server-key.pem -out certs/server.csr -subj "/C=US/ST=NY/L=NewYork/O=Foxhunt/OU=HFT/CN=foxhunt-services"

openssl x509 -req -in certs/server.csr \
  -CA certs/ca/ca-cert.pem \
  -CAkey certs/ca/ca-key.pem \
  -CAcreateserial \
  -out certs/server-cert.pem \
  -days 365 \
  -sha256 \
  -extfile server-extensions.cnf \
  -extensions v3_req

# Verify SANs
openssl x509 -in certs/server-cert.pem -noout -ext subjectAltName

# Restart services
docker-compose restart ml_training_service api_gateway

Time estimate: 15-30 minutes

Option B: Comprehensive Certificate Audit (Recommended)

Steps:

1. Audit ALL services and their certificate requirements
2. Create unified certificate with all service names
3. Document certificate generation process
4. Create script for future certificate rotation

Services to include:

• foxhunt-services (generic)
• api_gateway (port 50051)
• trading_service (port 50052)
• backtesting_service (port 50053)
• ml_training_service (port 50054)
• localhost (host access)
• 127.0.0.1 (IP access)

Time estimate: 45-60 minutes

---

Validation Plan

After Certificate Regeneration

1. Verify certificate SANs:

   openssl x509 -in certs/server-cert.pem -noout -ext subjectAltName
   # Should show: DNS:ml_training_service
   

2. Restart services:

   docker-compose restart ml_training_service api_gateway
   

3. Check API Gateway logs:

   docker logs foxhunt-api-gateway 2>&1 | grep "ML Training"
   # Should show: ✓ AVAILABLE
   

4. Run E2E tests:

   cd tests/e2e
   source ../../.env
   cargo test --test ml_training_tls_test -- --nocapture
   # Both tests should pass
   

5. Run full E2E suite:

   ./scripts/run_e2e_tests.sh
   

---

Files Modified

1. tests/e2e/tests/ml_training_tls_test.rs

• Lines changed: 15 lines (37-51)
• Purpose: Fix certificate path defaults from /tmp/foxhunt/certs to repository paths
• Impact: E2E tests now find certificates correctly on host machine

2. tests/e2e/Cargo.toml

• Lines changed: 1 line (13)
• Purpose: Add TLS features to tonic dependency
• Before: tonic = "0.14"
• After: tonic = { version = "0.14", features = ["transport", "tls-ring", "tls-webpki-roots"] }
• Impact: Enable TLS support for E2E tests

---

Recommendations

Immediate Actions (Wave 158)

1. Regenerate server certificate with ml_training_service SAN (Option A: 15-30 min)
2. Restart services to pick up new certificate
3. Validate API Gateway connects successfully
4. Run E2E tests to confirm both tests pass

Short-term (Next 2 weeks)

1. Document certificate generation process in docs/security/
2. Create certificate rotation script for future updates
3. Add certificate validation to CI/CD pipeline
4. Monitor certificate expiration dates (current: expires in <1 year?)

Long-term (Next quarter)

1. Implement cert-manager or similar for automated rotation
2. Migrate to Kubernetes secrets for certificate management
3. Add certificate monitoring alerts (30/60/90 days before expiration)
4. Consider wildcard certificate for *.foxhunt.local or similar

---

Performance Impact

Direct TLS Connection (Working)

• Latency: 7.86ms end-to-end
• Overhead: ~7ms (TLS handshake + gRPC + Health check)
• Within target: ✅ Yes (<100ms)

API Gateway Proxy (Currently Failing)

• Expected latency: 15-25ms (additional proxy hop)
• Current status: N/A (blocked by certificate issue)

---

Security Considerations

Certificate Validation

• ✅ CA chain validation working
• ✅ mTLS authentication working
• ✅ Certificate not expired
• ❌ Hostname validation failing for ml_training_service

Risk Assessment

• Severity: Medium (blocks API Gateway proxy)
• Impact: ML Training Service unavailable via API Gateway
• Workaround: Direct connection to ML Training Service (port 50054) works
• Data exposure: None (TLS still encrypts when it works)
• Recommendation: Fix before production deployment

---

Testing Summary

Test	Status	Latency	Notes
Direct TLS connectivity	✅ PASS	7.86ms	Uses localhost SAN
API Gateway proxy	❌ FAIL	N/A	Missing ml_training_service SAN
Certificate chain validation	✅ PASS	-	openssl verify OK
Port configuration	✅ CORRECT	-	No changes needed
Docker volume mounts	✅ CORRECT	-	No changes needed

---

Conclusion

What Worked ✅

1. Fixed E2E test certificate paths (repository-relative, not /tmp/foxhunt/certs)
2. Added TLS features to tonic in E2E tests
3. Validated direct TLS connectivity to ML Training Service
4. Confirmed port configuration is correct
5. Confirmed Docker volume mounts are correct

What's Blocked ⚠️

1. API Gateway → ML Training Service TLS connection
2. Root cause: Certificate missing ml_training_service in SANs
3. Solution: Regenerate certificate with additional SAN
4. Time estimate: 15-30 minutes (Option A) or 45-60 minutes (Option B)

Next Steps

1. Regenerate server certificate with ml_training_service SAN
2. Restart ML Training Service and API Gateway
3. Validate API Gateway logs show "✓ AVAILABLE"
4. Re-run E2E tests (both should pass)
5. Update CLAUDE.md with Wave 157 completion status

---

Report prepared by: Claude (Wave 157) Date: 2025-10-13 Time invested: ~45 minutes (investigation, fixes, testing, documentation)