foxhunt/WAVE_145_JWT_FIX_PLAN.md

Wave 145: JWT Authentication Fix - Comprehensive Plan

Created: 2025-10-12 Goal: Fix JWT authentication issues causing E2E test failures (26-62% pass rates) Root Cause: Backend services missing JWT environment variables Strategy: 12 parallel agents for configuration, validation, and testing Expected Outcome: 85%+ E2E test pass rate (35-40 tests passing)

---

Root Cause Analysis (via zen thinkdeep)

Confidence Level: 🟢 ALMOST CERTAIN (99%+)

Current State

API Gateway (docker-compose.yml lines 294-296): ✅ HAS JWT CONFIG

environment:
  - JWT_SECRET=${JWT_SECRET:-dev_secret_key_change_in_production}
  - JWT_ISSUER=foxhunt-api-gateway
  - JWT_AUDIENCE=foxhunt-services

Trading Service (lines 174-179): ❌ MISSING JWT CONFIG

environment:
  - DATABASE_URL=postgresql://foxhunt:foxhunt_dev_password@postgres:5432/foxhunt
  - REDIS_URL=redis://redis:6379
  - VAULT_ADDR=http://vault:8200
  - VAULT_TOKEN=foxhunt-dev-root
  # NO JWT_SECRET, JWT_ISSUER, JWT_AUDIENCE

Backtesting Service (lines 208-214): ❌ MISSING JWT CONFIG ML Training Service (lines 246-251): ❌ MISSING JWT CONFIG

Authentication Flow Analysis

Current Flow (FAILING):

1. Client → API Gateway ✅ Success
   - API Gateway has JWT_SECRET
   - Validates token successfully
   - Authenticates user

2. API Gateway → Trading Service ❌ Failure
   - API Gateway forwards JWT token
   - Trading Service attempts to validate JWT
   - Trading Service has NO JWT_SECRET
   - Validation fails: "Invalid or expired token"

Why Tests Fail:

• Backend services receive JWT from API Gateway
• Backend services attempt independent JWT validation
• Backend services missing JWT_SECRET environment variable
• Validation fails with cryptic error messages
• E2E tests see authentication failures (26-62% pass rates)

---

Solution: Unified JWT Configuration

Approach

Option A: Unified JWT Configuration (RECOMMENDED ✅):

• Add JWT environment variables to ALL backend services
• All services use same JWT_SECRET, JWT_ISSUER, JWT_AUDIENCE
• Simple, maintains security, no code changes required
• Effort: 15-30 minutes
• Risk: LOW

Option B: Service Trust Model (NOT RECOMMENDED ❌):

• Only API Gateway validates JWT
• Backend services trust API Gateway metadata
• Requires code changes in all backend services
• Effort: 4-8 hours
• Risk: MEDIUM-HIGH

DECISION: Implement Option A (Unified JWT Configuration)

---

Implementation Plan (12 Agents)

Phase 1: Configuration Updates (3 agents, 15 min)

Agent 331: Update Trading Service JWT Config

• File: docker-compose.yml (lines 174-179)
• Add 3 environment variables:

  - JWT_SECRET=${JWT_SECRET:-dev_secret_key_change_in_production}
  - JWT_ISSUER=foxhunt-api-gateway
  - JWT_AUDIENCE=foxhunt-services
  

• Position: After VAULT_TOKEN, before RUST_LOG

Agent 332: Update Backtesting Service JWT Config

• File: docker-compose.yml (lines 208-214)
• Add same 3 JWT environment variables
• Position: After VAULT_TOKEN, before BENZINGA_API_KEY

Agent 333: Update ML Training Service JWT Config

• File: docker-compose.yml (lines 246-251)
• Add same 3 JWT environment variables
• Position: After VAULT_TOKEN, before RUST_LOG

---

Phase 2: Service Restart (2 agents, 5 min)

Agent 334: Verify .env File

• File: .env
• Ensure JWT_SECRET is set correctly
• Expected value: YZg5/mpqzH0NehGJXiR1yUgUg74HqdOUj/q9tnVSX+gqZvuzHKI1n0NhL4yP8CkUx7WyrVs3X86OSSxIUA6sxQ==
• Validate JWT_ISSUER and JWT_AUDIENCE match

Agent 335: Restart Backend Services

• Stop services: docker-compose stop trading_service backtesting_service ml_training_service
• Start services: docker-compose up -d trading_service backtesting_service ml_training_service
• Wait 60 seconds for health checks
• Verify all services healthy: docker-compose ps

---

Phase 3: Validation Tests (4 agents, 30 min)

Agent 336: Validate Service Health E2E Tests (15 tests)

• File: services/integration_tests/tests/service_health_resilience_e2e.rs
• Run tests: cargo test -p integration_tests --test service_health_resilience_e2e -- --test-threads=1
• Current pass rate: 26.7% (4/15)
• Expected pass rate: 85%+ (13+/15)
• Report: Total passing, specific failures

Agent 337: Validate Backtesting E2E Tests (12 tests)

• File: services/integration_tests/tests/backtesting_service_e2e.rs
• Run tests: cargo test -p integration_tests --test backtesting_service_e2e -- --test-threads=1
• Current pass rate: 62.5% (15/23 including auth helpers)
• Expected pass rate: 85%+ (19+/23)
• Report: Total passing, specific failures

Agent 338: Validate Trading E2E Tests (15 tests)

• File: services/integration_tests/tests/trading_service_e2e.rs
• Run tests: cargo test -p integration_tests --test trading_service_e2e -- --test-threads=1
• Current pass rate: 57.7% (15/26 including auth helpers)
• Expected pass rate: 85%+ (22+/26)
• Report: Total passing, specific failures

Agent 339: Cross-Service Integration Validation

• Test API Gateway → Trading Service authentication
• Test API Gateway → Backtesting Service authentication
• Verify JWT metadata forwarding
• Check service logs for auth errors
• Report: Any remaining authentication issues

---

Phase 4: Regression Testing (2 agents, 20 min)

Agent 340: Validate Infrastructure Tests Still Pass

• Run PostgreSQL tests (41 tests): cargo test -p trading_engine --test persistence_integration_tests
• Run Redis tests (18 tests): cargo test -p trading_engine redis -- --test-threads=1
• Expected: 95-100% pass rate maintained
• Report: Any regressions from JWT changes

Agent 341: Validate Library Tests Still Pass

• Run workspace library tests: cargo test --workspace --lib
• Expected: 1,586+ tests passing (100%)
• Report: Zero regressions

---

Phase 5: Documentation & Reporting (1 agent, 15 min)

Agent 342: Create Comprehensive Wave 145 Report

• Aggregate results from agents 331-341
• Calculate overall pass rate improvement
• Document JWT configuration pattern
• Create before/after comparison
• Make commit recommendation
• Deliverable: WAVE_145_JWT_FIX_RESULTS.md

---

Success Criteria

Must Achieve ✅

• All 3 backend services have JWT configuration
• Services restart successfully
• Service Health E2E: 85%+ pass rate (13+/15 tests)
• Backtesting E2E: 85%+ pass rate (19+/23 tests)
• Trading E2E: 85%+ pass rate (22+/26 tests)
• Zero regressions in infrastructure tests
• Zero regressions in library tests

Target Metrics 🎯

• Overall E2E Pass Rate: 85-90% (35-40 tests passing out of 42 E2E tests)
• Infrastructure Tests: 95-100% maintained (59 tests)
• Library Tests: 100% maintained (1,586+ tests)
• Total Pass Rate: 98-99% (1,680+/1,698+ tests)

Nice to Have 🌟

• 90%+ E2E pass rate (38+/42 tests)
• Zero authentication errors in service logs
• JWT metadata forwarding validated

---

Expected Outcomes

Before Wave 145

E2E Tests:
- Service Health: 4/15 passing (26.7%) ❌
- Backtesting: 15/23 passing (62.5%) ⚠️
- Trading: 15/26 passing (57.7%) ⚠️
Total E2E: ~34/64 passing (53%)

Overall Tests: 1,645/1,698 passing (97%)

After Wave 145

E2E Tests:
- Service Health: 13+/15 passing (85%+) ✅
- Backtesting: 19+/23 passing (85%+) ✅
- Trading: 22+/26 passing (85%+) ✅
Total E2E: ~54/64 passing (85%)

Overall Tests: 1,680+/1,698 passing (99%)

Improvement

• E2E Tests: +20 tests passing (+37% improvement)
• Overall Tests: +35 tests passing (+2% improvement)
• Production Readiness: FULLY VALIDATED ✅

---

Risk Assessment

Low Risks ✅

1. Simple configuration change: No code modifications
2. Easily reversible: Can revert docker-compose.yml changes
3. Well-tested pattern: JWT config already working in API Gateway
4. Non-breaking: Existing functionality preserved

Medium Risks ⚠️

1. Service restart downtime: 30-60 seconds (acceptable for dev environment)
2. Environment variable propagation: May need container rebuild
3. Test timing: Some tests may have race conditions

Mitigation Strategies

1. Backup docker-compose.yml before changes
2. Validate .env file before restart
3. Monitor service logs during restart
4. Run tests sequentially (--test-threads=1) to avoid race conditions

---

Timeline

Phase	Duration	Agents	Activities
Phase 1	15 min	3	Configuration updates
Phase 2	5 min	2	Service restart
Phase 3	30 min	4	E2E validation tests
Phase 4	20 min	2	Regression testing
Phase 5	15 min	1	Documentation & reporting
Total	85 min	12	Complete JWT fix

Parallel Efficiency:

• Phases 1, 3, 4 can run agents in parallel
• Phases 2, 5 are sequential
• Estimated wall-clock time: ~60 minutes

---

Validation Checklist

Pre-Execution ✅

• .env file exists with JWT_SECRET
• All services currently running
• docker-compose.yml backed up
• Test environment clean (no stale processes)

Post-Execution ✅

• All 3 backend services restarted successfully
• All services showing "healthy" status
• E2E tests achieve 85%+ pass rate
• No regressions in infrastructure tests
• No regressions in library tests
• Service logs show no authentication errors

Commit Criteria ✅

• Overall test pass rate: 98%+
• E2E test pass rate: 85%+
• Zero critical failures
• Wave 145 report completed
• CLAUDE.md updated

---

Rollback Plan

If E2E tests still fail after JWT configuration:

1. Immediate Rollback (5 minutes):

   git checkout docker-compose.yml  # Revert changes
   docker-compose restart trading_service backtesting_service ml_training_service
   

2. Investigation (30-60 minutes):

   • Check if JWT_SECRET in .env matches API Gateway
   • Verify JWT_ISSUER and JWT_AUDIENCE are consistent
   • Review service logs for specific JWT validation errors
   • Test JWT generation with jwt_token_generator.sh

3. Alternative Solution (4-8 hours):

   • Implement Service Trust Model (Option B)
   • Remove JWT validation from backend services
   • Use API Gateway metadata only

---

Files Modified

Configuration Files (1 file)

1. docker-compose.yml - Add JWT env vars to 3 services (9 lines added)

No Code Changes Required ✅

• All fixes are configuration-only
• No Rust code modifications
• No protocol buffer changes
• No test code changes (already fixed in Wave 144)

---

Agent Coordination

Dependencies

• Agents 331-333: Independent (can run in parallel)
• Agent 334: Must complete before Agent 335
• Agent 335: Blocking for Agents 336-339
• Agents 336-339: Independent (can run in parallel)
• Agents 340-341: Independent (can run in parallel)
• Agent 342: Depends on all previous agents

Communication

• Each agent reports results to Agent 342 (coordinator)
• Agent 335 signals "services ready" before validation agents start
• Agent 342 makes final commit recommendation

---

Status: READY FOR EXECUTION Confidence: 99% (zen thinkdeep: "almost_certain") Expected Success Rate: 85%+ E2E test pass rate Timeline: 60-85 minutes Risk Level: LOW ✅