foxhunt/WAVE_145_JWT_FIX_PLAN.md

# Wave 145: JWT Authentication Fix - Comprehensive Plan

**Created**: 2025-10-12
**Goal**: Fix JWT authentication issues causing E2E test failures (26-62% pass rates)
**Root Cause**: Backend services missing JWT environment variables
**Strategy**: 12 parallel agents for configuration, validation, and testing
**Expected Outcome**: 85%+ E2E test pass rate (35-40 tests passing)

---

## Root Cause Analysis (via zen thinkdeep)

**Confidence Level**: 🟢 ALMOST CERTAIN (99%+)

### Current State

**API Gateway** (docker-compose.yml lines 294-296): ✅ HAS JWT CONFIG
```yaml
environment:
  - JWT_SECRET=${JWT_SECRET:-dev_secret_key_change_in_production}
  - JWT_ISSUER=foxhunt-api-gateway
  - JWT_AUDIENCE=foxhunt-services
```

**Trading Service** (lines 174-179): ❌ MISSING JWT CONFIG
```yaml
environment:
  - DATABASE_URL=postgresql://foxhunt:foxhunt_dev_password@postgres:5432/foxhunt
  - REDIS_URL=redis://redis:6379
  - VAULT_ADDR=http://vault:8200
  - VAULT_TOKEN=foxhunt-dev-root
  # NO JWT_SECRET, JWT_ISSUER, JWT_AUDIENCE
```

**Backtesting Service** (lines 208-214): ❌ MISSING JWT CONFIG
**ML Training Service** (lines 246-251): ❌ MISSING JWT CONFIG

### Authentication Flow Analysis

**Current Flow** (FAILING):
```
1. Client → API Gateway ✅ Success
   - API Gateway has JWT_SECRET
   - Validates token successfully
   - Authenticates user

2. API Gateway → Trading Service ❌ Failure
   - API Gateway forwards JWT token
   - Trading Service attempts to validate JWT
   - Trading Service has NO JWT_SECRET
   - Validation fails: "Invalid or expired token"
```

**Why Tests Fail**:
- Backend services receive JWT from API Gateway
- Backend services attempt independent JWT validation
- Backend services missing JWT_SECRET environment variable
- Validation fails with cryptic error messages
- E2E tests see authentication failures (26-62% pass rates)

---

## Solution: Unified JWT Configuration

### Approach

**Option A: Unified JWT Configuration** (RECOMMENDED ✅):
- Add JWT environment variables to ALL backend services
- All services use same JWT_SECRET, JWT_ISSUER, JWT_AUDIENCE
- Simple, maintains security, no code changes required
- **Effort**: 15-30 minutes
- **Risk**: LOW

**Option B: Service Trust Model** (NOT RECOMMENDED ❌):
- Only API Gateway validates JWT
- Backend services trust API Gateway metadata
- Requires code changes in all backend services
- **Effort**: 4-8 hours
- **Risk**: MEDIUM-HIGH

**DECISION**: Implement Option A (Unified JWT Configuration)

---

## Implementation Plan (12 Agents)

### Phase 1: Configuration Updates (3 agents, 15 min)

**Agent 331: Update Trading Service JWT Config**
- File: `docker-compose.yml` (lines 174-179)
- Add 3 environment variables:
  ```yaml
  - JWT_SECRET=${JWT_SECRET:-dev_secret_key_change_in_production}
  - JWT_ISSUER=foxhunt-api-gateway
  - JWT_AUDIENCE=foxhunt-services
  ```
- Position: After VAULT_TOKEN, before RUST_LOG

**Agent 332: Update Backtesting Service JWT Config**
- File: `docker-compose.yml` (lines 208-214)
- Add same 3 JWT environment variables
- Position: After VAULT_TOKEN, before BENZINGA_API_KEY

**Agent 333: Update ML Training Service JWT Config**
- File: `docker-compose.yml` (lines 246-251)
- Add same 3 JWT environment variables
- Position: After VAULT_TOKEN, before RUST_LOG

---

### Phase 2: Service Restart (2 agents, 5 min)

**Agent 334: Verify .env File**
- File: `.env`
- Ensure JWT_SECRET is set correctly
- Expected value: `YZg5/mpqzH0NehGJXiR1yUgUg74HqdOUj/q9tnVSX+gqZvuzHKI1n0NhL4yP8CkUx7WyrVs3X86OSSxIUA6sxQ==`
- Validate JWT_ISSUER and JWT_AUDIENCE match

**Agent 335: Restart Backend Services**
- Stop services: `docker-compose stop trading_service backtesting_service ml_training_service`
- Start services: `docker-compose up -d trading_service backtesting_service ml_training_service`
- Wait 60 seconds for health checks
- Verify all services healthy: `docker-compose ps`

---

### Phase 3: Validation Tests (4 agents, 30 min)

**Agent 336: Validate Service Health E2E Tests (15 tests)**
- File: `services/integration_tests/tests/service_health_resilience_e2e.rs`
- Run tests: `cargo test -p integration_tests --test service_health_resilience_e2e -- --test-threads=1`
- Current pass rate: 26.7% (4/15)
- Expected pass rate: 85%+ (13+/15)
- Report: Total passing, specific failures

**Agent 337: Validate Backtesting E2E Tests (12 tests)**
- File: `services/integration_tests/tests/backtesting_service_e2e.rs`
- Run tests: `cargo test -p integration_tests --test backtesting_service_e2e -- --test-threads=1`
- Current pass rate: 62.5% (15/23 including auth helpers)
- Expected pass rate: 85%+ (19+/23)
- Report: Total passing, specific failures

**Agent 338: Validate Trading E2E Tests (15 tests)**
- File: `services/integration_tests/tests/trading_service_e2e.rs`
- Run tests: `cargo test -p integration_tests --test trading_service_e2e -- --test-threads=1`
- Current pass rate: 57.7% (15/26 including auth helpers)
- Expected pass rate: 85%+ (22+/26)
- Report: Total passing, specific failures

**Agent 339: Cross-Service Integration Validation**
- Test API Gateway → Trading Service authentication
- Test API Gateway → Backtesting Service authentication
- Verify JWT metadata forwarding
- Check service logs for auth errors
- Report: Any remaining authentication issues

---

### Phase 4: Regression Testing (2 agents, 20 min)

**Agent 340: Validate Infrastructure Tests Still Pass**
- Run PostgreSQL tests (41 tests): `cargo test -p trading_engine --test persistence_integration_tests`
- Run Redis tests (18 tests): `cargo test -p trading_engine redis -- --test-threads=1`
- Expected: 95-100% pass rate maintained
- Report: Any regressions from JWT changes

**Agent 341: Validate Library Tests Still Pass**
- Run workspace library tests: `cargo test --workspace --lib`
- Expected: 1,586+ tests passing (100%)
- Report: Zero regressions

---

### Phase 5: Documentation & Reporting (1 agent, 15 min)

**Agent 342: Create Comprehensive Wave 145 Report**
- Aggregate results from agents 331-341
- Calculate overall pass rate improvement
- Document JWT configuration pattern
- Create before/after comparison
- Make commit recommendation
- Deliverable: `WAVE_145_JWT_FIX_RESULTS.md`

---

## Success Criteria

### Must Achieve ✅
- [ ] All 3 backend services have JWT configuration
- [ ] Services restart successfully
- [ ] Service Health E2E: 85%+ pass rate (13+/15 tests)
- [ ] Backtesting E2E: 85%+ pass rate (19+/23 tests)
- [ ] Trading E2E: 85%+ pass rate (22+/26 tests)
- [ ] Zero regressions in infrastructure tests
- [ ] Zero regressions in library tests

### Target Metrics 🎯
- **Overall E2E Pass Rate**: 85-90% (35-40 tests passing out of 42 E2E tests)
- **Infrastructure Tests**: 95-100% maintained (59 tests)
- **Library Tests**: 100% maintained (1,586+ tests)
- **Total Pass Rate**: 98-99% (1,680+/1,698+ tests)

### Nice to Have 🌟
- 90%+ E2E pass rate (38+/42 tests)
- Zero authentication errors in service logs
- JWT metadata forwarding validated

---

## Expected Outcomes

### Before Wave 145
```
E2E Tests:
- Service Health: 4/15 passing (26.7%) ❌
- Backtesting: 15/23 passing (62.5%) ⚠️
- Trading: 15/26 passing (57.7%) ⚠️
Total E2E: ~34/64 passing (53%)

Overall Tests: 1,645/1,698 passing (97%)
```

### After Wave 145
```
E2E Tests:
- Service Health: 13+/15 passing (85%+) ✅
- Backtesting: 19+/23 passing (85%+) ✅
- Trading: 22+/26 passing (85%+) ✅
Total E2E: ~54/64 passing (85%)

Overall Tests: 1,680+/1,698 passing (99%)
```

### Improvement
- **E2E Tests**: +20 tests passing (+37% improvement)
- **Overall Tests**: +35 tests passing (+2% improvement)
- **Production Readiness**: FULLY VALIDATED ✅

---

## Risk Assessment

### Low Risks ✅
1. **Simple configuration change**: No code modifications
2. **Easily reversible**: Can revert docker-compose.yml changes
3. **Well-tested pattern**: JWT config already working in API Gateway
4. **Non-breaking**: Existing functionality preserved

### Medium Risks ⚠️
1. **Service restart downtime**: 30-60 seconds (acceptable for dev environment)
2. **Environment variable propagation**: May need container rebuild
3. **Test timing**: Some tests may have race conditions

### Mitigation Strategies
1. **Backup docker-compose.yml** before changes
2. **Validate .env file** before restart
3. **Monitor service logs** during restart
4. **Run tests sequentially** (--test-threads=1) to avoid race conditions

---

## Timeline

| Phase | Duration | Agents | Activities |
|-------|----------|--------|------------|
| **Phase 1** | 15 min | 3 | Configuration updates |
| **Phase 2** | 5 min | 2 | Service restart |
| **Phase 3** | 30 min | 4 | E2E validation tests |
| **Phase 4** | 20 min | 2 | Regression testing |
| **Phase 5** | 15 min | 1 | Documentation & reporting |
| **Total** | **85 min** | **12** | **Complete JWT fix** |

**Parallel Efficiency**:
- Phases 1, 3, 4 can run agents in parallel
- Phases 2, 5 are sequential
- Estimated wall-clock time: ~60 minutes

---

## Validation Checklist

### Pre-Execution ✅
- [ ] `.env` file exists with JWT_SECRET
- [ ] All services currently running
- [ ] docker-compose.yml backed up
- [ ] Test environment clean (no stale processes)

### Post-Execution ✅
- [ ] All 3 backend services restarted successfully
- [ ] All services showing "healthy" status
- [ ] E2E tests achieve 85%+ pass rate
- [ ] No regressions in infrastructure tests
- [ ] No regressions in library tests
- [ ] Service logs show no authentication errors

### Commit Criteria ✅
- [ ] Overall test pass rate: 98%+
- [ ] E2E test pass rate: 85%+
- [ ] Zero critical failures
- [ ] Wave 145 report completed
- [ ] CLAUDE.md updated

---

## Rollback Plan

If E2E tests still fail after JWT configuration:

1. **Immediate Rollback** (5 minutes):
   ```bash
   git checkout docker-compose.yml  # Revert changes
   docker-compose restart trading_service backtesting_service ml_training_service
   ```

2. **Investigation** (30-60 minutes):
   - Check if JWT_SECRET in .env matches API Gateway
   - Verify JWT_ISSUER and JWT_AUDIENCE are consistent
   - Review service logs for specific JWT validation errors
   - Test JWT generation with `jwt_token_generator.sh`

3. **Alternative Solution** (4-8 hours):
   - Implement Service Trust Model (Option B)
   - Remove JWT validation from backend services
   - Use API Gateway metadata only

---

## Files Modified

### Configuration Files (1 file)
1. `docker-compose.yml` - Add JWT env vars to 3 services (9 lines added)

### No Code Changes Required ✅
- All fixes are configuration-only
- No Rust code modifications
- No protocol buffer changes
- No test code changes (already fixed in Wave 144)

---

## Agent Coordination

### Dependencies
- **Agents 331-333**: Independent (can run in parallel)
- **Agent 334**: Must complete before Agent 335
- **Agent 335**: Blocking for Agents 336-339
- **Agents 336-339**: Independent (can run in parallel)
- **Agents 340-341**: Independent (can run in parallel)
- **Agent 342**: Depends on all previous agents

### Communication
- Each agent reports results to Agent 342 (coordinator)
- Agent 335 signals "services ready" before validation agents start
- Agent 342 makes final commit recommendation

---

**Status**: READY FOR EXECUTION
**Confidence**: 99% (zen thinkdeep: "almost_certain")
**Expected Success Rate**: 85%+ E2E test pass rate
**Timeline**: 60-85 minutes
**Risk Level**: LOW ✅