foxhunt/SECURITY_HARDENING_CHECKLIST.md

# Security Hardening Checklist - Foxhunt HFT Trading System

**Date**: 2025-10-18
**Agent**: SECURITY-01
**Total Time**: 7.5 hours (3 hours minimum for production)

---

## 🚨 P0: PRODUCTION BLOCKERS (3 HOURS - REQUIRED)

### [ ] Task 1: Generate Production Database Passwords (1 hour)

**Files to Modify**: `docker-compose.yml`, Vault

```bash
# Step 1: Generate secure passwords (20 minutes)
export POSTGRES_PASSWORD=$(openssl rand -base64 32)
export GRAFANA_PASSWORD=$(openssl rand -base64 24)
export MINIO_PASSWORD=$(openssl rand -base64 32)
export INFLUXDB_PASSWORD=$(openssl rand -base64 32)
export VAULT_TOKEN=$(openssl rand -hex 16)

# Step 2: Store in Vault (15 minutes)
vault kv put secret/foxhunt/postgres password="$POSTGRES_PASSWORD"
vault kv put secret/foxhunt/grafana password="$GRAFANA_PASSWORD"
vault kv put secret/foxhunt/minio password="$MINIO_PASSWORD"
vault kv put secret/foxhunt/influxdb password="$INFLUXDB_PASSWORD"

# Step 3: Update docker-compose.yml (15 minutes)
# Find these lines and replace:
# Line 11:  POSTGRES_PASSWORD: foxhunt_dev_password
# Line 51:  DOCKER_INFLUXDB_INIT_PASSWORD: foxhunt_dev_password
# Line 73:  VAULT_DEV_ROOT_TOKEN_ID: foxhunt-dev-root
# Line 124: GF_SECURITY_ADMIN_PASSWORD=foxhunt123
# Line 147: MINIO_ROOT_PASSWORD: foxhunt_dev_password

# Replace with:
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
DOCKER_INFLUXDB_INIT_PASSWORD: ${INFLUXDB_PASSWORD}
VAULT_DEV_ROOT_TOKEN_ID: ${VAULT_TOKEN}
GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_PASSWORD}
MINIO_ROOT_PASSWORD: ${MINIO_PASSWORD}

# Step 4: Test (10 minutes)
docker-compose down
docker-compose up -d
cargo sqlx migrate run
cargo test -p api_gateway -- test_database_connection
```

**Validation Checklist**:
- [ ] All 5 services start without errors
- [ ] SQLx migrations apply successfully
- [ ] JWT authentication works end-to-end
- [ ] `grep -r "foxhunt_dev_password" .` returns 0 results (except .env.example)
- [ ] No plaintext passwords in git history

---

### [ ] Task 2: Implement OCSP Certificate Revocation (1 hour)

**Files to Modify**: `services/api_gateway/src/auth/mtls/revocation.rs`

```bash
# Step 1: Add dependencies (5 minutes)
cd services/api_gateway
cargo add reqwest
cargo add x509-parser --features verify

# Step 2: Implement OCSP client (40 minutes)
# Edit: services/api_gateway/src/auth/mtls/revocation.rs
# Replace lines 152-160 (check_ocsp_revocation stub) with full implementation
# See AGENT_SECURITY_01_COMPREHENSIVE_AUDIT.md for complete code

# Step 3: Test (15 minutes)
cargo test -p api_gateway -- test_ocsp_revocation
cargo test -p api_gateway -- test_certificate_revoked_via_ocsp
cargo test -p api_gateway -- test_ocsp_fail_closed
```

**Validation Checklist**:
- [ ] OCSP tests pass
- [ ] Revoked certificates are rejected
- [ ] Fail-closed policy works (deny on OCSP timeout)
- [ ] OCSP responder URL configured in docker-compose.yml

---

### [ ] Task 3: Enable TLS for All Services (1 hour)

**Files to Modify**: `docker-compose.yml`, service URLs

```bash
# Step 1: Generate production certificates (20 minutes)
cd certs/
./scripts/generate_production_certs.sh

# Verify generated files:
ls -la certs/ca/ca-cert.pem
ls -la certs/server-cert.pem
ls -la certs/server-key.pem
ls -la certs/client-cert.pem
ls -la certs/client-key.pem

# Step 2: Update docker-compose.yml (10 minutes)
# Find and replace ALL 5 occurrences:
TLS_ENABLED: ${TLS_ENABLED:-false}
# Replace with:
TLS_ENABLED: "true"

# Also update:
TLS_PROTOCOL_VERSION: "TLS13"
TLS_REQUIRE_CLIENT_CERT: "true"

# Step 3: Update service endpoints (15 minutes)
# Change ALL http:// to https:// in docker-compose.yml:
TRADING_SERVICE_URL: https://trading_service:50051
BACKTESTING_SERVICE_URL: https://backtesting_service:50053
ML_TRAINING_SERVICE_URL: https://ml_training_service:50053

# Step 4: Test (15 minutes)
docker-compose restart
cargo test -p integration_tests -- test_mtls_authentication
cargo test -p integration_tests -- test_tls_version_enforcement

# Verify with openssl:
openssl s_client -connect localhost:50051 -showcerts
```

**Validation Checklist**:
- [ ] All 5 services start with TLS enabled
- [ ] gRPC calls use TLS 1.3 (verify with `openssl s_client`)
- [ ] Client certificate validation works
- [ ] Wireshark shows encrypted traffic (no plaintext)
- [ ] `grpcurl -plaintext localhost:50051 list` FAILS

---

## ⚠️ P1: HIGH PRIORITY FIXES (1.5 HOURS - RECOMMENDED)

### [ ] Task 4: Generate Production JWT Secret (15 minutes)

**Files to Modify**: `.env.production`, Vault

```bash
# Generate 128-character secret (512-bit security)
export JWT_SECRET=$(openssl rand -base64 96 | tr -d '\n')

# Validate
echo $JWT_SECRET | wc -c  # Should be 128+ chars
echo $JWT_SECRET | grep -o '[A-Za-z0-9+/]' | sort -u | wc -l  # Should be 60+

# Store in Vault
vault kv put secret/foxhunt/jwt \
  jwt_secret="$JWT_SECRET" \
  jwt_issuer="foxhunt-api-gateway" \
  jwt_audience="foxhunt-services" \
  rotation_date="2025-10-18"

# Update .env.production
echo "JWT_SECRET=$JWT_SECRET" >> .env.production

# Remove weak default from docker-compose.yml
# Find: JWT_SECRET=${JWT_SECRET:-dev_secret_key_change_in_production}
# Replace: JWT_SECRET=${JWT_SECRET}  # No fallback - MUST be set

# Test
cargo test -p config -- test_jwt_config_validation
cargo test -p api_gateway -- test_jwt_signing_verification
```

**Validation Checklist**:
- [ ] Secret is 128+ characters
- [ ] Secret passes entropy checks (3+ char types)
- [ ] JWT signing/verification works
- [ ] No `dev_secret_key_change_in_production` in configs
- [ ] Application fails to start if JWT_SECRET not set

---

### [ ] Task 5: Implement TOTP Nonce Tracking (45 minutes)

**Files to Modify**: `services/api_gateway/src/auth/mfa/totp.rs`

```rust
// Add to services/api_gateway/src/auth/mfa/totp.rs

pub async fn verify_with_nonce_check(
    &self,
    secret: &str,
    code: &str,
    user_id: &str,
    redis: &redis::Client,
) -> Result<bool> {
    // Check if code was already used (replay attack prevention)
    let nonce_key = format!("totp:nonce:{}:{}", user_id, code);

    if redis.exists(&nonce_key).await? {
        warn!("TOTP replay attack detected: user_id={}, code={}", user_id, code);
        return Err(anyhow::anyhow!("TOTP code already used (replay attack)"));
    }

    // Verify code against secret
    let valid = self.verify(secret, code)?;

    if valid {
        // Store nonce with 60-second TTL (covers 2 time periods)
        redis.set_ex(&nonce_key, "1", 60).await?;
        info!("TOTP code validated and nonce stored: user_id={}", user_id);
    }

    Ok(valid)
}
```

```bash
# Test
cargo test -p api_gateway -- test_totp_replay_prevention_with_nonce
cargo test -p api_gateway -- test_totp_nonce_expiration

# Update auth flow to use new method
# File: services/api_gateway/src/auth/mfa/verification.rs
# Replace: verifier.verify(secret, code)
# With: verifier.verify_with_nonce_check(secret, code, user_id, redis)
```

**Validation Checklist**:
- [ ] Tests pass
- [ ] TOTP codes cannot be reused
- [ ] Nonces expire after 60 seconds
- [ ] Redis connection failures handled gracefully

---

### [ ] Task 6: Encrypt TLI Token Storage (30 minutes)

**Files to Modify**: `tli/src/auth/token_storage.rs`

```bash
# Add dependencies
cd tli
cargo add keyring
cargo add aes-gcm

# Implement encrypted storage (see AGENT_SECURITY_01_COMPREHENSIVE_AUDIT.md)

# Test
cargo test -p tli -- test_encrypted_token_storage
cargo test -p tli -- test_keyring_fallback
cargo test -p tli -- test_token_encryption_decryption
```

**Validation Checklist**:
- [ ] OS keyring integration works (macOS, Windows, Linux)
- [ ] AES-256-GCM fallback works
- [ ] Existing tokens migrated to encrypted storage
- [ ] Decryption works after restart

---

## 📋 P2: MEDIUM PRIORITY ENHANCEMENTS (3 HOURS - OPTIONAL)

### [ ] Task 7: Brute Force Protection (1.5 hours)

**Files to Modify**: `services/api_gateway/src/auth/jwt/service.rs`

```rust
// Add rate limiting for failed attempts
// See AGENT_SECURITY_01_COMPREHENSIVE_AUDIT.md for complete implementation
```

**Validation Checklist**:
- [ ] Failed attempts tracked in Redis
- [ ] Account locks after 5 failures
- [ ] Lock duration: 15 minutes
- [ ] Automatic unlock after timeout

---

### [ ] Task 8: Enhanced Audit Logging (1.5 hours)

**Files to Modify**: `services/api_gateway/src/audit/logger.rs`

```rust
// Implement comprehensive audit logging
// - Failed authentication attempts
// - PII access tracking
// - Database query trail
```

**Validation Checklist**:
- [ ] Failed auth attempts logged
- [ ] PII access tracked
- [ ] Logs shipped to InfluxDB + PostgreSQL
- [ ] Grafana dashboards updated

---

## ✅ FINAL VALIDATION (30 minutes)

### Security Smoke Tests

```bash
# Run all security tests
cargo test --workspace -- security
cargo test --workspace -- auth
cargo test --workspace -- mfa

# Run integration tests
cargo test -p integration_tests

# Check for vulnerabilities
cargo audit

# Verify TLS
openssl s_client -connect localhost:50051 -showcerts
openssl s_client -connect localhost:50052 -showcerts
openssl s_client -connect localhost:50053 -showcerts
openssl s_client -connect localhost:50054 -showcerts

# Verify no hardcoded secrets
grep -r "foxhunt_dev_password" . --exclude-dir=target
grep -r "dev_secret_key_change_in_production" . --exclude-dir=target
grep -r "foxhunt-dev-root" . --exclude-dir=target

# Verify credential strength
echo "JWT_SECRET length: $(echo $JWT_SECRET | wc -c)"
echo "POSTGRES_PASSWORD length: $(echo $POSTGRES_PASSWORD | wc -c)"
```

### Compliance Checks

- [ ] PCI DSS Req 2.3: Encryption in transit (TLS enabled)
- [ ] PCI DSS Req 8.2.1: Unique passwords (no hardcoded creds)
- [ ] PCI DSS Req 6.5.1: SQL Injection (SQLx macros)
- [ ] SOC2 CC6.1: Encryption controls (TLS + AES-256-GCM)
- [ ] SOC2 CC6.6: Authentication (JWT + MFA + RBAC)
- [ ] SOC2 CC6.7: Secrets management (Vault + SecretString)

### Production Readiness

- [ ] All P0 tasks complete
- [ ] All P1 tasks complete (recommended)
- [ ] Integration tests passing
- [ ] TLS operational on all services
- [ ] OCSP revocation working
- [ ] No hardcoded credentials
- [ ] Grafana dashboards monitoring security metrics
- [ ] Penetration test scheduled (external)

---

## 📊 PROGRESS TRACKER

### P0: Production Blockers (REQUIRED)
- [ ] Task 1: Database passwords (1h)
- [ ] Task 2: OCSP implementation (1h)
- [ ] Task 3: TLS enablement (1h)
**Total P0**: 3 hours

### P1: High Priority (RECOMMENDED)
- [ ] Task 4: JWT secret (15m)
- [ ] Task 5: TOTP nonce tracking (45m)
- [ ] Task 6: TLI encryption (30m)
**Total P1**: 1.5 hours

### P2: Medium Priority (OPTIONAL)
- [ ] Task 7: Brute force protection (1.5h)
- [ ] Task 8: Audit logging (1.5h)
**Total P2**: 3 hours

**GRAND TOTAL**: 7.5 hours

---

## 🎯 SUCCESS CRITERIA

### Minimum (Production Deployment)
- [x] All P0 tasks complete
- [x] Integration tests passing
- [x] No hardcoded credentials
- [x] TLS operational

### Recommended (Secure Production)
- [x] All P0 + P1 tasks complete
- [x] MFA replay attack prevented
- [x] TLI tokens encrypted
- [x] Strong JWT secret

### Full Hardening (Enterprise-Grade)
- [x] All P0 + P1 + P2 tasks complete
- [x] Brute force protection
- [x] Comprehensive audit logging
- [x] External pentest passed

---

**Next Steps**: Start with P0 Task 1 (Database Passwords)
**Time to Production**: 3 hours (P0 only) or 4.5 hours (P0 + P1 recommended)