foxhunt/SECURITY_HARDENING_CHECKLIST.md

Security Hardening Checklist - Foxhunt HFT Trading System

Date: 2025-10-18 Agent: SECURITY-01 Total Time: 7.5 hours (3 hours minimum for production)

---

🚨 P0: PRODUCTION BLOCKERS (3 HOURS - REQUIRED)

[ ] Task 1: Generate Production Database Passwords (1 hour)

Files to Modify: docker-compose.yml, Vault

# Step 1: Generate secure passwords (20 minutes)
export POSTGRES_PASSWORD=$(openssl rand -base64 32)
export GRAFANA_PASSWORD=$(openssl rand -base64 24)
export MINIO_PASSWORD=$(openssl rand -base64 32)
export INFLUXDB_PASSWORD=$(openssl rand -base64 32)
export VAULT_TOKEN=$(openssl rand -hex 16)

# Step 2: Store in Vault (15 minutes)
vault kv put secret/foxhunt/postgres password="$POSTGRES_PASSWORD"
vault kv put secret/foxhunt/grafana password="$GRAFANA_PASSWORD"
vault kv put secret/foxhunt/minio password="$MINIO_PASSWORD"
vault kv put secret/foxhunt/influxdb password="$INFLUXDB_PASSWORD"

# Step 3: Update docker-compose.yml (15 minutes)
# Find these lines and replace:
# Line 11:  POSTGRES_PASSWORD: foxhunt_dev_password
# Line 51:  DOCKER_INFLUXDB_INIT_PASSWORD: foxhunt_dev_password
# Line 73:  VAULT_DEV_ROOT_TOKEN_ID: foxhunt-dev-root
# Line 124: GF_SECURITY_ADMIN_PASSWORD=foxhunt123
# Line 147: MINIO_ROOT_PASSWORD: foxhunt_dev_password

# Replace with:
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
DOCKER_INFLUXDB_INIT_PASSWORD: ${INFLUXDB_PASSWORD}
VAULT_DEV_ROOT_TOKEN_ID: ${VAULT_TOKEN}
GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_PASSWORD}
MINIO_ROOT_PASSWORD: ${MINIO_PASSWORD}

# Step 4: Test (10 minutes)
docker-compose down
docker-compose up -d
cargo sqlx migrate run
cargo test -p api_gateway -- test_database_connection

Validation Checklist:

• All 5 services start without errors
• SQLx migrations apply successfully
• JWT authentication works end-to-end
• grep -r "foxhunt_dev_password" . returns 0 results (except .env.example)
• No plaintext passwords in git history

---

[ ] Task 2: Implement OCSP Certificate Revocation (1 hour)

Files to Modify: services/api_gateway/src/auth/mtls/revocation.rs

# Step 1: Add dependencies (5 minutes)
cd services/api_gateway
cargo add reqwest
cargo add x509-parser --features verify

# Step 2: Implement OCSP client (40 minutes)
# Edit: services/api_gateway/src/auth/mtls/revocation.rs
# Replace lines 152-160 (check_ocsp_revocation stub) with full implementation
# See AGENT_SECURITY_01_COMPREHENSIVE_AUDIT.md for complete code

# Step 3: Test (15 minutes)
cargo test -p api_gateway -- test_ocsp_revocation
cargo test -p api_gateway -- test_certificate_revoked_via_ocsp
cargo test -p api_gateway -- test_ocsp_fail_closed

Validation Checklist:

• OCSP tests pass
• Revoked certificates are rejected
• Fail-closed policy works (deny on OCSP timeout)
• OCSP responder URL configured in docker-compose.yml

---

[ ] Task 3: Enable TLS for All Services (1 hour)

Files to Modify: docker-compose.yml, service URLs

# Step 1: Generate production certificates (20 minutes)
cd certs/
./scripts/generate_production_certs.sh

# Verify generated files:
ls -la certs/ca/ca-cert.pem
ls -la certs/server-cert.pem
ls -la certs/server-key.pem
ls -la certs/client-cert.pem
ls -la certs/client-key.pem

# Step 2: Update docker-compose.yml (10 minutes)
# Find and replace ALL 5 occurrences:
TLS_ENABLED: ${TLS_ENABLED:-false}
# Replace with:
TLS_ENABLED: "true"

# Also update:
TLS_PROTOCOL_VERSION: "TLS13"
TLS_REQUIRE_CLIENT_CERT: "true"

# Step 3: Update service endpoints (15 minutes)
# Change ALL http:// to https:// in docker-compose.yml:
TRADING_SERVICE_URL: https://trading_service:50051
BACKTESTING_SERVICE_URL: https://backtesting_service:50053
ML_TRAINING_SERVICE_URL: https://ml_training_service:50053

# Step 4: Test (15 minutes)
docker-compose restart
cargo test -p integration_tests -- test_mtls_authentication
cargo test -p integration_tests -- test_tls_version_enforcement

# Verify with openssl:
openssl s_client -connect localhost:50051 -showcerts

Validation Checklist:

• All 5 services start with TLS enabled
• gRPC calls use TLS 1.3 (verify with openssl s_client)
• Client certificate validation works
• Wireshark shows encrypted traffic (no plaintext)
• grpcurl -plaintext localhost:50051 list FAILS

---

⚠️ P1: HIGH PRIORITY FIXES (1.5 HOURS - RECOMMENDED)

[ ] Task 4: Generate Production JWT Secret (15 minutes)

Files to Modify: .env.production, Vault

# Generate 128-character secret (512-bit security)
export JWT_SECRET=$(openssl rand -base64 96 | tr -d '\n')

# Validate
echo $JWT_SECRET | wc -c  # Should be 128+ chars
echo $JWT_SECRET | grep -o '[A-Za-z0-9+/]' | sort -u | wc -l  # Should be 60+

# Store in Vault
vault kv put secret/foxhunt/jwt \
  jwt_secret="$JWT_SECRET" \
  jwt_issuer="foxhunt-api-gateway" \
  jwt_audience="foxhunt-services" \
  rotation_date="2025-10-18"

# Update .env.production
echo "JWT_SECRET=$JWT_SECRET" >> .env.production

# Remove weak default from docker-compose.yml
# Find: JWT_SECRET=${JWT_SECRET:-dev_secret_key_change_in_production}
# Replace: JWT_SECRET=${JWT_SECRET}  # No fallback - MUST be set

# Test
cargo test -p config -- test_jwt_config_validation
cargo test -p api_gateway -- test_jwt_signing_verification

Validation Checklist:

• Secret is 128+ characters
• Secret passes entropy checks (3+ char types)
• JWT signing/verification works
• No dev_secret_key_change_in_production in configs
• Application fails to start if JWT_SECRET not set

---

[ ] Task 5: Implement TOTP Nonce Tracking (45 minutes)

Files to Modify: services/api_gateway/src/auth/mfa/totp.rs

// Add to services/api_gateway/src/auth/mfa/totp.rs

pub async fn verify_with_nonce_check(
    &self,
    secret: &str,
    code: &str,
    user_id: &str,
    redis: &redis::Client,
) -> Result<bool> {
    // Check if code was already used (replay attack prevention)
    let nonce_key = format!("totp:nonce:{}:{}", user_id, code);

    if redis.exists(&nonce_key).await? {
        warn!("TOTP replay attack detected: user_id={}, code={}", user_id, code);
        return Err(anyhow::anyhow!("TOTP code already used (replay attack)"));
    }

    // Verify code against secret
    let valid = self.verify(secret, code)?;

    if valid {
        // Store nonce with 60-second TTL (covers 2 time periods)
        redis.set_ex(&nonce_key, "1", 60).await?;
        info!("TOTP code validated and nonce stored: user_id={}", user_id);
    }

    Ok(valid)
}

# Test
cargo test -p api_gateway -- test_totp_replay_prevention_with_nonce
cargo test -p api_gateway -- test_totp_nonce_expiration

# Update auth flow to use new method
# File: services/api_gateway/src/auth/mfa/verification.rs
# Replace: verifier.verify(secret, code)
# With: verifier.verify_with_nonce_check(secret, code, user_id, redis)

Validation Checklist:

• Tests pass
• TOTP codes cannot be reused
• Nonces expire after 60 seconds
• Redis connection failures handled gracefully

---

[ ] Task 6: Encrypt TLI Token Storage (30 minutes)

Files to Modify: tli/src/auth/token_storage.rs

# Add dependencies
cd tli
cargo add keyring
cargo add aes-gcm

# Implement encrypted storage (see AGENT_SECURITY_01_COMPREHENSIVE_AUDIT.md)

# Test
cargo test -p tli -- test_encrypted_token_storage
cargo test -p tli -- test_keyring_fallback
cargo test -p tli -- test_token_encryption_decryption

Validation Checklist:

• OS keyring integration works (macOS, Windows, Linux)
• AES-256-GCM fallback works
• Existing tokens migrated to encrypted storage
• Decryption works after restart

---

📋 P2: MEDIUM PRIORITY ENHANCEMENTS (3 HOURS - OPTIONAL)

[ ] Task 7: Brute Force Protection (1.5 hours)

Files to Modify: services/api_gateway/src/auth/jwt/service.rs

// Add rate limiting for failed attempts
// See AGENT_SECURITY_01_COMPREHENSIVE_AUDIT.md for complete implementation

Validation Checklist:

• Failed attempts tracked in Redis
• Account locks after 5 failures
• Lock duration: 15 minutes
• Automatic unlock after timeout

---

[ ] Task 8: Enhanced Audit Logging (1.5 hours)

Files to Modify: services/api_gateway/src/audit/logger.rs

// Implement comprehensive audit logging
// - Failed authentication attempts
// - PII access tracking
// - Database query trail

Validation Checklist:

• Failed auth attempts logged
• PII access tracked
• Logs shipped to InfluxDB + PostgreSQL
• Grafana dashboards updated

---

✅ FINAL VALIDATION (30 minutes)

Security Smoke Tests

# Run all security tests
cargo test --workspace -- security
cargo test --workspace -- auth
cargo test --workspace -- mfa

# Run integration tests
cargo test -p integration_tests

# Check for vulnerabilities
cargo audit

# Verify TLS
openssl s_client -connect localhost:50051 -showcerts
openssl s_client -connect localhost:50052 -showcerts
openssl s_client -connect localhost:50053 -showcerts
openssl s_client -connect localhost:50054 -showcerts

# Verify no hardcoded secrets
grep -r "foxhunt_dev_password" . --exclude-dir=target
grep -r "dev_secret_key_change_in_production" . --exclude-dir=target
grep -r "foxhunt-dev-root" . --exclude-dir=target

# Verify credential strength
echo "JWT_SECRET length: $(echo $JWT_SECRET | wc -c)"
echo "POSTGRES_PASSWORD length: $(echo $POSTGRES_PASSWORD | wc -c)"

Compliance Checks

• PCI DSS Req 2.3: Encryption in transit (TLS enabled)
• PCI DSS Req 8.2.1: Unique passwords (no hardcoded creds)
• PCI DSS Req 6.5.1: SQL Injection (SQLx macros)
• SOC2 CC6.1: Encryption controls (TLS + AES-256-GCM)
• SOC2 CC6.6: Authentication (JWT + MFA + RBAC)
• SOC2 CC6.7: Secrets management (Vault + SecretString)

Production Readiness

• All P0 tasks complete
• All P1 tasks complete (recommended)
• Integration tests passing
• TLS operational on all services
• OCSP revocation working
• No hardcoded credentials
• Grafana dashboards monitoring security metrics
• Penetration test scheduled (external)

---

📊 PROGRESS TRACKER

P0: Production Blockers (REQUIRED)

• Task 1: Database passwords (1h)
• Task 2: OCSP implementation (1h)
• Task 3: TLS enablement (1h) Total P0: 3 hours

P1: High Priority (RECOMMENDED)

• Task 4: JWT secret (15m)
• Task 5: TOTP nonce tracking (45m)
• Task 6: TLI encryption (30m) Total P1: 1.5 hours

P2: Medium Priority (OPTIONAL)

• Task 7: Brute force protection (1.5h)
• Task 8: Audit logging (1.5h) Total P2: 3 hours

GRAND TOTAL: 7.5 hours

---

🎯 SUCCESS CRITERIA

Minimum (Production Deployment)

• All P0 tasks complete
• Integration tests passing
• No hardcoded credentials
• TLS operational

Recommended (Secure Production)

• All P0 + P1 tasks complete
• MFA replay attack prevented
• TLI tokens encrypted
• Strong JWT secret

Full Hardening (Enterprise-Grade)

• All P0 + P1 + P2 tasks complete
• Brute force protection
• Comprehensive audit logging
• External pentest passed

---

Next Steps: Start with P0 Task 1 (Database Passwords) Time to Production: 3 hours (P0 only) or 4.5 hours (P0 + P1 recommended)