foxhunt/docs/SECURITY_HARDENING.md

Foxhunt HFT Trading System - Security Hardening Guide

Version: 1.0 Last Updated: 2025-10-03 Wave: 71 - Production Security Audience: Security Engineers, DevOps, System Administrators

---

Table of Contents

1. Security Architecture Overview
2. Compliance Mandates
3. Network Security
4. Host Security
5. Application Security
6. Data Security
7. Secrets Management
8. Authentication & Authorization
9. Logging & Monitoring
10. Incident Response
11. Security Checklist

---

Security Architecture Overview

Defense-in-Depth Strategy

The Foxhunt HFT system implements a 6-layer security architecture:

┌─────────────────────────────────────────────────────────────┐
│               FOXHUNT SECURITY ARCHITECTURE                 │
└─────────────────────────────────────────────────────────────┘

Layer 1: NETWORK PERIMETER
├─ Firewall/WAF (external)
├─ DDoS protection
├─ IP allowlisting
└─ TLS 1.3 termination

Layer 2: API GATEWAY (6-Layer Auth)
├─ JWT validation (HS512)
├─ JWT revocation check (Redis)
├─ MFA/TOTP (RFC 6238)
├─ RBAC authorization
├─ Rate limiting (100 req/s)
└─ Audit logging

Layer 3: MUTUAL TLS (Inter-Service)
├─ X.509 client certificates
├─ CA-signed service certificates
└─ Certificate pinning

Layer 4: APPLICATION SECURITY
├─ Input validation
├─ SQL injection prevention (parameterized queries)
├─ Dependency vulnerability scanning
└─ Secure coding practices (Rust memory safety)

Layer 5: DATA SECURITY
├─ Encryption at rest (PostgreSQL, Redis, S3)
├─ Encryption in transit (TLS 1.3)
└─ Database access control (RBAC)

Layer 6: AUDIT & MONITORING
├─ Immutable audit trails (SOX/MiFID II)
├─ SIEM integration
├─ Security event alerting
└─ Compliance reporting

Security Principles

1. Least Privilege: Every user, service, and process has minimum required permissions
2. Zero Trust: No implicit trust based on network location
3. Defense-in-Depth: Multiple overlapping security layers
4. Fail Secure: System fails to a secure state during errors
5. Audit Everything: Comprehensive logging for compliance and forensics

---

Compliance Mandates

SOX (Sarbanes-Oxley Act)

Requirements:

• Audit trails for all financial transactions
• Access controls for financial systems
• Data integrity and non-repudiation
• Separation of duties

Implementation:

• ✅ Immutable audit trails in audit_events table
• ✅ RBAC with role-permission separation
• ✅ Database-level constraints for data integrity
• ✅ Multi-signature approval workflows (future enhancement)

Verification:

-- Verify audit trail immutability
SELECT COUNT(*) FROM audit_events;  -- Should never decrease

-- Verify role separation
SELECT u.username, r.name FROM users u
JOIN user_roles ur ON u.id = ur.user_id
JOIN roles r ON ur.role_id = r.id
WHERE r.name IN ('admin', 'trader', 'auditor');

MiFID II (Markets in Financial Instruments Directive)

Requirements:

• Microsecond-precision timestamping
• Trade reconstruction capability
• Best execution reporting
• Client order handling transparency

Implementation:

• ✅ NTP/PTP time synchronization (±100μs)
• ✅ Comprehensive order lifecycle logging
• ✅ Execution venue tracking
• ✅ Order routing audit trails

Verification:

# Verify time synchronization
chronyc tracking
# Expected: System time offset < 100μs

# Check order timestamps
psql $DATABASE_URL -c "SELECT order_id, created_at FROM orders LIMIT 5;"

Data Retention Policies

Data Type	Retention Period	Storage Location
Audit Trails	7 years	PostgreSQL + S3
Trade Records	7 years	PostgreSQL + S3
User Activity Logs	1 year	ELK/Loki
System Logs	90 days	ELK/Loki
MFA Backup Codes	Until used	PostgreSQL (encrypted)

Implementation:

-- Automated data archival (run monthly)
CREATE OR REPLACE FUNCTION archive_old_audit_events() RETURNS void AS $$
BEGIN
    -- Archive audit events older than 7 years to S3
    COPY (SELECT * FROM audit_events WHERE created_at < NOW() - INTERVAL '7 years')
    TO PROGRAM 'aws s3 cp - s3://foxhunt-archives/audit_events_$(date +%Y%m).csv';

    -- DO NOT DELETE (compliance requirement)
END;
$$ LANGUAGE plpgsql;

---

Network Security

Firewall Rules

Inbound Rules (iptables):

#!/bin/bash
# firewall_rules.sh - Production firewall configuration

# Flush existing rules
iptables -F
iptables -X

# Default policies: DENY all
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT

# Allow loopback
iptables -A INPUT -i lo -j ACCEPT

# Allow established connections
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allow SSH (from management IPs only)
iptables -A INPUT -p tcp --dport 22 -s 10.0.0.0/8 -j ACCEPT

# Allow API Gateway (from trusted sources only)
iptables -A INPUT -p tcp --dport 50051 -s 203.0.113.0/24 -j ACCEPT  # Exchange IPs
iptables -A INPUT -p tcp --dport 8080 -s 10.0.0.0/8 -j ACCEPT       # Internal monitoring

# Allow PostgreSQL (from app servers only)
iptables -A INPUT -p tcp --dport 5432 -s 10.0.1.0/24 -j ACCEPT

# Allow Redis (from app servers only)
iptables -A INPUT -p tcp --dport 6379 -s 10.0.1.0/24 -j ACCEPT

# Rate limiting (anti-DDoS)
iptables -A INPUT -p tcp --dport 50051 -m hashlimit \
    --hashlimit-name api_gateway \
    --hashlimit-above 1000/sec \
    --hashlimit-mode srcip \
    --hashlimit-burst 2000 \
    -j DROP

# Log dropped packets
iptables -A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables-dropped: "

# Save rules
iptables-save > /etc/iptables/rules.v4

Apply firewall rules:

sudo bash firewall_rules.sh

VPC/Subnet Segmentation

Recommended AWS VPC Architecture:

VPC: 10.0.0.0/16
├─ Public Subnet (DMZ): 10.0.0.0/24
│  ├─ NAT Gateway
│  └─ Bastion Host (SSH jump box)
│
├─ Private Subnet (Application): 10.0.1.0/24
│  ├─ API Gateway
│  ├─ Trading Service
│  ├─ Backtesting Service
│  └─ ML Training Service
│
└─ Private Subnet (Data): 10.0.2.0/24
   ├─ PostgreSQL (primary + replica)
   ├─ Redis (cluster)
   └─ S3 VPC Endpoint

Security Groups:

# API Gateway SG
Inbound:
  - Port 50051 (gRPC): 0.0.0.0/0 (internet)
  - Port 8080 (health): 10.0.0.0/16 (internal monitoring)
Outbound:
  - Port 5432: 10.0.2.0/24 (PostgreSQL)
  - Port 6379: 10.0.2.0/24 (Redis)

# Backend Services SG
Inbound:
  - Port 50052-50054 (gRPC): 10.0.1.0/24 (API Gateway only)
Outbound:
  - Port 5432: 10.0.2.0/24 (PostgreSQL)
  - Port 6379: 10.0.2.0/24 (Redis)
  - Port 443: 0.0.0.0/0 (S3 HTTPS)

# Database SG
Inbound:
  - Port 5432: 10.0.1.0/24 (backend services)
  - Port 6379: 10.0.1.0/24 (backend services)
Outbound:
  - DENY all (databases should not initiate outbound)

Mutual TLS Enforcement

Verify mTLS is enforced:

# Test without client certificate (should fail)
curl https://trading-service:50052/health
# Expected: certificate required

# Test with valid client certificate (should succeed)
curl --cert /etc/foxhunt/certs/client.crt \
     --key /etc/foxhunt/certs/client.key \
     --cacert /etc/foxhunt/certs/ca.crt \
     https://trading-service:50052/health
# Expected: {"status":"ok"}

Tonic gRPC mTLS Configuration:

// services/trading_service/src/tls_config.rs
use tonic::transport::{Certificate, Identity, ServerTlsConfig};

pub fn create_tls_config() -> ServerTlsConfig {
    let cert = std::fs::read("/etc/foxhunt/trading_service/trading_service.crt").unwrap();
    let key = std::fs::read("/etc/foxhunt/trading_service/trading_service.key").unwrap();
    let ca_cert = std::fs::read("/etc/foxhunt/certs/ca.crt").unwrap();

    let identity = Identity::from_pem(cert, key);
    let client_ca = Certificate::from_pem(ca_cert);

    ServerTlsConfig::new()
        .identity(identity)
        .client_ca_root(client_ca)  // Enforce client certificate validation
}

TLS 1.3 Configuration

Strong Cipher Suites (in order of preference):

// Recommended ciphers for Rust/Tonic
const STRONG_CIPHERS: &[&str] = &[
    "TLS_AES_256_GCM_SHA384",
    "TLS_AES_128_GCM_SHA256",
    "TLS_CHACHA20_POLY1305_SHA256",
];

// Disable weak ciphers
const FORBIDDEN_CIPHERS: &[&str] = &[
    "TLS_RSA_*",           // No forward secrecy
    "TLS_*_CBC_*",         // Vulnerable to BEAST/POODLE
    "TLS_*_3DES_*",        // Weak encryption
];

Nginx Proxy (if used):

server {
    listen 443 ssl http2;

    # TLS 1.3 only
    ssl_protocols TLSv1.3;

    # Strong ciphers
    ssl_ciphers 'TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256:TLS_CHACHA20_POLY1305_SHA256';
    ssl_prefer_server_ciphers on;

    # Certificate pinning (HPKP)
    add_header Public-Key-Pins 'pin-sha256="BASE64_HASH"; max-age=2592000';

    # Security headers
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
    add_header X-Frame-Options "DENY" always;
    add_header X-Content-Type-Options "nosniff" always;

    # Proxy to API Gateway
    location / {
        grpc_pass grpc://localhost:50051;
    }
}

DDoS Protection

Rate Limiting (already implemented in API Gateway):

// services/api_gateway/src/auth/rate_limiter.rs
pub struct RateLimiter {
    limit: u32,  // 100 req/s per user
    // Token bucket algorithm
}

CloudFlare/AWS Shield (if using cloud):

• Enable CloudFlare Pro with DDoS protection
• Enable AWS Shield Standard (free) or Advanced (paid)
• Configure rate limiting rules at edge

---

Host Security

OS Hardening (CIS Benchmarks)

Apply CIS Ubuntu 22.04 Benchmark:

#!/bin/bash
# cis_hardening.sh - CIS Level 1 hardening

# 1. Disable unnecessary services
sudo systemctl disable bluetooth.service
sudo systemctl disable cups.service
sudo systemctl disable avahi-daemon.service

# 2. Enable firewall
sudo ufw enable
sudo ufw default deny incoming
sudo ufw default allow outgoing

# 3. Secure SSH
sudo sed -i 's/#PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config
sudo sed -i 's/#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
sudo sed -i 's/#PubkeyAuthentication yes/PubkeyAuthentication yes/' /etc/ssh/sshd_config
sudo systemctl restart sshd

# 4. Enable automatic security updates
sudo apt install -y unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades

# 5. Set file permissions
sudo chmod 600 /etc/shadow
sudo chmod 600 /etc/gshadow
sudo chmod 644 /etc/passwd
sudo chmod 644 /etc/group

# 6. Disable core dumps
echo "* hard core 0" | sudo tee -a /etc/security/limits.conf

# 7. Enable ASLR (address space layout randomization)
sudo sysctl -w kernel.randomize_va_space=2

# 8. Disable IP forwarding (unless needed)
sudo sysctl -w net.ipv4.ip_forward=0

File System Permissions

Critical Files:

# Binaries
sudo chown root:root /usr/local/bin/{api_gateway,trading_service,backtesting_service,ml_training_service}
sudo chmod 755 /usr/local/bin/{api_gateway,trading_service,backtesting_service,ml_training_service}

# Configuration files
sudo chown foxhunt:foxhunt /etc/foxhunt/*/.env
sudo chmod 600 /etc/foxhunt/*/.env

# TLS private keys
sudo chown foxhunt:foxhunt /etc/foxhunt/certs/*.key
sudo chmod 400 /etc/foxhunt/certs/*.key

# TLS certificates
sudo chown foxhunt:foxhunt /etc/foxhunt/certs/*.crt
sudo chmod 644 /etc/foxhunt/certs/*.crt

# Logs
sudo chown foxhunt:foxhunt /var/log/foxhunt/
sudo chmod 750 /var/log/foxhunt/

Regular Patching

Automated Security Updates:

# Configure unattended-upgrades
sudo tee /etc/apt/apt.conf.d/50unattended-upgrades <<EOF
Unattended-Upgrade::Allowed-Origins {
    "\${distro_id}:\${distro_codename}-security";
    "\${distro_id}ESMApps:\${distro_codename}-apps-security";
};
Unattended-Upgrade::AutoFixInterruptedDpkg "true";
Unattended-Upgrade::MinimalSteps "true";
Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-Time "03:00";
EOF

# Enable automatic updates
sudo systemctl enable unattended-upgrades
sudo systemctl start unattended-upgrades

Manual Patching Schedule:

• Critical patches: Within 24 hours
• Security patches: Within 7 days
• Regular updates: Monthly (during maintenance window)

Intrusion Detection (OSSEC/Fail2Ban)

Install Fail2Ban:

sudo apt install -y fail2ban

# Configure jail for SSH
sudo tee /etc/fail2ban/jail.local <<EOF
[sshd]
enabled = true
port = 22
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
bantime = 3600
findtime = 600

[api-gateway]
enabled = true
port = 50051
filter = api-gateway
logpath = /var/log/foxhunt/api_gateway.log
maxretry = 10
bantime = 600
EOF

# Create custom filter
sudo tee /etc/fail2ban/filter.d/api-gateway.conf <<EOF
[Definition]
failregex = WARN.*Authentication failed for user <HOST>
ignoreregex =
EOF

sudo systemctl restart fail2ban

---

Application Security

Input Validation

API Gateway Input Validation:

// services/api_gateway/src/validation.rs
use validator::{Validate, ValidationError};

#[derive(Debug, Validate)]
pub struct OrderRequest {
    #[validate(length(min = 1, max = 10))]
    pub symbol: String,

    #[validate(range(min = 1, max = 1000000))]
    pub quantity: u64,

    #[validate(custom = "validate_order_side")]
    pub side: String,
}

fn validate_order_side(side: &str) -> Result<(), ValidationError> {
    if !["buy", "sell"].contains(&side) {
        return Err(ValidationError::new("invalid_order_side"));
    }
    Ok(())
}

SQL Injection Prevention

✅ CORRECT (Parameterized Queries):

// trading_engine/src/repositories/trading_repository.rs
pub async fn get_order_by_id(&self, order_id: Uuid) -> Result<Option<Order>> {
    let order = sqlx::query_as!(
        Order,
        "SELECT * FROM orders WHERE id = $1",
        order_id  // Parameterized query
    )
    .fetch_optional(&self.pool)
    .await?;
    Ok(order)
}

❌ INCORRECT (String Concatenation - NEVER DO THIS):

// VULNERABLE TO SQL INJECTION - DO NOT USE
let query = format!("SELECT * FROM orders WHERE id = '{}'", user_input);
sqlx::query(&query).fetch_all(&pool).await?;

Verification:

# Scan codebase for SQL injection vulnerabilities
cd /home/jgrusewski/Work/foxhunt
grep -r "format!.*SELECT" --include="*.rs" .
# Expected: No results (all queries should use $1, $2 placeholders)

Dependency Vulnerability Scanning

Automated Scanning (CI/CD):

# Install cargo-audit
cargo install cargo-audit

# Scan for known vulnerabilities
cargo audit

# Expected output: No vulnerabilities found

Add to CI/CD pipeline (.github/workflows/security.yml):

name: Security Scan
on: [push, pull_request]
jobs:
  audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions-rs/toolchain@v1
        with:
          toolchain: stable
      - run: cargo install cargo-audit
      - run: cargo audit --deny warnings

Rust unsafe Code Review

Policy: Minimize unsafe code. Every unsafe block must be reviewed by 2+ engineers.

Audit existing unsafe blocks:

# Find all unsafe blocks in codebase
cd /home/jgrusewski/Work/foxhunt
grep -r "unsafe" --include="*.rs" . | grep -v "// SAFETY:"

# All unsafe blocks must have SAFETY comment explaining invariants

Example of well-documented unsafe:

// SAFETY: This is safe because:
// 1. The pointer is guaranteed to be valid by the caller
// 2. The lifetime is bounded by the scope
// 3. No concurrent access is possible due to mutex
unsafe {
    // Unsafe code here
}

---

Data Security

Encryption at Rest

PostgreSQL Encryption

Option 1: Transparent Data Encryption (pgcrypto):

-- Encrypt sensitive columns
CREATE EXTENSION IF NOT EXISTS pgcrypto;

-- Encrypt TOTP secret
CREATE TABLE mfa_config (
    id UUID PRIMARY KEY,
    user_id UUID REFERENCES users(id),
    totp_secret_encrypted BYTEA,  -- Encrypted with pgp_sym_encrypt
    -- ...
);

-- Insert encrypted data
INSERT INTO mfa_config (user_id, totp_secret_encrypted)
VALUES ($1, pgp_sym_encrypt($2, $3));  -- $3 is encryption key

-- Query encrypted data
SELECT user_id, pgp_sym_decrypt(totp_secret_encrypted, $1) AS totp_secret
FROM mfa_config WHERE user_id = $2;

Option 2: Full Disk Encryption (LUKS):

# Encrypt PostgreSQL data directory
sudo cryptsetup luksFormat /dev/sdb  # PostgreSQL data disk
sudo cryptsetup luksOpen /dev/sdb pgdata
sudo mkfs.ext4 /dev/mapper/pgdata
sudo mount /dev/mapper/pgdata /var/lib/postgresql

# Add to /etc/crypttab for auto-mount
echo "pgdata /dev/sdb none luks" | sudo tee -a /etc/crypttab

Redis Encryption

Option 1: Disk Encryption (LUKS):

# Encrypt Redis RDB/AOF files
sudo cryptsetup luksFormat /dev/sdc  # Redis disk
sudo cryptsetup luksOpen /dev/sdc redisdata
sudo mkfs.ext4 /dev/mapper/redisdata
sudo mount /dev/mapper/redisdata /var/lib/redis

Option 2: Application-Level Encryption:

// Encrypt JWT tokens before storing in Redis
let encrypted_token = encrypt_aes_gcm(&jwt_token, &encryption_key);
redis_conn.set::<_, _, ()>(&key, encrypted_token).await?;

S3 Encryption

Server-Side Encryption (SSE-S3):

# Enable default encryption on S3 bucket
aws s3api put-bucket-encryption \
    --bucket foxhunt-models \
    --server-side-encryption-configuration '{
        "Rules": [{
            "ApplyServerSideEncryptionByDefault": {
                "SSEAlgorithm": "AES256"
            }
        }]
    }'

Client-Side Encryption (CSE):

// ml/src/model_loader.rs
use aws_sdk_s3::primitives::ByteStream;
use aes_gcm::{Aes256Gcm, KeyInit};

pub async fn download_encrypted_model(
    s3_path: &str,
    encryption_key: &[u8; 32]
) -> Result<Vec<u8>> {
    // Download encrypted model from S3
    let encrypted_data = download_from_s3(s3_path).await?;

    // Decrypt locally
    let cipher = Aes256Gcm::new_from_slice(encryption_key)?;
    let decrypted = cipher.decrypt(&nonce, encrypted_data.as_ref())?;

    Ok(decrypted)
}

Database Access Control

Principle of Least Privilege:

-- Create read-only user for reporting
CREATE USER foxhunt_readonly WITH PASSWORD 'VAULT_PASSWORD';
GRANT CONNECT ON DATABASE foxhunt_production TO foxhunt_readonly;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO foxhunt_readonly;

-- Create service-specific users
CREATE USER trading_service_user WITH PASSWORD 'VAULT_PASSWORD';
GRANT SELECT, INSERT, UPDATE ON orders TO trading_service_user;
GRANT SELECT, INSERT, UPDATE ON positions TO trading_service_user;
-- NO DELETE permissions for safety

-- Verify permissions
SELECT grantee, privilege_type FROM information_schema.role_table_grants
WHERE table_name = 'orders';

---

Secrets Management

HashiCorp Vault Integration

Deploy Vault (Docker):

# docker-compose.vault.yml
version: '3.8'
services:
  vault:
    image: vault:1.15
    container_name: foxhunt-vault
    ports:
      - "8200:8200"
    environment:
      VAULT_DEV_ROOT_TOKEN_ID: "root"  # Change in production
      VAULT_DEV_LISTEN_ADDRESS: "0.0.0.0:8200"
    cap_add:
      - IPC_LOCK
    volumes:
      - ./vault-data:/vault/data
    command: server -dev

Store Secrets in Vault:

# Initialize Vault
export VAULT_ADDR='http://localhost:8200'
export VAULT_TOKEN='root'

# Store database password
vault kv put secret/foxhunt/postgres password="STRONG_PASSWORD"

# Store JWT secret
vault kv put secret/foxhunt/jwt secret="$(openssl rand -base64 64)"

# Store Redis password
vault kv put secret/foxhunt/redis password="REDIS_PASSWORD"

# Store S3 credentials
vault kv put secret/foxhunt/s3 \
    access_key="AWS_ACCESS_KEY" \
    secret_key="AWS_SECRET_KEY"

Retrieve Secrets in Application:

// config/src/vault.rs
use vaultrs::client::{VaultClient, VaultClientSettingsBuilder};

pub async fn get_database_password() -> Result<String> {
    let client = VaultClient::new(
        VaultClientSettingsBuilder::default()
            .address("http://localhost:8200")
            .token(std::env::var("VAULT_TOKEN")?)
            .build()?
    )?;

    let secret: HashMap<String, String> = client
        .kv2("secret")
        .read("foxhunt/postgres")
        .await?;

    Ok(secret.get("password").unwrap().to_string())
}

Secret Rotation Policy

Secret Type	Rotation Frequency	Process
JWT Signing Key	90 days	Automated via Vault
Database Passwords	90 days	Manual + service restart
Redis Password	90 days	Manual + service restart
TLS Certificates	13 months	Automated via cert-manager
S3 Access Keys	180 days	Manual via AWS IAM
MFA Recovery Codes	On use	Automated

Automated JWT Secret Rotation:

#!/bin/bash
# rotate_jwt_secret.sh

# Generate new JWT secret
NEW_SECRET=$(openssl rand -base64 64)

# Store in Vault
vault kv put secret/foxhunt/jwt secret="$NEW_SECRET"

# Restart API Gateway to pick up new secret
sudo systemctl restart api_gateway

# Verify
sleep 5
sudo systemctl status api_gateway

---

Authentication & Authorization

JWT Configuration

Secure JWT Settings:

// services/api_gateway/src/auth/jwt_service.rs
use jsonwebtoken::{Algorithm, EncodingKey, DecodingKey, Validation};

pub struct JwtService {
    encoding_key: EncodingKey,
    decoding_key: DecodingKey,  // Cached for performance
}

impl JwtService {
    pub fn new(secret: String, issuer: String, audience: String) -> Self {
        // Validate secret strength
        assert!(secret.len() >= 64, "JWT secret must be >= 64 bytes");

        Self {
            encoding_key: EncodingKey::from_secret(secret.as_bytes()),
            decoding_key: DecodingKey::from_secret(secret.as_bytes()),
        }
    }

    pub fn create_token(&self, user_id: Uuid) -> Result<String> {
        let claims = Claims {
            sub: user_id.to_string(),
            iss: "foxhunt-api-gateway".to_string(),
            aud: "foxhunt-services".to_string(),
            exp: Utc::now().timestamp() + 3600,  // 1 hour expiry
            iat: Utc::now().timestamp(),
            nbf: Utc::now().timestamp(),
            jti: Uuid::new_v4().to_string(),  // Unique token ID
        };

        let token = encode(
            &Header::new(Algorithm::HS512),  // Strong algorithm
            &claims,
            &self.encoding_key
        )?;

        Ok(token)
    }
}

JWT Revocation

Redis-Backed Revocation:

// services/api_gateway/src/auth/revocation_service.rs
use redis::AsyncCommands;

pub struct RevocationService {
    redis: redis::aio::MultiplexedConnection,
}

impl RevocationService {
    pub async fn revoke_token(&self, jti: &str, ttl: u64) -> Result<()> {
        let mut conn = self.redis.clone();
        let key = format!("revoked:{}", jti);

        // Store in Redis with TTL matching token expiry
        conn.set_ex::<_, _, ()>(&key, "revoked", ttl).await?;

        Ok(())
    }

    pub async fn is_revoked(&self, jti: &str) -> Result<bool> {
        let mut conn = self.redis.clone();
        let key = format!("revoked:{}", jti);

        // Check if token is in revocation list
        let exists: bool = conn.exists(&key).await?;

        Ok(exists)
    }
}

MFA/TOTP Implementation

RFC 6238 Compliance:

// services/trading_service/src/mfa/totp.rs
use totp_lite::{totp, totp_custom};

pub fn verify_totp_code(
    secret: &[u8],
    user_code: &str,
    time_step: u64
) -> bool {
    let valid_codes = vec![
        // Current time window
        totp_custom::<Sha1>(time_step, 30, secret, 6),
        // Previous time window (30s tolerance)
        totp_custom::<Sha1>(time_step - 30, 30, secret, 6),
        // Next time window (30s tolerance)
        totp_custom::<Sha1>(time_step + 30, 30, secret, 6),
    ];

    valid_codes.iter().any(|code| code == user_code)
}

Backup Codes:

-- Store backup codes (hashed)
CREATE TABLE mfa_backup_codes (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id UUID REFERENCES users(id) ON DELETE CASCADE,
    code_hash VARCHAR(128) NOT NULL,  -- bcrypt hash
    used_at TIMESTAMP,
    created_at TIMESTAMP DEFAULT NOW()
);

-- Generate 10 backup codes per user during MFA enrollment
-- Each code is 16 characters alphanumeric

RBAC Implementation

Permission Check (cached for performance):

// services/api_gateway/src/auth/authz_service.rs
use lru::LruCache;

pub struct AuthzService {
    permission_cache: Mutex<LruCache<(Uuid, String), bool>>,
}

impl AuthzService {
    pub async fn check_permission(
        &self,
        user_id: Uuid,
        endpoint: &str
    ) -> Result<bool> {
        // Check cache first (sub-100ns)
        let cache_key = (user_id, endpoint.to_string());
        if let Some(&allowed) = self.permission_cache.lock().await.get(&cache_key) {
            return Ok(allowed);
        }

        // Cache miss: Query database
        let allowed = sqlx::query_scalar!(
            r#"
            SELECT EXISTS(
                SELECT 1 FROM user_roles ur
                JOIN role_permissions rp ON ur.role_id = rp.role_id
                JOIN permissions p ON rp.permission_id = p.id
                WHERE ur.user_id = $1 AND p.endpoint = $2
            ) AS "allowed!"
            "#,
            user_id,
            endpoint
        )
        .fetch_one(&self.pool)
        .await?;

        // Update cache (TTL: 5 minutes)
        self.permission_cache.lock().await.put(cache_key, allowed);

        Ok(allowed)
    }
}

Permission Invalidation (on role changes):

-- Trigger to send NOTIFY when permissions change
CREATE OR REPLACE FUNCTION notify_permission_change() RETURNS TRIGGER AS $$
BEGIN
    PERFORM pg_notify('permission_change', NEW.user_id::text);
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER permission_change_trigger
AFTER INSERT OR UPDATE OR DELETE ON user_roles
FOR EACH ROW EXECUTE FUNCTION notify_permission_change();

---

Logging & Monitoring

Audit Trail Requirements

Immutable Audit Logs:

-- Prevent deletion of audit events
CREATE POLICY audit_events_no_delete ON audit_events
FOR DELETE USING (false);

-- Prevent updates of audit events
CREATE POLICY audit_events_no_update ON audit_events
FOR UPDATE USING (false);

-- Verify immutability
DELETE FROM audit_events WHERE id = 'some-id';
-- Expected: ERROR:  new row violates row-level security policy

Audit Event Structure:

// trading_engine/src/compliance/audit_trails.rs
#[derive(Debug, Serialize, Deserialize)]
pub struct AuditEvent {
    pub id: Uuid,
    pub event_type: String,        // login, order_submit, config_change, etc.
    pub user_id: Option<Uuid>,
    pub ip_address: Option<String>,
    pub timestamp: DateTime<Utc>,  // Microsecond precision
    pub resource: String,           // orders, users, config, etc.
    pub action: String,             // create, read, update, delete
    pub old_value: Option<serde_json::Value>,
    pub new_value: Option<serde_json::Value>,
    pub success: bool,
    pub error_message: Option<String>,
}

SIEM Integration

Export Audit Logs to ELK:

# Filebeat configuration
sudo tee /etc/filebeat/filebeat.yml <<EOF
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/foxhunt/*.log
    fields:
      environment: production
      application: foxhunt

output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  index: "foxhunt-audit-%{+yyyy.MM.dd}"

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
EOF

sudo systemctl restart filebeat

Kibana Dashboards:

• Failed login attempts (security)
• Order submission rate (business)
• API Gateway error rate (operational)
• Database query performance (performance)

Security Event Alerting

PagerDuty/Slack Integration:

# Prometheus AlertManager configuration
groups:
  - name: security
    interval: 30s
    rules:
      - alert: HighFailedLoginRate
        expr: rate(failed_logins[5m]) > 10
        annotations:
          summary: "High failed login rate detected"
          description: "{{ $value }} failed logins/min in last 5 minutes"

      - alert: UnauthorizedAccessAttempt
        expr: unauthorized_access_attempts > 0
        annotations:
          summary: "Unauthorized access attempt"
          description: "User {{ $labels.user_id }} attempted to access {{ $labels.endpoint }}"

      - alert: JWTRevocationServiceDown
        expr: up{job="redis"} == 0
        for: 1m
        annotations:
          summary: "JWT revocation service (Redis) is down"
          description: "All authentication will fail"

---

Incident Response

Security Incident Classification

Severity	Definition	Response Time	Example
P0 (Critical)	Active breach, data exfiltration	Immediate	Database dump stolen
P1 (High)	Potential breach, exploit detected	15 minutes	SQL injection attempt
P2 (Medium)	Suspicious activity, failed attacks	1 hour	Brute force login attempts
P3 (Low)	Policy violation, minor vulnerability	24 hours	Weak password detected

Incident Response Playbook

P0: Active Breach:

1. Contain (0-5 minutes):

   # Emergency shutdown
   sudo systemctl stop api_gateway trading_service backtesting_service ml_training_service
   
   # Block all network traffic
   sudo iptables -P INPUT DROP
   sudo iptables -P OUTPUT DROP
   

2. Assess (5-15 minutes):

   # Check recent logins
   psql $DATABASE_URL -c "SELECT * FROM audit_events WHERE event_type='login' AND created_at > NOW() - INTERVAL '1 hour' ORDER BY created_at DESC;"
   
   # Check active database connections
   psql $DATABASE_URL -c "SELECT * FROM pg_stat_activity;"
   

3. Eradicate (15-60 minutes):

   • Rotate all credentials (database, Redis, JWT secret)
   • Revoke all JWT tokens
   • Patch exploited vulnerability

4. Recover (1-4 hours):

   • Restore from clean backup if needed
   • Gradually restore services
   • Monitor for re-compromise

5. Post-Mortem (within 48 hours):

   • Document timeline
   • Identify root cause
   • Implement preventive measures

P1: SQL Injection Attempt:

# Check PostgreSQL logs for suspicious queries
sudo grep -i "ERROR" /var/log/postgresql/postgresql-*.log | tail -100

# Block attacking IP
sudo iptables -A INPUT -s ATTACKER_IP -j DROP

# Review audit trail
psql $DATABASE_URL -c "SELECT * FROM audit_events WHERE success = false AND created_at > NOW() - INTERVAL '1 hour';"

---

Security Checklist

Pre-Production Security Review

• Network Security

  • Firewall rules configured and tested
  • VPC/subnet segmentation implemented
  • mTLS enforced for all inter-service communication
  • TLS 1.3 with strong ciphers configured
  • DDoS protection enabled

• Host Security

  • CIS benchmarks applied
  • Automatic security updates enabled
  • SSH hardened (no root login, key-based auth only)
  • File permissions verified
  • Intrusion detection (Fail2Ban) configured

• Application Security

  • Input validation on all API endpoints
  • SQL injection prevention verified (parameterized queries only)
  • Dependency vulnerabilities scanned (cargo audit passing)
  • unsafe code reviewed
  • Rate limiting enforced (100 req/s per user)

• Data Security

  • Encryption at rest enabled (PostgreSQL, Redis, S3)
  • Encryption in transit enforced (TLS 1.3)
  • Database access control implemented (least privilege)
  • Sensitive data encrypted (TOTP secrets, MFA backup codes)

• Secrets Management

  • Vault deployed and accessible
  • All secrets stored in Vault (no hardcoded secrets)
  • Secret rotation policies defined
  • JWT secret >= 64 bytes of entropy

• Authentication & Authorization

  • JWT validation with strong algorithm (HS512)
  • JWT revocation working (Redis connectivity verified)
  • MFA/TOTP implemented and tested
  • RBAC permissions configured
  • Audit logging enabled for all auth events

• Logging & Monitoring

  • Audit trails immutable (DELETE/UPDATE blocked)
  • SIEM integration configured (ELK/Splunk)
  • Security alerts configured (PagerDuty/Slack)
  • Log retention policy enforced (7 years for audit trails)

• Compliance

  • SOX requirements documented and verified
  • MiFID II timestamping accurate (±100μs)
  • Data retention policies implemented
  • Incident response plan reviewed

Ongoing Security Operations

Daily:

• Review failed login attempts
• Check security alerts (PagerDuty/Slack)
• Verify all services healthy

Weekly:

• Review audit trail for anomalies
• Scan dependencies for vulnerabilities (cargo audit)
• Review firewall logs for blocked IPs

Monthly:

• Rotate JWT secret
• Review and update RBAC permissions
• Test disaster recovery procedures
• Security training for team

Quarterly:

• Penetration testing
• Compliance audit (SOX/MiFID II)
• Security policy review and update
• Certificate renewal (if needed)

---

End of Security Hardening Guide

This document must be reviewed quarterly and updated after every security incident.