foxhunt/docs/SECURITY_HARDENING.md

# Foxhunt HFT Trading System - Security Hardening Guide

**Version**: 1.0
**Last Updated**: 2025-10-03
**Wave**: 71 - Production Security
**Audience**: Security Engineers, DevOps, System Administrators

---

## Table of Contents

1. [Security Architecture Overview](#security-architecture-overview)
2. [Compliance Mandates](#compliance-mandates)
3. [Network Security](#network-security)
4. [Host Security](#host-security)
5. [Application Security](#application-security)
6. [Data Security](#data-security)
7. [Secrets Management](#secrets-management)
8. [Authentication & Authorization](#authentication--authorization)
9. [Logging & Monitoring](#logging--monitoring)
10. [Incident Response](#incident-response)
11. [Security Checklist](#security-checklist)

---

## Security Architecture Overview

### Defense-in-Depth Strategy

The Foxhunt HFT system implements a **6-layer security architecture**:

```
┌─────────────────────────────────────────────────────────────┐
│               FOXHUNT SECURITY ARCHITECTURE                 │
└─────────────────────────────────────────────────────────────┘

Layer 1: NETWORK PERIMETER
├─ Firewall/WAF (external)
├─ DDoS protection
├─ IP allowlisting
└─ TLS 1.3 termination

Layer 2: API GATEWAY (6-Layer Auth)
├─ JWT validation (HS512)
├─ JWT revocation check (Redis)
├─ MFA/TOTP (RFC 6238)
├─ RBAC authorization
├─ Rate limiting (100 req/s)
└─ Audit logging

Layer 3: MUTUAL TLS (Inter-Service)
├─ X.509 client certificates
├─ CA-signed service certificates
└─ Certificate pinning

Layer 4: APPLICATION SECURITY
├─ Input validation
├─ SQL injection prevention (parameterized queries)
├─ Dependency vulnerability scanning
└─ Secure coding practices (Rust memory safety)

Layer 5: DATA SECURITY
├─ Encryption at rest (PostgreSQL, Redis, S3)
├─ Encryption in transit (TLS 1.3)
└─ Database access control (RBAC)

Layer 6: AUDIT & MONITORING
├─ Immutable audit trails (SOX/MiFID II)
├─ SIEM integration
├─ Security event alerting
└─ Compliance reporting
```

### Security Principles

1. **Least Privilege**: Every user, service, and process has minimum required permissions
2. **Zero Trust**: No implicit trust based on network location
3. **Defense-in-Depth**: Multiple overlapping security layers
4. **Fail Secure**: System fails to a secure state during errors
5. **Audit Everything**: Comprehensive logging for compliance and forensics

---

## Compliance Mandates

### SOX (Sarbanes-Oxley Act)

**Requirements**:
- Audit trails for all financial transactions
- Access controls for financial systems
- Data integrity and non-repudiation
- Separation of duties

**Implementation**:
- ✅ Immutable audit trails in `audit_events` table
- ✅ RBAC with role-permission separation
- ✅ Database-level constraints for data integrity
- ✅ Multi-signature approval workflows (future enhancement)

**Verification**:
```sql
-- Verify audit trail immutability
SELECT COUNT(*) FROM audit_events;  -- Should never decrease

-- Verify role separation
SELECT u.username, r.name FROM users u
JOIN user_roles ur ON u.id = ur.user_id
JOIN roles r ON ur.role_id = r.id
WHERE r.name IN ('admin', 'trader', 'auditor');
```

### MiFID II (Markets in Financial Instruments Directive)

**Requirements**:
- Microsecond-precision timestamping
- Trade reconstruction capability
- Best execution reporting
- Client order handling transparency

**Implementation**:
- ✅ NTP/PTP time synchronization (±100μs)
- ✅ Comprehensive order lifecycle logging
- ✅ Execution venue tracking
- ✅ Order routing audit trails

**Verification**:
```bash
# Verify time synchronization
chronyc tracking
# Expected: System time offset < 100μs

# Check order timestamps
psql $DATABASE_URL -c "SELECT order_id, created_at FROM orders LIMIT 5;"
```

### Data Retention Policies

| Data Type | Retention Period | Storage Location |
|-----------|------------------|------------------|
| Audit Trails | 7 years | PostgreSQL + S3 |
| Trade Records | 7 years | PostgreSQL + S3 |
| User Activity Logs | 1 year | ELK/Loki |
| System Logs | 90 days | ELK/Loki |
| MFA Backup Codes | Until used | PostgreSQL (encrypted) |

**Implementation**:
```sql
-- Automated data archival (run monthly)
CREATE OR REPLACE FUNCTION archive_old_audit_events() RETURNS void AS $$
BEGIN
    -- Archive audit events older than 7 years to S3
    COPY (SELECT * FROM audit_events WHERE created_at < NOW() - INTERVAL '7 years')
    TO PROGRAM 'aws s3 cp - s3://foxhunt-archives/audit_events_$(date +%Y%m).csv';

    -- DO NOT DELETE (compliance requirement)
END;
$$ LANGUAGE plpgsql;
```

---

## Network Security

### Firewall Rules

**Inbound Rules** (iptables):

```bash
#!/bin/bash
# firewall_rules.sh - Production firewall configuration

# Flush existing rules
iptables -F
iptables -X

# Default policies: DENY all
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT

# Allow loopback
iptables -A INPUT -i lo -j ACCEPT

# Allow established connections
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allow SSH (from management IPs only)
iptables -A INPUT -p tcp --dport 22 -s 10.0.0.0/8 -j ACCEPT

# Allow API Gateway (from trusted sources only)
iptables -A INPUT -p tcp --dport 50051 -s 203.0.113.0/24 -j ACCEPT  # Exchange IPs
iptables -A INPUT -p tcp --dport 8080 -s 10.0.0.0/8 -j ACCEPT       # Internal monitoring

# Allow PostgreSQL (from app servers only)
iptables -A INPUT -p tcp --dport 5432 -s 10.0.1.0/24 -j ACCEPT

# Allow Redis (from app servers only)
iptables -A INPUT -p tcp --dport 6379 -s 10.0.1.0/24 -j ACCEPT

# Rate limiting (anti-DDoS)
iptables -A INPUT -p tcp --dport 50051 -m hashlimit \
    --hashlimit-name api_gateway \
    --hashlimit-above 1000/sec \
    --hashlimit-mode srcip \
    --hashlimit-burst 2000 \
    -j DROP

# Log dropped packets
iptables -A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables-dropped: "

# Save rules
iptables-save > /etc/iptables/rules.v4
```

**Apply firewall rules**:
```bash
sudo bash firewall_rules.sh
```

### VPC/Subnet Segmentation

**Recommended AWS VPC Architecture**:

```
VPC: 10.0.0.0/16
├─ Public Subnet (DMZ): 10.0.0.0/24
│  ├─ NAT Gateway
│  └─ Bastion Host (SSH jump box)
│
├─ Private Subnet (Application): 10.0.1.0/24
│  ├─ API Gateway
│  ├─ Trading Service
│  ├─ Backtesting Service
│  └─ ML Training Service
│
└─ Private Subnet (Data): 10.0.2.0/24
   ├─ PostgreSQL (primary + replica)
   ├─ Redis (cluster)
   └─ S3 VPC Endpoint
```

**Security Groups**:

```yaml
# API Gateway SG
Inbound:
  - Port 50051 (gRPC): 0.0.0.0/0 (internet)
  - Port 8080 (health): 10.0.0.0/16 (internal monitoring)
Outbound:
  - Port 5432: 10.0.2.0/24 (PostgreSQL)
  - Port 6379: 10.0.2.0/24 (Redis)

# Backend Services SG
Inbound:
  - Port 50052-50054 (gRPC): 10.0.1.0/24 (API Gateway only)
Outbound:
  - Port 5432: 10.0.2.0/24 (PostgreSQL)
  - Port 6379: 10.0.2.0/24 (Redis)
  - Port 443: 0.0.0.0/0 (S3 HTTPS)

# Database SG
Inbound:
  - Port 5432: 10.0.1.0/24 (backend services)
  - Port 6379: 10.0.1.0/24 (backend services)
Outbound:
  - DENY all (databases should not initiate outbound)
```

### Mutual TLS Enforcement

**Verify mTLS is enforced**:

```bash
# Test without client certificate (should fail)
curl https://trading-service:50052/health
# Expected: certificate required

# Test with valid client certificate (should succeed)
curl --cert /etc/foxhunt/certs/client.crt \
     --key /etc/foxhunt/certs/client.key \
     --cacert /etc/foxhunt/certs/ca.crt \
     https://trading-service:50052/health
# Expected: {"status":"ok"}
```

**Tonic gRPC mTLS Configuration**:

```rust
// services/trading_service/src/tls_config.rs
use tonic::transport::{Certificate, Identity, ServerTlsConfig};

pub fn create_tls_config() -> ServerTlsConfig {
    let cert = std::fs::read("/etc/foxhunt/trading_service/trading_service.crt").unwrap();
    let key = std::fs::read("/etc/foxhunt/trading_service/trading_service.key").unwrap();
    let ca_cert = std::fs::read("/etc/foxhunt/certs/ca.crt").unwrap();

    let identity = Identity::from_pem(cert, key);
    let client_ca = Certificate::from_pem(ca_cert);

    ServerTlsConfig::new()
        .identity(identity)
        .client_ca_root(client_ca)  // Enforce client certificate validation
}
```

### TLS 1.3 Configuration

**Strong Cipher Suites** (in order of preference):

```rust
// Recommended ciphers for Rust/Tonic
const STRONG_CIPHERS: &[&str] = &[
    "TLS_AES_256_GCM_SHA384",
    "TLS_AES_128_GCM_SHA256",
    "TLS_CHACHA20_POLY1305_SHA256",
];

// Disable weak ciphers
const FORBIDDEN_CIPHERS: &[&str] = &[
    "TLS_RSA_*",           // No forward secrecy
    "TLS_*_CBC_*",         // Vulnerable to BEAST/POODLE
    "TLS_*_3DES_*",        // Weak encryption
];
```

**Nginx Proxy (if used)**:

```nginx
server {
    listen 443 ssl http2;

    # TLS 1.3 only
    ssl_protocols TLSv1.3;

    # Strong ciphers
    ssl_ciphers 'TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256:TLS_CHACHA20_POLY1305_SHA256';
    ssl_prefer_server_ciphers on;

    # Certificate pinning (HPKP)
    add_header Public-Key-Pins 'pin-sha256="BASE64_HASH"; max-age=2592000';

    # Security headers
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
    add_header X-Frame-Options "DENY" always;
    add_header X-Content-Type-Options "nosniff" always;

    # Proxy to API Gateway
    location / {
        grpc_pass grpc://localhost:50051;
    }
}
```

### DDoS Protection

**Rate Limiting** (already implemented in API Gateway):

```rust
// services/api_gateway/src/auth/rate_limiter.rs
pub struct RateLimiter {
    limit: u32,  // 100 req/s per user
    // Token bucket algorithm
}
```

**CloudFlare/AWS Shield** (if using cloud):
- Enable CloudFlare Pro with DDoS protection
- Enable AWS Shield Standard (free) or Advanced (paid)
- Configure rate limiting rules at edge

---

## Host Security

### OS Hardening (CIS Benchmarks)

**Apply CIS Ubuntu 22.04 Benchmark**:

```bash
#!/bin/bash
# cis_hardening.sh - CIS Level 1 hardening

# 1. Disable unnecessary services
sudo systemctl disable bluetooth.service
sudo systemctl disable cups.service
sudo systemctl disable avahi-daemon.service

# 2. Enable firewall
sudo ufw enable
sudo ufw default deny incoming
sudo ufw default allow outgoing

# 3. Secure SSH
sudo sed -i 's/#PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config
sudo sed -i 's/#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
sudo sed -i 's/#PubkeyAuthentication yes/PubkeyAuthentication yes/' /etc/ssh/sshd_config
sudo systemctl restart sshd

# 4. Enable automatic security updates
sudo apt install -y unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades

# 5. Set file permissions
sudo chmod 600 /etc/shadow
sudo chmod 600 /etc/gshadow
sudo chmod 644 /etc/passwd
sudo chmod 644 /etc/group

# 6. Disable core dumps
echo "* hard core 0" | sudo tee -a /etc/security/limits.conf

# 7. Enable ASLR (address space layout randomization)
sudo sysctl -w kernel.randomize_va_space=2

# 8. Disable IP forwarding (unless needed)
sudo sysctl -w net.ipv4.ip_forward=0
```

### File System Permissions

**Critical Files**:

```bash
# Binaries
sudo chown root:root /usr/local/bin/{api_gateway,trading_service,backtesting_service,ml_training_service}
sudo chmod 755 /usr/local/bin/{api_gateway,trading_service,backtesting_service,ml_training_service}

# Configuration files
sudo chown foxhunt:foxhunt /etc/foxhunt/*/.env
sudo chmod 600 /etc/foxhunt/*/.env

# TLS private keys
sudo chown foxhunt:foxhunt /etc/foxhunt/certs/*.key
sudo chmod 400 /etc/foxhunt/certs/*.key

# TLS certificates
sudo chown foxhunt:foxhunt /etc/foxhunt/certs/*.crt
sudo chmod 644 /etc/foxhunt/certs/*.crt

# Logs
sudo chown foxhunt:foxhunt /var/log/foxhunt/
sudo chmod 750 /var/log/foxhunt/
```

### Regular Patching

**Automated Security Updates**:

```bash
# Configure unattended-upgrades
sudo tee /etc/apt/apt.conf.d/50unattended-upgrades <<EOF
Unattended-Upgrade::Allowed-Origins {
    "\${distro_id}:\${distro_codename}-security";
    "\${distro_id}ESMApps:\${distro_codename}-apps-security";
};
Unattended-Upgrade::AutoFixInterruptedDpkg "true";
Unattended-Upgrade::MinimalSteps "true";
Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-Time "03:00";
EOF

# Enable automatic updates
sudo systemctl enable unattended-upgrades
sudo systemctl start unattended-upgrades
```

**Manual Patching Schedule**:
- **Critical patches**: Within 24 hours
- **Security patches**: Within 7 days
- **Regular updates**: Monthly (during maintenance window)

### Intrusion Detection (OSSEC/Fail2Ban)

**Install Fail2Ban**:

```bash
sudo apt install -y fail2ban

# Configure jail for SSH
sudo tee /etc/fail2ban/jail.local <<EOF
[sshd]
enabled = true
port = 22
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
bantime = 3600
findtime = 600

[api-gateway]
enabled = true
port = 50051
filter = api-gateway
logpath = /var/log/foxhunt/api_gateway.log
maxretry = 10
bantime = 600
EOF

# Create custom filter
sudo tee /etc/fail2ban/filter.d/api-gateway.conf <<EOF
[Definition]
failregex = WARN.*Authentication failed for user <HOST>
ignoreregex =
EOF

sudo systemctl restart fail2ban
```

---

## Application Security

### Input Validation

**API Gateway Input Validation**:

```rust
// services/api_gateway/src/validation.rs
use validator::{Validate, ValidationError};

#[derive(Debug, Validate)]
pub struct OrderRequest {
    #[validate(length(min = 1, max = 10))]
    pub symbol: String,

    #[validate(range(min = 1, max = 1000000))]
    pub quantity: u64,

    #[validate(custom = "validate_order_side")]
    pub side: String,
}

fn validate_order_side(side: &str) -> Result<(), ValidationError> {
    if !["buy", "sell"].contains(&side) {
        return Err(ValidationError::new("invalid_order_side"));
    }
    Ok(())
}
```

### SQL Injection Prevention

**✅ CORRECT (Parameterized Queries)**:

```rust
// trading_engine/src/repositories/trading_repository.rs
pub async fn get_order_by_id(&self, order_id: Uuid) -> Result<Option<Order>> {
    let order = sqlx::query_as!(
        Order,
        "SELECT * FROM orders WHERE id = $1",
        order_id  // Parameterized query
    )
    .fetch_optional(&self.pool)
    .await?;
    Ok(order)
}
```

**❌ INCORRECT (String Concatenation - NEVER DO THIS)**:

```rust
// VULNERABLE TO SQL INJECTION - DO NOT USE
let query = format!("SELECT * FROM orders WHERE id = '{}'", user_input);
sqlx::query(&query).fetch_all(&pool).await?;
```

**Verification**:
```bash
# Scan codebase for SQL injection vulnerabilities
cd /home/jgrusewski/Work/foxhunt
grep -r "format!.*SELECT" --include="*.rs" .
# Expected: No results (all queries should use $1, $2 placeholders)
```

### Dependency Vulnerability Scanning

**Automated Scanning (CI/CD)**:

```bash
# Install cargo-audit
cargo install cargo-audit

# Scan for known vulnerabilities
cargo audit

# Expected output: No vulnerabilities found
```

**Add to CI/CD pipeline** (.github/workflows/security.yml):

```yaml
name: Security Scan
on: [push, pull_request]
jobs:
  audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions-rs/toolchain@v1
        with:
          toolchain: stable
      - run: cargo install cargo-audit
      - run: cargo audit --deny warnings
```

### Rust `unsafe` Code Review

**Policy**: Minimize `unsafe` code. Every `unsafe` block must be reviewed by 2+ engineers.

**Audit existing `unsafe` blocks**:

```bash
# Find all unsafe blocks in codebase
cd /home/jgrusewski/Work/foxhunt
grep -r "unsafe" --include="*.rs" . | grep -v "// SAFETY:"

# All unsafe blocks must have SAFETY comment explaining invariants
```

**Example of well-documented `unsafe`**:

```rust
// SAFETY: This is safe because:
// 1. The pointer is guaranteed to be valid by the caller
// 2. The lifetime is bounded by the scope
// 3. No concurrent access is possible due to mutex
unsafe {
    // Unsafe code here
}
```

---

## Data Security

### Encryption at Rest

#### PostgreSQL Encryption

**Option 1: Transparent Data Encryption (pgcrypto)**:

```sql
-- Encrypt sensitive columns
CREATE EXTENSION IF NOT EXISTS pgcrypto;

-- Encrypt TOTP secret
CREATE TABLE mfa_config (
    id UUID PRIMARY KEY,
    user_id UUID REFERENCES users(id),
    totp_secret_encrypted BYTEA,  -- Encrypted with pgp_sym_encrypt
    -- ...
);

-- Insert encrypted data
INSERT INTO mfa_config (user_id, totp_secret_encrypted)
VALUES ($1, pgp_sym_encrypt($2, $3));  -- $3 is encryption key

-- Query encrypted data
SELECT user_id, pgp_sym_decrypt(totp_secret_encrypted, $1) AS totp_secret
FROM mfa_config WHERE user_id = $2;
```

**Option 2: Full Disk Encryption (LUKS)**:

```bash
# Encrypt PostgreSQL data directory
sudo cryptsetup luksFormat /dev/sdb  # PostgreSQL data disk
sudo cryptsetup luksOpen /dev/sdb pgdata
sudo mkfs.ext4 /dev/mapper/pgdata
sudo mount /dev/mapper/pgdata /var/lib/postgresql

# Add to /etc/crypttab for auto-mount
echo "pgdata /dev/sdb none luks" | sudo tee -a /etc/crypttab
```

#### Redis Encryption

**Option 1: Disk Encryption (LUKS)**:

```bash
# Encrypt Redis RDB/AOF files
sudo cryptsetup luksFormat /dev/sdc  # Redis disk
sudo cryptsetup luksOpen /dev/sdc redisdata
sudo mkfs.ext4 /dev/mapper/redisdata
sudo mount /dev/mapper/redisdata /var/lib/redis
```

**Option 2: Application-Level Encryption**:

```rust
// Encrypt JWT tokens before storing in Redis
let encrypted_token = encrypt_aes_gcm(&jwt_token, &encryption_key);
redis_conn.set::<_, _, ()>(&key, encrypted_token).await?;
```

#### S3 Encryption

**Server-Side Encryption (SSE-S3)**:

```bash
# Enable default encryption on S3 bucket
aws s3api put-bucket-encryption \
    --bucket foxhunt-models \
    --server-side-encryption-configuration '{
        "Rules": [{
            "ApplyServerSideEncryptionByDefault": {
                "SSEAlgorithm": "AES256"
            }
        }]
    }'
```

**Client-Side Encryption (CSE)**:

```rust
// ml/src/model_loader.rs
use aws_sdk_s3::primitives::ByteStream;
use aes_gcm::{Aes256Gcm, KeyInit};

pub async fn download_encrypted_model(
    s3_path: &str,
    encryption_key: &[u8; 32]
) -> Result<Vec<u8>> {
    // Download encrypted model from S3
    let encrypted_data = download_from_s3(s3_path).await?;

    // Decrypt locally
    let cipher = Aes256Gcm::new_from_slice(encryption_key)?;
    let decrypted = cipher.decrypt(&nonce, encrypted_data.as_ref())?;

    Ok(decrypted)
}
```

### Database Access Control

**Principle of Least Privilege**:

```sql
-- Create read-only user for reporting
CREATE USER foxhunt_readonly WITH PASSWORD 'VAULT_PASSWORD';
GRANT CONNECT ON DATABASE foxhunt_production TO foxhunt_readonly;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO foxhunt_readonly;

-- Create service-specific users
CREATE USER trading_service_user WITH PASSWORD 'VAULT_PASSWORD';
GRANT SELECT, INSERT, UPDATE ON orders TO trading_service_user;
GRANT SELECT, INSERT, UPDATE ON positions TO trading_service_user;
-- NO DELETE permissions for safety

-- Verify permissions
SELECT grantee, privilege_type FROM information_schema.role_table_grants
WHERE table_name = 'orders';
```

---

## Secrets Management

### HashiCorp Vault Integration

**Deploy Vault** (Docker):

```yaml
# docker-compose.vault.yml
version: '3.8'
services:
  vault:
    image: vault:1.15
    container_name: foxhunt-vault
    ports:
      - "8200:8200"
    environment:
      VAULT_DEV_ROOT_TOKEN_ID: "root"  # Change in production
      VAULT_DEV_LISTEN_ADDRESS: "0.0.0.0:8200"
    cap_add:
      - IPC_LOCK
    volumes:
      - ./vault-data:/vault/data
    command: server -dev
```

**Store Secrets in Vault**:

```bash
# Initialize Vault
export VAULT_ADDR='http://localhost:8200'
export VAULT_TOKEN='root'

# Store database password
vault kv put secret/foxhunt/postgres password="STRONG_PASSWORD"

# Store JWT secret
vault kv put secret/foxhunt/jwt secret="$(openssl rand -base64 64)"

# Store Redis password
vault kv put secret/foxhunt/redis password="REDIS_PASSWORD"

# Store S3 credentials
vault kv put secret/foxhunt/s3 \
    access_key="AWS_ACCESS_KEY" \
    secret_key="AWS_SECRET_KEY"
```

**Retrieve Secrets in Application**:

```rust
// config/src/vault.rs
use vaultrs::client::{VaultClient, VaultClientSettingsBuilder};

pub async fn get_database_password() -> Result<String> {
    let client = VaultClient::new(
        VaultClientSettingsBuilder::default()
            .address("http://localhost:8200")
            .token(std::env::var("VAULT_TOKEN")?)
            .build()?
    )?;

    let secret: HashMap<String, String> = client
        .kv2("secret")
        .read("foxhunt/postgres")
        .await?;

    Ok(secret.get("password").unwrap().to_string())
}
```

### Secret Rotation Policy

| Secret Type | Rotation Frequency | Process |
|-------------|-------------------|---------|
| JWT Signing Key | 90 days | Automated via Vault |
| Database Passwords | 90 days | Manual + service restart |
| Redis Password | 90 days | Manual + service restart |
| TLS Certificates | 13 months | Automated via cert-manager |
| S3 Access Keys | 180 days | Manual via AWS IAM |
| MFA Recovery Codes | On use | Automated |

**Automated JWT Secret Rotation**:

```bash
#!/bin/bash
# rotate_jwt_secret.sh

# Generate new JWT secret
NEW_SECRET=$(openssl rand -base64 64)

# Store in Vault
vault kv put secret/foxhunt/jwt secret="$NEW_SECRET"

# Restart API Gateway to pick up new secret
sudo systemctl restart api_gateway

# Verify
sleep 5
sudo systemctl status api_gateway
```

---

## Authentication & Authorization

### JWT Configuration

**Secure JWT Settings**:

```rust
// services/api_gateway/src/auth/jwt_service.rs
use jsonwebtoken::{Algorithm, EncodingKey, DecodingKey, Validation};

pub struct JwtService {
    encoding_key: EncodingKey,
    decoding_key: DecodingKey,  // Cached for performance
}

impl JwtService {
    pub fn new(secret: String, issuer: String, audience: String) -> Self {
        // Validate secret strength
        assert!(secret.len() >= 64, "JWT secret must be >= 64 bytes");

        Self {
            encoding_key: EncodingKey::from_secret(secret.as_bytes()),
            decoding_key: DecodingKey::from_secret(secret.as_bytes()),
        }
    }

    pub fn create_token(&self, user_id: Uuid) -> Result<String> {
        let claims = Claims {
            sub: user_id.to_string(),
            iss: "foxhunt-api-gateway".to_string(),
            aud: "foxhunt-services".to_string(),
            exp: Utc::now().timestamp() + 3600,  // 1 hour expiry
            iat: Utc::now().timestamp(),
            nbf: Utc::now().timestamp(),
            jti: Uuid::new_v4().to_string(),  // Unique token ID
        };

        let token = encode(
            &Header::new(Algorithm::HS512),  // Strong algorithm
            &claims,
            &self.encoding_key
        )?;

        Ok(token)
    }
}
```

### JWT Revocation

**Redis-Backed Revocation**:

```rust
// services/api_gateway/src/auth/revocation_service.rs
use redis::AsyncCommands;

pub struct RevocationService {
    redis: redis::aio::MultiplexedConnection,
}

impl RevocationService {
    pub async fn revoke_token(&self, jti: &str, ttl: u64) -> Result<()> {
        let mut conn = self.redis.clone();
        let key = format!("revoked:{}", jti);

        // Store in Redis with TTL matching token expiry
        conn.set_ex::<_, _, ()>(&key, "revoked", ttl).await?;

        Ok(())
    }

    pub async fn is_revoked(&self, jti: &str) -> Result<bool> {
        let mut conn = self.redis.clone();
        let key = format!("revoked:{}", jti);

        // Check if token is in revocation list
        let exists: bool = conn.exists(&key).await?;

        Ok(exists)
    }
}
```

### MFA/TOTP Implementation

**RFC 6238 Compliance**:

```rust
// services/trading_service/src/mfa/totp.rs
use totp_lite::{totp, totp_custom};

pub fn verify_totp_code(
    secret: &[u8],
    user_code: &str,
    time_step: u64
) -> bool {
    let valid_codes = vec![
        // Current time window
        totp_custom::<Sha1>(time_step, 30, secret, 6),
        // Previous time window (30s tolerance)
        totp_custom::<Sha1>(time_step - 30, 30, secret, 6),
        // Next time window (30s tolerance)
        totp_custom::<Sha1>(time_step + 30, 30, secret, 6),
    ];

    valid_codes.iter().any(|code| code == user_code)
}
```

**Backup Codes**:

```sql
-- Store backup codes (hashed)
CREATE TABLE mfa_backup_codes (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id UUID REFERENCES users(id) ON DELETE CASCADE,
    code_hash VARCHAR(128) NOT NULL,  -- bcrypt hash
    used_at TIMESTAMP,
    created_at TIMESTAMP DEFAULT NOW()
);

-- Generate 10 backup codes per user during MFA enrollment
-- Each code is 16 characters alphanumeric
```

### RBAC Implementation

**Permission Check** (cached for performance):

```rust
// services/api_gateway/src/auth/authz_service.rs
use lru::LruCache;

pub struct AuthzService {
    permission_cache: Mutex<LruCache<(Uuid, String), bool>>,
}

impl AuthzService {
    pub async fn check_permission(
        &self,
        user_id: Uuid,
        endpoint: &str
    ) -> Result<bool> {
        // Check cache first (sub-100ns)
        let cache_key = (user_id, endpoint.to_string());
        if let Some(&allowed) = self.permission_cache.lock().await.get(&cache_key) {
            return Ok(allowed);
        }

        // Cache miss: Query database
        let allowed = sqlx::query_scalar!(
            r#"
            SELECT EXISTS(
                SELECT 1 FROM user_roles ur
                JOIN role_permissions rp ON ur.role_id = rp.role_id
                JOIN permissions p ON rp.permission_id = p.id
                WHERE ur.user_id = $1 AND p.endpoint = $2
            ) AS "allowed!"
            "#,
            user_id,
            endpoint
        )
        .fetch_one(&self.pool)
        .await?;

        // Update cache (TTL: 5 minutes)
        self.permission_cache.lock().await.put(cache_key, allowed);

        Ok(allowed)
    }
}
```

**Permission Invalidation** (on role changes):

```sql
-- Trigger to send NOTIFY when permissions change
CREATE OR REPLACE FUNCTION notify_permission_change() RETURNS TRIGGER AS $$
BEGIN
    PERFORM pg_notify('permission_change', NEW.user_id::text);
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER permission_change_trigger
AFTER INSERT OR UPDATE OR DELETE ON user_roles
FOR EACH ROW EXECUTE FUNCTION notify_permission_change();
```

---

## Logging & Monitoring

### Audit Trail Requirements

**Immutable Audit Logs**:

```sql
-- Prevent deletion of audit events
CREATE POLICY audit_events_no_delete ON audit_events
FOR DELETE USING (false);

-- Prevent updates of audit events
CREATE POLICY audit_events_no_update ON audit_events
FOR UPDATE USING (false);

-- Verify immutability
DELETE FROM audit_events WHERE id = 'some-id';
-- Expected: ERROR:  new row violates row-level security policy
```

**Audit Event Structure**:

```rust
// trading_engine/src/compliance/audit_trails.rs
#[derive(Debug, Serialize, Deserialize)]
pub struct AuditEvent {
    pub id: Uuid,
    pub event_type: String,        // login, order_submit, config_change, etc.
    pub user_id: Option<Uuid>,
    pub ip_address: Option<String>,
    pub timestamp: DateTime<Utc>,  // Microsecond precision
    pub resource: String,           // orders, users, config, etc.
    pub action: String,             // create, read, update, delete
    pub old_value: Option<serde_json::Value>,
    pub new_value: Option<serde_json::Value>,
    pub success: bool,
    pub error_message: Option<String>,
}
```

### SIEM Integration

**Export Audit Logs to ELK**:

```bash
# Filebeat configuration
sudo tee /etc/filebeat/filebeat.yml <<EOF
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/foxhunt/*.log
    fields:
      environment: production
      application: foxhunt

output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  index: "foxhunt-audit-%{+yyyy.MM.dd}"

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
EOF

sudo systemctl restart filebeat
```

**Kibana Dashboards**:
- Failed login attempts (security)
- Order submission rate (business)
- API Gateway error rate (operational)
- Database query performance (performance)

### Security Event Alerting

**PagerDuty/Slack Integration**:

```yaml
# Prometheus AlertManager configuration
groups:
  - name: security
    interval: 30s
    rules:
      - alert: HighFailedLoginRate
        expr: rate(failed_logins[5m]) > 10
        annotations:
          summary: "High failed login rate detected"
          description: "{{ $value }} failed logins/min in last 5 minutes"

      - alert: UnauthorizedAccessAttempt
        expr: unauthorized_access_attempts > 0
        annotations:
          summary: "Unauthorized access attempt"
          description: "User {{ $labels.user_id }} attempted to access {{ $labels.endpoint }}"

      - alert: JWTRevocationServiceDown
        expr: up{job="redis"} == 0
        for: 1m
        annotations:
          summary: "JWT revocation service (Redis) is down"
          description: "All authentication will fail"
```

---

## Incident Response

### Security Incident Classification

| Severity | Definition | Response Time | Example |
|----------|------------|---------------|---------|
| P0 (Critical) | Active breach, data exfiltration | Immediate | Database dump stolen |
| P1 (High) | Potential breach, exploit detected | 15 minutes | SQL injection attempt |
| P2 (Medium) | Suspicious activity, failed attacks | 1 hour | Brute force login attempts |
| P3 (Low) | Policy violation, minor vulnerability | 24 hours | Weak password detected |

### Incident Response Playbook

**P0: Active Breach**:

1. **Contain** (0-5 minutes):
   ```bash
   # Emergency shutdown
   sudo systemctl stop api_gateway trading_service backtesting_service ml_training_service

   # Block all network traffic
   sudo iptables -P INPUT DROP
   sudo iptables -P OUTPUT DROP
   ```

2. **Assess** (5-15 minutes):
   ```bash
   # Check recent logins
   psql $DATABASE_URL -c "SELECT * FROM audit_events WHERE event_type='login' AND created_at > NOW() - INTERVAL '1 hour' ORDER BY created_at DESC;"

   # Check active database connections
   psql $DATABASE_URL -c "SELECT * FROM pg_stat_activity;"
   ```

3. **Eradicate** (15-60 minutes):
   - Rotate all credentials (database, Redis, JWT secret)
   - Revoke all JWT tokens
   - Patch exploited vulnerability

4. **Recover** (1-4 hours):
   - Restore from clean backup if needed
   - Gradually restore services
   - Monitor for re-compromise

5. **Post-Mortem** (within 48 hours):
   - Document timeline
   - Identify root cause
   - Implement preventive measures

**P1: SQL Injection Attempt**:

```bash
# Check PostgreSQL logs for suspicious queries
sudo grep -i "ERROR" /var/log/postgresql/postgresql-*.log | tail -100

# Block attacking IP
sudo iptables -A INPUT -s ATTACKER_IP -j DROP

# Review audit trail
psql $DATABASE_URL -c "SELECT * FROM audit_events WHERE success = false AND created_at > NOW() - INTERVAL '1 hour';"
```

---

## Security Checklist

### Pre-Production Security Review

- [ ] **Network Security**
  - [ ] Firewall rules configured and tested
  - [ ] VPC/subnet segmentation implemented
  - [ ] mTLS enforced for all inter-service communication
  - [ ] TLS 1.3 with strong ciphers configured
  - [ ] DDoS protection enabled

- [ ] **Host Security**
  - [ ] CIS benchmarks applied
  - [ ] Automatic security updates enabled
  - [ ] SSH hardened (no root login, key-based auth only)
  - [ ] File permissions verified
  - [ ] Intrusion detection (Fail2Ban) configured

- [ ] **Application Security**
  - [ ] Input validation on all API endpoints
  - [ ] SQL injection prevention verified (parameterized queries only)
  - [ ] Dependency vulnerabilities scanned (`cargo audit` passing)
  - [ ] `unsafe` code reviewed
  - [ ] Rate limiting enforced (100 req/s per user)

- [ ] **Data Security**
  - [ ] Encryption at rest enabled (PostgreSQL, Redis, S3)
  - [ ] Encryption in transit enforced (TLS 1.3)
  - [ ] Database access control implemented (least privilege)
  - [ ] Sensitive data encrypted (TOTP secrets, MFA backup codes)

- [ ] **Secrets Management**
  - [ ] Vault deployed and accessible
  - [ ] All secrets stored in Vault (no hardcoded secrets)
  - [ ] Secret rotation policies defined
  - [ ] JWT secret >= 64 bytes of entropy

- [ ] **Authentication & Authorization**
  - [ ] JWT validation with strong algorithm (HS512)
  - [ ] JWT revocation working (Redis connectivity verified)
  - [ ] MFA/TOTP implemented and tested
  - [ ] RBAC permissions configured
  - [ ] Audit logging enabled for all auth events

- [ ] **Logging & Monitoring**
  - [ ] Audit trails immutable (DELETE/UPDATE blocked)
  - [ ] SIEM integration configured (ELK/Splunk)
  - [ ] Security alerts configured (PagerDuty/Slack)
  - [ ] Log retention policy enforced (7 years for audit trails)

- [ ] **Compliance**
  - [ ] SOX requirements documented and verified
  - [ ] MiFID II timestamping accurate (±100μs)
  - [ ] Data retention policies implemented
  - [ ] Incident response plan reviewed

### Ongoing Security Operations

**Daily**:
- [ ] Review failed login attempts
- [ ] Check security alerts (PagerDuty/Slack)
- [ ] Verify all services healthy

**Weekly**:
- [ ] Review audit trail for anomalies
- [ ] Scan dependencies for vulnerabilities (`cargo audit`)
- [ ] Review firewall logs for blocked IPs

**Monthly**:
- [ ] Rotate JWT secret
- [ ] Review and update RBAC permissions
- [ ] Test disaster recovery procedures
- [ ] Security training for team

**Quarterly**:
- [ ] Penetration testing
- [ ] Compliance audit (SOX/MiFID II)
- [ ] Security policy review and update
- [ ] Certificate renewal (if needed)

---

**End of Security Hardening Guide**

*This document must be reviewed quarterly and updated after every security incident.*