foxhunt/docs/TLI_SECURITY_DOCUMENTATION.md

# TLI Security Documentation
**Foxhunt Trading System - Security Guide**

Version: 1.0
Last Updated: 2025-01-23
Document Classification: Security Sensitive

---

## Table of Contents

1. [Security Overview](#security-overview)
2. [Authentication Setup](#authentication-setup)
3. [Certificate Management](#certificate-management)
4. [Access Control (RBAC)](#access-control-rbac)
5. [API Security](#api-security)
6. [Network Security](#network-security)
7. [Security Best Practices](#security-best-practices)
8. [Incident Response](#incident-response)
9. [Compliance Framework](#compliance-framework)
10. [Security Monitoring](#security-monitoring)

---

## Security Overview

The TLI system implements enterprise-grade security controls designed for financial trading environments, meeting regulatory requirements including:

- **SOX (Sarbanes-Oxley)**: Financial reporting and audit trail requirements
- **FINRA**: Broker-dealer regulation compliance
- **ISO 27001**: Information security management standards
- **PCI DSS**: Payment card industry security (where applicable)

### Security Architecture

```
┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   TLI Client    │────│  mTLS Channel    │────│ Trading Service │
│  (Authenticated)│    │  (Certificate    │    │   (Secured)     │
│                 │    │   Validation)    │    │                 │
└─────────────────┘    └──────────────────┘    └─────────────────┘
         │                       │                       │
         │              ┌────────┴────────┐              │
         │              │  Auth Service   │              │
         └──────────────│ - Session Mgmt  │──────────────┘
                        │ - API Keys      │
                        │ - RBAC          │
                        │ - Audit Logs    │
                        └─────────────────┘
```

### Security Layers

1. **Transport Security**: TLS 1.3 with mutual authentication
2. **Authentication**: Multi-factor authentication with session management
3. **Authorization**: Role-based access control (RBAC)
4. **Network Security**: Firewall, VPN, and network segmentation
5. **Audit & Compliance**: Comprehensive logging and monitoring
6. **Data Protection**: Encryption at rest and in transit

---

## Authentication Setup

### User Authentication

#### Password-Based Authentication

```bash
# Create user with secure password policy
cargo run --bin tli-admin -- create-user \
  --username "trader_001" \
  --email "trader@company.com" \
  --role "trader" \
  --force-password-change

# Set password policy
cargo run --bin tli-admin -- set-password-policy \
  --min-length 12 \
  --require-uppercase \
  --require-lowercase \
  --require-numbers \
  --require-symbols \
  --max-age-days 90 \
  --history-count 12
```

#### Multi-Factor Authentication (MFA)

```bash
# Enable MFA for user
cargo run --bin tli-admin -- enable-mfa \
  --username "trader_001" \
  --method "totp" \
  --backup-codes 10

# Verify MFA setup
cargo run --bin tli-client -- login \
  --username "trader_001" \
  --password "secure_password" \
  --mfa-code "123456"
```

### API Key Management

#### Creating API Keys

```rust
use tli::auth::*;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let auth_service = AuthenticationService::new(security_config).await?;

    // Create API key with specific permissions
    let api_key = auth_service.create_api_key(
        "user_id_123",
        "Trading Algorithm v2.1",
        vec![
            "api:access".to_string(),
            "trade:execute".to_string(),
            "market_data:read".to_string(),
            "positions:read".to_string(),
        ],
        Some(90) // 90 days expiration
    ).await?;

    println!("API Key ID: {}", api_key.id);
    println!("API Key: {}", api_key.key);
    println!("Expires: {}", api_key.expires_at);

    Ok(())
}
```

#### API Key Rotation

```bash
# Automated rotation script
#!/bin/bash
set -e

USER_ID="trading_bot_001"
OLD_KEY_ID="api_key_123"
KEY_NAME="Trading Bot - Auto Rotated"

# Create new API key
NEW_KEY=$(cargo run --bin tli-admin -- create-api-key \
  --user-id "$USER_ID" \
  --name "$KEY_NAME" \
  --permissions "api:access,trade:execute,market_data:read" \
  --expires-days 90 \
  --output json)

# Extract new key details
NEW_KEY_ID=$(echo "$NEW_KEY" | jq -r '.id')
NEW_KEY_VALUE=$(echo "$NEW_KEY" | jq -r '.key')

# Update application configuration
echo "Updating application with new API key..."
kubectl create secret generic trading-bot-api-key \
  --from-literal=api-key="$NEW_KEY_VALUE" \
  --dry-run=client -o yaml | kubectl apply -f -

# Restart application pods
kubectl rollout restart deployment/trading-bot

# Wait for successful deployment
kubectl rollout status deployment/trading-bot

# Revoke old API key
cargo run --bin tli-admin -- revoke-api-key --key-id "$OLD_KEY_ID"

echo "API key rotation completed successfully"
echo "New API Key ID: $NEW_KEY_ID"
```

### Session Management

#### Session Configuration

```toml
[security.session]
timeout_seconds = 3600          # 1 hour session timeout
max_sessions_per_user = 3       # Maximum concurrent sessions
token_length = 32               # Session token length
refresh_interval_seconds = 300  # 5 minute refresh requirement
secure_cookies = true           # Secure cookie settings
same_site = "Strict"           # CSRF protection
```

#### Session Monitoring

```bash
# Monitor active sessions
cargo run --bin tli-admin -- list-sessions \
  --active-only \
  --format table

# Force logout suspicious sessions
cargo run --bin tli-admin -- revoke-session \
  --session-id "session_123" \
  --reason "Security incident"

# Monitor session statistics
cargo run --bin tli-admin -- session-stats \
  --period "24h" \
  --group-by user
```

---

## Certificate Management

### TLS Certificate Setup

#### Generating Production Certificates

```bash
#!/bin/bash
# Generate CA certificate
openssl genrsa -out ca.key 4096
openssl req -new -x509 -days 3650 -key ca.key -out ca.crt \
  -subj "/C=US/ST=NY/L=NYC/O=Trading Company/OU=IT/CN=Foxhunt CA"

# Generate server certificate
openssl genrsa -out server.key 4096
openssl req -new -key server.key -out server.csr \
  -subj "/C=US/ST=NY/L=NYC/O=Trading Company/OU=Trading/CN=trading.company.com"

# Create server certificate extensions
cat > server.ext << EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names

[alt_names]
DNS.1 = trading.company.com
DNS.2 = localhost
IP.1 = 127.0.0.1
IP.2 = 10.0.0.100
EOF

# Sign server certificate
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -out server.crt -days 365 -extensions v3_ext -extfile server.ext

# Generate client certificate for mTLS
openssl genrsa -out client.key 4096
openssl req -new -key client.key -out client.csr \
  -subj "/C=US/ST=NY/L=NYC/O=Trading Company/OU=Clients/CN=trading-client"

openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -out client.crt -days 365

# Set secure permissions
chmod 600 *.key
chmod 644 *.crt
chown foxhunt:foxhunt *.key *.crt

# Install certificates
mkdir -p /etc/foxhunt/tls
cp ca.crt server.crt server.key client.crt client.key /etc/foxhunt/tls/
```

#### Certificate Validation

```bash
# Verify certificate chain
openssl verify -CAfile /etc/foxhunt/tls/ca.crt /etc/foxhunt/tls/server.crt

# Check certificate expiration
openssl x509 -in /etc/foxhunt/tls/server.crt -text -noout | grep "Not After"

# Test TLS connection
openssl s_client -connect localhost:50051 -CAfile /etc/foxhunt/tls/ca.crt \
  -cert /etc/foxhunt/tls/client.crt -key /etc/foxhunt/tls/client.key
```

#### Automated Certificate Renewal

```bash
#!/bin/bash
# Certificate renewal script
set -e

CERT_DIR="/etc/foxhunt/tls"
BACKUP_DIR="/etc/foxhunt/tls/backup/$(date +%Y%m%d)"
DAYS_BEFORE_EXPIRY=30

# Check certificate expiration
EXPIRY_DATE=$(openssl x509 -in "$CERT_DIR/server.crt" -noout -enddate | cut -d= -f2)
EXPIRY_EPOCH=$(date -d "$EXPIRY_DATE" +%s)
CURRENT_EPOCH=$(date +%s)
DAYS_UNTIL_EXPIRY=$(( (EXPIRY_EPOCH - CURRENT_EPOCH) / 86400 ))

if [ $DAYS_UNTIL_EXPIRY -gt $DAYS_BEFORE_EXPIRY ]; then
    echo "Certificate valid for $DAYS_UNTIL_EXPIRY days, renewal not needed"
    exit 0
fi

echo "Certificate expires in $DAYS_UNTIL_EXPIRY days, renewing..."

# Backup existing certificates
mkdir -p "$BACKUP_DIR"
cp "$CERT_DIR"/*.crt "$CERT_DIR"/*.key "$BACKUP_DIR/"

# Generate new certificates (reuse CA)
# ... (certificate generation commands) ...

# Test new certificates
if openssl verify -CAfile "$CERT_DIR/ca.crt" "$CERT_DIR/server.crt"; then
    echo "New certificate verified successfully"

    # Restart services with new certificates
    systemctl reload foxhunt-trading-service
    systemctl reload foxhunt-backtesting-service

    echo "Certificate renewal completed successfully"
else
    echo "Certificate verification failed, rolling back"
    cp "$BACKUP_DIR"/* "$CERT_DIR/"
    exit 1
fi
```

### Certificate Monitoring

```bash
# Monitor certificate expiration
cargo run --bin tli-admin -- cert-status \
  --warn-days 30 \
  --critical-days 7

# Certificate health check
cargo run --bin tli-admin -- health-check \
  --component certificates \
  --format json
```

---

## Access Control (RBAC)

### Role Definitions

#### Predefined Roles

```yaml
# /etc/foxhunt/rbac/roles.yaml
roles:
  admin:
    description: "System administrator with full access"
    permissions:
      - "system:*"
      - "user:*"
      - "audit:*"
      - "config:*"

  trader:
    description: "Trader with execution permissions"
    permissions:
      - "trade:execute"
      - "order:place"
      - "order:cancel"
      - "positions:read"
      - "market_data:read"
      - "risk:read"

  analyst:
    description: "Research analyst with read-only access"
    permissions:
      - "market_data:read"
      - "positions:read"
      - "risk:read"
      - "backtest:run"
      - "backtest:read"

  risk_manager:
    description: "Risk management specialist"
    permissions:
      - "risk:*"
      - "positions:read"
      - "order:cancel"
      - "emergency:stop"
      - "limits:modify"

  auditor:
    description: "Compliance and audit access"
    permissions:
      - "audit:read"
      - "logs:read"
      - "compliance:read"
      - "reports:generate"

  api_service:
    description: "Service account for API access"
    permissions:
      - "api:access"
      - "trade:execute"
      - "market_data:read"
      - "positions:read"
```

#### Custom Permissions

```bash
# Create custom permission
cargo run --bin tli-admin -- create-permission \
  --name "algo:deploy" \
  --description "Deploy algorithmic trading strategies" \
  --resource "algorithm" \
  --action "deploy"

# Assign permission to role
cargo run --bin tli-admin -- add-permission-to-role \
  --role "senior_trader" \
  --permission "algo:deploy"

# Create user with custom role
cargo run --bin tli-admin -- create-user \
  --username "algo_trader" \
  --role "senior_trader" \
  --department "quantitative_trading"
```

### Permission Management

#### Granting Permissions

```rust
use tli::auth::*;

// Grant temporary elevated permissions
let rbac_manager = RbacManager::new(rbac_config).await?;

rbac_manager.grant_temporary_permission(
    "user_123",
    "emergency:override",
    chrono::Duration::hours(1)
).await?;

// Revoke permissions
rbac_manager.revoke_permission("user_123", "emergency:override").await?;
```

#### Permission Auditing

```bash
# Audit user permissions
cargo run --bin tli-admin -- audit-permissions \
  --user "trader_001" \
  --output detailed

# Check effective permissions
cargo run --bin tli-admin -- effective-permissions \
  --user "trader_001" \
  --resource "order" \
  --action "place"

# Permission usage report
cargo run --bin tli-admin -- permission-usage-report \
  --period "30d" \
  --format csv
```

---

## API Security

### Rate Limiting

#### Rate Limit Configuration

```toml
[security.rate_limiting]
# Per-user limits
authenticated_rpm = 1000        # Requests per minute for authenticated users
api_key_rpm = 5000             # Requests per minute for API keys

# Trading-specific limits
trading_burst = 100            # Burst allowance for trading operations
order_rpm = 300               # Orders per minute limit
cancel_rpm = 500              # Cancellations per minute limit

# Global limits
global_rpm = 50000            # Global system limit
window_seconds = 60           # Rate limiting window

# Security limits
failed_auth_limit = 5         # Failed authentication attempts
lockout_duration_minutes = 15 # Account lockout duration
```

#### Rate Limiting Implementation

```rust
use tli::auth::rate_limiter::*;

// Custom rate limiter for trading operations
let trading_limiter = RateLimiter::new(RateLimitConfig {
    requests_per_window: 100,
    window_duration: Duration::from_secs(60),
    burst_size: 10,
    rate_limit_type: RateLimitType::SlidingWindow,
})?;

// Check rate limit before processing
match trading_limiter.check_limit(&client_id).await {
    Ok(_) => {
        // Process trading request
        process_trade_request(request).await?;
    }
    Err(RateLimitError::LimitExceeded { limit, window, .. }) => {
        return Err(TradingError::RateLimitExceeded {
            limit,
            window,
            retry_after: calculate_retry_after(),
        });
    }
}
```

### Input Validation

#### Request Validation

```rust
use validator::{Validate, ValidationError};
use serde::{Deserialize, Serialize};

#[derive(Debug, Deserialize, Validate)]
pub struct OrderRequest {
    #[validate(length(min = 1, max = 20, message = "Symbol must be 1-20 characters"))]
    #[validate(regex = "SYMBOL_REGEX", message = "Invalid symbol format")]
    pub symbol: String,

    #[validate(range(min = 0.01, max = 1000000.0, message = "Quantity must be between 0.01 and 1,000,000"))]
    pub quantity: f64,

    #[validate(range(min = 0.01, message = "Price must be positive"))]
    pub price: Option<f64>,

    #[validate(custom = "validate_time_in_force")]
    pub time_in_force: String,
}

fn validate_time_in_force(tif: &str) -> Result<(), ValidationError> {
    match tif {
        "DAY" | "GTC" | "IOC" | "FOK" => Ok(()),
        _ => Err(ValidationError::new("Invalid time in force")),
    }
}

// Middleware for automatic validation
pub async fn validate_request<T: Validate>(request: T) -> Result<T, ValidationError> {
    request.validate()?;
    Ok(request)
}
```

#### SQL Injection Prevention

```rust
// Use parameterized queries
use sqlx::{PgPool, query};

pub async fn get_user_orders(
    pool: &PgPool,
    user_id: &str,
    symbol: Option<&str>
) -> Result<Vec<Order>, sqlx::Error> {
    let query = match symbol {
        Some(sym) => {
            query!(
                "SELECT * FROM orders WHERE user_id = $1 AND symbol = $2 ORDER BY created_at DESC",
                user_id,
                sym
            )
        }
        None => {
            query!(
                "SELECT * FROM orders WHERE user_id = $1 ORDER BY created_at DESC",
                user_id
            )
        }
    };

    // Execute query safely with parameters
    query.fetch_all(pool).await
}
```

---

## Network Security

### Firewall Configuration

#### iptables Rules

```bash
#!/bin/bash
# Firewall configuration for TLI services

# Clear existing rules
iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X

# Default policies
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT

# Allow loopback
iptables -A INPUT -i lo -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT

# Allow established connections
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# SSH access (from management network only)
iptables -A INPUT -p tcp --dport 22 -s 10.0.1.0/24 -j ACCEPT

# TLI services (from trusted networks only)
iptables -A INPUT -p tcp --dport 50051 -s 10.0.0.0/16 -j ACCEPT
iptables -A INPUT -p tcp --dport 50052 -s 10.0.0.0/16 -j ACCEPT

# Health check endpoint (internal monitoring)
iptables -A INPUT -p tcp --dport 8080 -s 10.0.0.100 -j ACCEPT

# Database access (from application servers only)
iptables -A INPUT -p tcp --dport 5432 -s 10.0.0.50 -j ACCEPT
iptables -A INPUT -p tcp --dport 5432 -s 10.0.0.51 -j ACCEPT

# Log dropped packets
iptables -A INPUT -j LOG --log-prefix "DROPPED: " --log-level 4
iptables -A INPUT -j DROP

# Save rules
iptables-save > /etc/iptables/rules.v4
```

#### Network Segmentation

```yaml
# Network segmentation design
networks:
  management:
    cidr: "10.0.1.0/24"
    purpose: "Administrative access"
    access: "SSH, monitoring"

  trading:
    cidr: "10.0.0.0/24"
    purpose: "Trading applications"
    access: "TLI services, databases"

  market_data:
    cidr: "10.0.2.0/24"
    purpose: "Market data feeds"
    access: "Market data providers"

  dmz:
    cidr: "192.168.100.0/24"
    purpose: "External facing services"
    access: "Limited internet access"
```

### VPN Configuration

#### WireGuard Setup

```ini
# /etc/wireguard/wg0.conf
[Interface]
PrivateKey = <server_private_key>
Address = 10.0.10.1/24
ListenPort = 51820
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

# Trading team access
[Peer]
PublicKey = <trader_public_key>
AllowedIPs = 10.0.10.10/32
PersistentKeepalive = 25

# Risk management access
[Peer]
PublicKey = <risk_manager_public_key>
AllowedIPs = 10.0.10.20/32
PersistentKeepalive = 25
```

---

## Security Best Practices

### Secure Coding Practices

#### Input Sanitization

```rust
use regex::Regex;
use lazy_static::lazy_static;

lazy_static! {
    static ref SYMBOL_REGEX: Regex = Regex::new(r"^[A-Z]{1,10}$").unwrap();
    static ref ORDER_ID_REGEX: Regex = Regex::new(r"^[A-Za-z0-9\-_]{1,50}$").unwrap();
}

pub fn validate_symbol(symbol: &str) -> Result<String, ValidationError> {
    if !SYMBOL_REGEX.is_match(symbol) {
        return Err(ValidationError::new("Invalid symbol format"));
    }

    // Additional validation
    if symbol.len() > 10 {
        return Err(ValidationError::new("Symbol too long"));
    }

    Ok(symbol.to_uppercase())
}

pub fn sanitize_order_id(order_id: &str) -> Result<String, ValidationError> {
    if !ORDER_ID_REGEX.is_match(order_id) {
        return Err(ValidationError::new("Invalid order ID format"));
    }

    // Remove any potentially dangerous characters
    let sanitized = order_id.chars()
        .filter(|c| c.is_alphanumeric() || *c == '-' || *c == '_')
        .collect::<String>();

    Ok(sanitized)
}
```

#### Secure Error Handling

```rust
use tracing::{error, warn};

pub enum TradingError {
    InvalidCredentials,
    InsufficientFunds { required: f64, available: f64 },
    OrderNotFound { order_id: String },
    SystemError { correlation_id: String },
}

impl TradingError {
    pub fn to_client_error(&self) -> ClientError {
        match self {
            TradingError::InvalidCredentials => {
                // Log security event but don't expose details
                warn!("Authentication attempt failed");
                ClientError::Unauthenticated("Authentication required".to_string())
            }
            TradingError::InsufficientFunds { required, available } => {
                ClientError::BusinessLogic(format!(
                    "Insufficient funds: required ${:.2}, available ${:.2}",
                    required, available
                ))
            }
            TradingError::OrderNotFound { order_id } => {
                // Log potential enumeration attempt
                warn!("Order lookup for non-existent order: {}", order_id);
                ClientError::NotFound("Order not found".to_string())
            }
            TradingError::SystemError { correlation_id } => {
                // Log detailed error internally, return generic message
                error!("System error occurred: {}", correlation_id);
                ClientError::Internal(format!("Internal error (ref: {})", correlation_id))
            }
        }
    }
}
```

### Environment Hardening

#### Service Configuration

```toml
# Secure service configuration
[security]
# Disable unnecessary features
enable_debug_endpoints = false
enable_metrics_export = false  # Only enable in monitoring environment
enable_pprof = false

# Restrict file access
config_file_permissions = "0600"
log_file_permissions = "0640"
data_directory = "/var/lib/foxhunt"

# Process isolation
run_as_user = "foxhunt"
run_as_group = "foxhunt"
enable_seccomp = true
enable_apparmor = true

# Resource limits
max_memory_mb = 4096
max_file_descriptors = 8192
max_cpu_percent = 80
```

#### Docker Security

```dockerfile
# Multi-stage build for security
FROM rust:1.75-slim as builder
WORKDIR /app
COPY . .
RUN cargo build --release

FROM debian:bullseye-slim
# Create non-root user
RUN groupadd -r foxhunt && useradd -r -g foxhunt foxhunt

# Install security updates only
RUN apt-get update && apt-get install -y --no-install-recommends \
    ca-certificates \
    && rm -rf /var/lib/apt/lists/*

# Copy binary and set permissions
COPY --from=builder /app/target/release/trading-service /usr/local/bin/
RUN chmod +x /usr/local/bin/trading-service

# Create necessary directories
RUN mkdir -p /var/lib/foxhunt /var/log/foxhunt && \
    chown -R foxhunt:foxhunt /var/lib/foxhunt /var/log/foxhunt

# Switch to non-root user
USER foxhunt

# Security labels
LABEL security.scan="enabled"
LABEL security.compliance="SOX,FINRA"

EXPOSE 50051
CMD ["/usr/local/bin/trading-service"]
```

### Secret Management

#### HashiCorp Vault Integration

```rust
use vault::VaultClient;
use serde_json::Value;

pub struct SecretManager {
    vault_client: VaultClient,
    mount_path: String,
}

impl SecretManager {
    pub async fn new(vault_url: &str, token: &str) -> Result<Self, VaultError> {
        let client = VaultClient::new(vault_url, token)?;

        Ok(Self {
            vault_client: client,
            mount_path: "foxhunt-secrets".to_string(),
        })
    }

    pub async fn get_database_credentials(&self) -> Result<DatabaseCredentials, VaultError> {
        let secret_path = format!("{}/database/primary", self.mount_path);
        let secret = self.vault_client.read_secret(&secret_path).await?;

        Ok(DatabaseCredentials {
            username: secret["username"].as_str().unwrap().to_string(),
            password: secret["password"].as_str().unwrap().to_string(),
            host: secret["host"].as_str().unwrap().to_string(),
            port: secret["port"].as_u64().unwrap() as u16,
        })
    }

    pub async fn get_api_encryption_key(&self) -> Result<Vec<u8>, VaultError> {
        let secret_path = format!("{}/encryption/api-keys", self.mount_path);
        let secret = self.vault_client.read_secret(&secret_path).await?;

        let key_b64 = secret["key"].as_str().unwrap();
        base64::decode(key_b64).map_err(|e| VaultError::DecodingError(e.to_string()))
    }
}
```

---

## Incident Response

### Security Incident Procedures

#### Incident Classification

| Level | Description | Response Time | Escalation |
|-------|-------------|---------------|------------|
| P1 | Active breach, trading impact | 5 minutes | CTO, Legal, PR |
| P2 | Security vulnerability, no breach | 30 minutes | Security Team |
| P3 | Policy violation, minor impact | 2 hours | Team Lead |
| P4 | Information gathering, no impact | 24 hours | Security Team |

#### Incident Response Playbook

```bash
#!/bin/bash
# Incident response automation script
set -e

INCIDENT_ID="INC-$(date +%Y%m%d-%H%M%S)"
INCIDENT_LEVEL="$1"
INCIDENT_TYPE="$2"
DESCRIPTION="$3"

echo "Starting incident response for $INCIDENT_ID"
echo "Level: $INCIDENT_LEVEL"
echo "Type: $INCIDENT_TYPE"

# Immediate containment actions
case "$INCIDENT_LEVEL" in
    "P1")
        echo "P1 Incident - Executing immediate containment"

        # Emergency stop all trading
        cargo run --bin tli-admin -- emergency-stop \
            --type full_shutdown \
            --reason "Security incident $INCIDENT_ID" \
            --confirm

        # Revoke all active sessions
        cargo run --bin tli-admin -- revoke-all-sessions \
            --reason "Security incident"

        # Block all external access
        iptables -A INPUT -s 0.0.0.0/0 -j DROP

        # Alert incident response team
        curl -X POST "$SLACK_WEBHOOK" \
            -H 'Content-type: application/json' \
            --data "{\"text\":\"🚨 P1 Security Incident: $INCIDENT_ID - $DESCRIPTION\"}"
        ;;

    "P2")
        echo "P2 Incident - Standard containment"

        # Increase monitoring sensitivity
        cargo run --bin tli-admin -- set-alert-level --level high

        # Capture system state
        /opt/foxhunt/scripts/capture-system-state.sh "$INCIDENT_ID"
        ;;
esac

# Evidence collection
mkdir -p "/var/log/incidents/$INCIDENT_ID"
cp /var/log/foxhunt/audit.log "/var/log/incidents/$INCIDENT_ID/"
cp /var/log/foxhunt/security.log "/var/log/incidents/$INCIDENT_ID/"

# Generate incident report
cargo run --bin tli-admin -- generate-incident-report \
    --incident-id "$INCIDENT_ID" \
    --level "$INCIDENT_LEVEL" \
    --type "$INCIDENT_TYPE" \
    --output "/var/log/incidents/$INCIDENT_ID/report.json"

echo "Incident response initiated for $INCIDENT_ID"
```

#### Evidence Collection

```bash
# Automated evidence collection script
#!/bin/bash
INCIDENT_ID="$1"
EVIDENCE_DIR="/secure/incidents/$INCIDENT_ID"

mkdir -p "$EVIDENCE_DIR"
cd "$EVIDENCE_DIR"

# System state
ps aux > processes.txt
netstat -tulnp > network_connections.txt
lsof > open_files.txt
df -h > disk_usage.txt
free -h > memory_usage.txt

# Network traffic capture
tcpdump -i any -w network_capture.pcap -c 10000 &
TCPDUMP_PID=$!

# Application logs
cp /var/log/foxhunt/*.log ./
cp /var/log/nginx/access.log ./
cp /var/log/postgresql/postgresql.log ./

# Configuration files
tar -czf configs.tar.gz /etc/foxhunt/

# Database snapshot
pg_dump foxhunt_db > database_snapshot.sql

# Stop network capture
sleep 60
kill $TCPDUMP_PID

# Create evidence manifest
sha256sum * > evidence_manifest.txt

# Encrypt evidence
tar -czf - * | gpg --cipher-algo AES256 --compress-algo 1 \
    --symmetric --output "evidence-$INCIDENT_ID.tar.gz.gpg"

echo "Evidence collection completed for incident $INCIDENT_ID"
```

### Recovery Procedures

#### System Recovery

```bash
#!/bin/bash
# System recovery after security incident
set -e

INCIDENT_ID="$1"
RECOVERY_TYPE="$2"  # full, partial, config-only

echo "Starting recovery for incident $INCIDENT_ID"

case "$RECOVERY_TYPE" in
    "full")
        echo "Full system recovery"

        # Stop all services
        systemctl stop foxhunt-*

        # Restore from clean backup
        systemctl stop postgresql
        pg_restore -d foxhunt_db /backup/clean/database.sql
        systemctl start postgresql

        # Restore configuration from vault
        /opt/foxhunt/scripts/restore-config-from-vault.sh

        # Regenerate all certificates
        /opt/foxhunt/scripts/generate-certificates.sh --force

        # Reset all passwords and API keys
        cargo run --bin tli-admin -- reset-all-credentials

        # Start services
        systemctl start foxhunt-trading-service
        systemctl start foxhunt-backtesting-service
        ;;

    "partial")
        echo "Partial recovery - configuration and credentials only"

        # Revoke compromised credentials
        cargo run --bin tli-admin -- revoke-compromised-credentials \
            --incident-id "$INCIDENT_ID"

        # Rotate API keys
        /opt/foxhunt/scripts/rotate-all-api-keys.sh

        # Update firewall rules
        /opt/foxhunt/scripts/update-firewall-rules.sh --strict
        ;;
esac

# Verify system integrity
cargo run --bin tli-admin -- verify-system-integrity

# Test all critical functions
cargo run --bin tli-admin -- run-health-checks --comprehensive

echo "Recovery completed for incident $INCIDENT_ID"
```

---

## Compliance Framework

### Regulatory Requirements

#### SOX Compliance

```yaml
# SOX compliance configuration
sox_compliance:
  financial_reporting:
    - audit_trail_retention: "7_years"
    - transaction_logging: "all_trades"
    - access_controls: "segregation_of_duties"
    - change_management: "approval_required"

  internal_controls:
    - user_access_reviews: "quarterly"
    - privilege_escalation_monitoring: "real_time"
    - configuration_change_approval: "required"
    - backup_verification: "monthly"

  documentation:
    - control_descriptions: "detailed"
    - risk_assessments: "annual"
    - testing_procedures: "documented"
    - remediation_tracking: "automated"
```

#### FINRA Compliance

```bash
# FINRA compliance monitoring
#!/bin/bash

# Trade reporting validation
cargo run --bin compliance-validator -- trade-reporting \
    --date "$(date +%Y-%m-%d)" \
    --format FINRA_CAT

# Order audit trail
cargo run --bin compliance-validator -- order-audit-trail \
    --include-cancellations \
    --include-modifications \
    --output /compliance/reports/oats-$(date +%Y%m%d).xml

# Supervision and surveillance
cargo run --bin compliance-validator -- surveillance-report \
    --type "unusual_activity" \
    --threshold-config /etc/foxhunt/surveillance-thresholds.yaml
```

### Audit Trail Management

#### Comprehensive Logging

```rust
use tracing::{info, warn, error};
use serde_json::json;

#[derive(Debug, Serialize)]
pub struct AuditEvent {
    pub event_id: String,
    pub timestamp: DateTime<Utc>,
    pub user_id: Option<String>,
    pub session_id: Option<String>,
    pub client_ip: String,
    pub event_type: AuditEventType,
    pub resource: String,
    pub action: String,
    pub outcome: AuditOutcome,
    pub details: serde_json::Value,
}

impl AuditEvent {
    pub fn trading_operation(
        user_id: &str,
        client_ip: &str,
        operation: &str,
        symbol: &str,
        quantity: f64,
        outcome: AuditOutcome,
    ) -> Self {
        Self {
            event_id: Uuid::new_v4().to_string(),
            timestamp: Utc::now(),
            user_id: Some(user_id.to_string()),
            session_id: None, // Set by middleware
            client_ip: client_ip.to_string(),
            event_type: AuditEventType::Trading,
            resource: "order".to_string(),
            action: operation.to_string(),
            outcome,
            details: json!({
                "symbol": symbol,
                "quantity": quantity,
                "timestamp_nanos": Utc::now().timestamp_nanos(),
            }),
        }
    }
}

// Audit middleware for all gRPC requests
pub async fn audit_middleware<T>(
    request: Request<T>,
    handler: impl Future<Output = Result<Response<U>, Status>>,
) -> Result<Response<U>, Status> {
    let start_time = Instant::now();
    let user_id = extract_user_id(&request)?;
    let client_ip = extract_client_ip(&request)?;

    let result = handler.await;
    let duration = start_time.elapsed();

    let outcome = match &result {
        Ok(_) => AuditOutcome::Success,
        Err(status) => AuditOutcome::Failure {
            error_code: status.code().to_string(),
            error_message: status.message().to_string(),
        },
    };

    // Log audit event
    let audit_event = AuditEvent {
        event_id: Uuid::new_v4().to_string(),
        timestamp: Utc::now(),
        user_id: Some(user_id),
        client_ip,
        event_type: AuditEventType::Api,
        resource: extract_resource_from_request(&request),
        action: extract_action_from_request(&request),
        outcome,
        details: json!({
            "duration_ms": duration.as_millis(),
            "request_size": request.get_ref().len(),
        }),
    };

    AUDIT_LOGGER.log_event(audit_event).await?;

    result
}
```

#### Audit Log Retention

```bash
#!/bin/bash
# Audit log retention and archival script

RETENTION_YEARS=7
AUDIT_LOG_DIR="/var/log/foxhunt/audit"
ARCHIVE_DIR="/archive/audit"

# Find logs older than retention period
find "$AUDIT_LOG_DIR" -name "*.log" -mtime +$((RETENTION_YEARS * 365)) -print0 | \
while IFS= read -r -d '' log_file; do
    echo "Archiving old audit log: $log_file"

    # Compress and encrypt
    gzip "$log_file"
    gpg --cipher-algo AES256 --compress-algo 1 --symmetric \
        --output "${log_file}.gz.gpg" "${log_file}.gz"

    # Move to archive
    mkdir -p "$ARCHIVE_DIR/$(date -r "$log_file" +%Y/%m)"
    mv "${log_file}.gz.gpg" "$ARCHIVE_DIR/$(date -r "$log_file" +%Y/%m)/"

    # Remove original
    rm -f "${log_file}.gz"
done

# Verify archive integrity
find "$ARCHIVE_DIR" -name "*.gpg" -exec gpg --list-packets {} \; > /dev/null
```

---

## Security Monitoring

### Real-time Monitoring

#### Security Event Detection

```rust
use tokio::sync::mpsc;
use tracing::{info, warn, error};

pub struct SecurityMonitor {
    alert_sender: mpsc::Sender<SecurityAlert>,
    thresholds: SecurityThresholds,
    state: MonitoringState,
}

impl SecurityMonitor {
    pub async fn monitor_authentication_events(&self) {
        let mut failed_attempts = HashMap::new();
        let mut event_stream = self.get_auth_event_stream().await;

        while let Some(event) = event_stream.next().await {
            match event.event_type {
                AuthEventType::FailedLogin => {
                    let count = failed_attempts.entry(event.client_ip.clone())
                        .and_modify(|c| *c += 1)
                        .or_insert(1);

                    if *count >= self.thresholds.max_failed_attempts {
                        let alert = SecurityAlert {
                            severity: AlertSeverity::High,
                            alert_type: AlertType::BruteForceAttempt,
                            source_ip: event.client_ip.clone(),
                            description: format!(
                                "Brute force attempt detected from {} ({} failed attempts)",
                                event.client_ip, count
                            ),
                            recommended_action: "Block IP address".to_string(),
                        };

                        self.alert_sender.send(alert).await.ok();
                    }
                }
                AuthEventType::SuccessfulLogin => {
                    // Clear failed attempt counter on successful login
                    failed_attempts.remove(&event.client_ip);
                }
                _ => {}
            }
        }
    }

    pub async fn monitor_trading_anomalies(&self) {
        let mut trading_stream = self.get_trading_event_stream().await;

        while let Some(event) = trading_stream.next().await {
            // Detect unusual trading patterns
            if self.is_unusual_trading_pattern(&event).await {
                let alert = SecurityAlert {
                    severity: AlertSeverity::Medium,
                    alert_type: AlertType::UnusualTradingActivity,
                    source_ip: event.client_ip.clone(),
                    description: format!(
                        "Unusual trading pattern detected for user {} - {} orders in {}",
                        event.user_id, event.order_count, event.time_window
                    ),
                    recommended_action: "Review user activity".to_string(),
                };

                self.alert_sender.send(alert).await.ok();
            }
        }
    }
}
```

#### Automated Response

```bash
#!/bin/bash
# Automated security response script
set -e

ALERT_TYPE="$1"
SOURCE_IP="$2"
USER_ID="$3"
SEVERITY="$4"

echo "Processing security alert: $ALERT_TYPE from $SOURCE_IP"

case "$ALERT_TYPE" in
    "brute_force")
        echo "Brute force attack detected - blocking IP $SOURCE_IP"

        # Block IP immediately
        iptables -A INPUT -s "$SOURCE_IP" -j DROP

        # Add to permanent blocklist
        echo "$SOURCE_IP" >> /etc/foxhunt/security/blocked_ips.txt

        # Notify security team
        send_security_alert "Brute force attack blocked" "$SOURCE_IP" "high"
        ;;

    "unusual_trading")
        echo "Unusual trading activity detected for user $USER_ID"

        # Temporarily restrict user
        cargo run --bin tli-admin -- restrict-user \
            --user-id "$USER_ID" \
            --duration "1h" \
            --reason "Unusual trading activity"

        # Require additional verification
        cargo run --bin tli-admin -- require-mfa \
            --user-id "$USER_ID" \
            --next-login

        # Alert risk management
        send_risk_alert "Unusual trading activity" "$USER_ID" "medium"
        ;;

    "privilege_escalation")
        echo "Privilege escalation attempt detected"

        # Revoke all sessions for user
        cargo run --bin tli-admin -- revoke-user-sessions \
            --user-id "$USER_ID"

        # Lock account
        cargo run --bin tli-admin -- lock-account \
            --user-id "$USER_ID" \
            --reason "Security incident"

        # Immediate security team notification
        send_security_alert "Privilege escalation attempt" "$USER_ID" "critical"
        ;;
esac

# Log response action
logger -t foxhunt-security "Automated response executed for $ALERT_TYPE: $SOURCE_IP/$USER_ID"
```

### Vulnerability Management

#### Security Scanning

```bash
#!/bin/bash
# Automated security scanning script

SCAN_DATE=$(date +%Y%m%d)
REPORT_DIR="/var/log/security/scans/$SCAN_DATE"
mkdir -p "$REPORT_DIR"

echo "Starting security scan for $SCAN_DATE"

# Dependency vulnerability scanning
cargo audit --format json > "$REPORT_DIR/dependency_audit.json"

# Container image scanning
if command -v trivy &> /dev/null; then
    trivy image --format json --output "$REPORT_DIR/container_scan.json" \
        foxhunt/trading-service:latest
fi

# Network service scanning
nmap -sS -O -sV --script vuln localhost > "$REPORT_DIR/network_scan.txt"

# Configuration security check
/opt/foxhunt/scripts/security-config-check.sh > "$REPORT_DIR/config_check.txt"

# TLS configuration test
testssl.sh --jsonfile "$REPORT_DIR/tls_scan.json" localhost:50051

# Generate summary report
python3 /opt/foxhunt/scripts/generate-security-report.py \
    --input-dir "$REPORT_DIR" \
    --output "$REPORT_DIR/security_summary.html"

echo "Security scan completed. Report available at: $REPORT_DIR/security_summary.html"
```

#### Patch Management

```bash
#!/bin/bash
# Security patch management script

# Check for security updates
apt list --upgradable 2>/dev/null | grep -i security > /tmp/security_updates.txt

if [ -s /tmp/security_updates.txt ]; then
    echo "Security updates available:"
    cat /tmp/security_updates.txt

    # Create system snapshot before patching
    lvcreate -L 10G -s -n system-snapshot-$(date +%Y%m%d) /dev/vg0/root

    # Apply security updates
    apt update
    apt upgrade -y $(grep "security" /tmp/security_updates.txt | cut -d'/' -f1)

    # Restart affected services
    systemctl restart foxhunt-trading-service
    systemctl restart foxhunt-backtesting-service

    # Verify system functionality
    /opt/foxhunt/scripts/post-patch-verification.sh

    if [ $? -eq 0 ]; then
        echo "Patching completed successfully"
        # Remove snapshot after successful verification
        lvremove -f /dev/vg0/system-snapshot-$(date +%Y%m%d)
    else
        echo "Patching failed - rolling back"
        # Rollback to snapshot
        lvconvert --merge /dev/vg0/system-snapshot-$(date +%Y%m%d)
        reboot
    fi
else
    echo "No security updates available"
fi
```

---

*This security documentation is maintained by the Security Team. For security incidents or questions, contact: security@company.com*

*Classification: Security Sensitive - Authorized Personnel Only*