foxhunt/vault-migration/MIGRATION_GUIDE.md

Foxhunt HFT System - HashiCorp Vault Migration Guide

Overview

This guide provides comprehensive instructions for migrating the Foxhunt HFT trading system from environment variable-based secret management to HashiCorp Vault. The migration ensures secure, centralized, and automated secret management with zero-downtime deployment.

Pre-Migration Assessment

Current Secret Inventory

Based on comprehensive codebase analysis, the following secrets have been identified:

Database Secrets (100+ references)

• PRIMARY: DATABASE_URL - PostgreSQL connection for trading data
• CACHE: REDIS_URL - Redis for caching and sessions
• ANALYTICS: INFLUX_URL, INFLUX_TOKEN, INFLUX_ORG, INFLUX_BUCKET - InfluxDB time-series
• WAREHOUSE: CLICKHOUSE_URL, CLICKHOUSE_USER, CLICKHOUSE_PASSWORD, CLICKHOUSE_DB - Analytics warehouse
• TEST: TEST_DATABASE_URL - Test environment databases

API Keys (37+ references)

• MARKET DATA: DATABENTO_API_KEY - Primary market data provider (22 refs)
• NEWS/SENTIMENT: BENZINGA_API_KEY - Financial news and sentiment (15 refs)
• BACKUP: ALPHA_VANTAGE_API_KEY - Fallback market data provider

Authentication & Security (30+ references)

• JWT: JWT_SECRET, FOXHUNT_JWT_SECRET - JWT signing keys
• ENCRYPTION: FOXHUNT_ENCRYPTION_KEY - Application-level encryption
• TLS: Private keys and certificates for mTLS

Broker Integration (25+ references)

• ICMARKETS: ICMARKETS_USERNAME, ICMARKETS_PASSWORD - FIX protocol trading
• INTERACTIVE BROKERS: IB_HOST, IB_PORT, IB_CLIENT_ID, IB_ACCOUNT_ID - TWS API
• FIX PROTOCOL: Sender/Target CompIDs, session credentials

Risk Assessment

Critical Risks Identified

1. Hardcoded fallbacks with demo/placeholder values in 200+ locations
2. No secret rotation capabilities in current implementation
3. Plaintext secrets in configuration files and test environments
4. Inconsistent secret loading patterns across services

Security Improvements with Vault

1. Centralized secret management with access control policies
2. Automatic secret rotation with configurable policies
3. Audit logging for all secret access
4. Dynamic secrets for database credentials
5. Encrypted storage with configurable key management

Migration Strategy

Phase 1: Infrastructure Setup (Week 1)

1.1 Vault Cluster Deployment

Production Environment:

# Deploy Vault cluster (3-node HA setup)
helm install vault hashicorp/vault \
  --set server.ha.enabled=true \
  --set server.ha.replicas=3 \
  --set server.dataStorage.size=10Gi \
  --set server.auditStorage.enabled=true

Development/Staging:

# Single-node development setup
helm install vault-dev hashicorp/vault \
  --set server.dev.enabled=true \
  --set server.dataStorage.size=1Gi

1.2 Vault Initialization

# Initialize Vault
vault operator init -key-shares=5 -key-threshold=3

# Unseal Vault (repeat with 3 different keys)
vault operator unseal <key1>
vault operator unseal <key2>
vault operator unseal <key3>

# Authenticate with root token
vault auth <root-token>

1.3 Secret Engine Setup

# Enable KV v2 secret engine
vault secrets enable -version=2 kv

# Enable database secret engine for dynamic credentials
vault secrets enable database

# Enable PKI for certificate management
vault secrets enable pki

1.4 Authentication Setup

Kubernetes Service Accounts (Recommended):

# Enable Kubernetes auth
vault auth enable kubernetes

# Configure Kubernetes auth
vault write auth/kubernetes/config \
    token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
    kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
    kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

AppRole for Standalone Deployments:

# Enable AppRole auth
vault auth enable approle

# Create AppRole for trading service
vault write auth/approle/role/trading-service \
    token_policies="trading-service-policy" \
    token_ttl=1h \
    token_max_ttl=4h

Phase 2: Secret Migration (Week 2)

2.1 Create Access Policies

# Trading service policy
vault policy write trading-service-policy - <<EOF
# Database secrets
path "secret/data/foxhunt/*/trading-service/database/*" {
  capabilities = ["read"]
}

# API keys
path "secret/data/foxhunt/*/trading-service/api-keys/*" {
  capabilities = ["read"]
}

# Authentication secrets
path "secret/data/foxhunt/*/trading-service/authentication/*" {
  capabilities = ["read"]
}

# Broker credentials
path "secret/data/foxhunt/*/trading-service/brokers/*" {
  capabilities = ["read"]
}

# Certificate management
path "secret/data/foxhunt/*/trading-service/certificates/*" {
  capabilities = ["read"]
}
EOF

2.2 Populate Secrets

Using Migration Script:

# Set environment variables for current secrets
export DATABASE_URL="postgresql://..."
export DATABENTO_API_KEY="..."
export BENZINGA_API_KEY="..."
export JWT_SECRET="..."
export ICMARKETS_USERNAME="..."
export ICMARKETS_PASSWORD="..."

# Run migration script
cargo run --bin populate-vault -- \
    --vault-url https://vault.foxhunt.local:8200 \
    --environment production \
    --service trading-service \
    --auth-method kubernetes \
    --k8s-role trading-service

# Verify secrets were populated
vault kv list secret/foxhunt/production/trading-service

Manual Population (if needed):

# Database secrets
vault kv put secret/foxhunt/production/trading-service/database/postgresql \
    url="postgresql://user:pass@host:5432/foxhunt" \
    host="postgres.foxhunt.local" \
    port=5432 \
    database="foxhunt" \
    username="foxhunt_user"

# API keys
vault kv put secret/foxhunt/production/trading-service/api-keys/databento \
    api_key="db_live_..." \
    endpoint="wss://gateway.databento.com/v2" \
    rate_limit=10

vault kv put secret/foxhunt/production/trading-service/api-keys/benzinga \
    api_key="bz_..." \
    endpoint="wss://api.benzinga.com/api/v1/news/stream" \
    rate_limit=5

2.3 Configure Dynamic Database Secrets

# Configure PostgreSQL connection
vault write database/config/foxhunt-postgres \
    plugin_name=postgresql-database-plugin \
    connection_url="postgresql://{{username}}:{{password}}@postgres.foxhunt.local:5432/foxhunt" \
    allowed_roles="foxhunt-role" \
    username="vault-admin" \
    password="admin-password"

# Create dynamic role
vault write database/roles/foxhunt-role \
    db_name=foxhunt-postgres \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
        GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    default_ttl="1h" \
    max_ttl="24h"

Phase 3: Service Integration (Week 3)

3.1 Update Service Configurations

Trading Service Integration:

// services/trading_service/src/main.rs
use vault_migration::VaultConfigLoader;
use foxhunt_vault_client::{FoxhuntVaultClient, VaultClientConfig};
use foxhunt_vault_client::auth::KubernetesTokenProvider;

#[tokio::main]
async fn main() -> Result<()> {
    // Initialize Vault client
    let vault_config = VaultClientConfig {
        vault_url: std::env::var("VAULT_URL")
            .unwrap_or_else(|_| "https://vault.foxhunt.local:8200".to_string()),
        environment: std::env::var("ENVIRONMENT")
            .unwrap_or_else(|_| "production".to_string()),
        service_name: "trading-service".to_string(),
        ..Default::default()
    };

    let token_provider = Box::new(KubernetesTokenProvider::new(
        url::Url::parse(&vault_config.vault_url)?,
        reqwest::Client::new(),
        "trading-service".to_string(),
        None,
    ));

    let vault_client = Arc::new(FoxhuntVaultClient::new(vault_config, token_provider).await?);
    
    // Initialize Vault-integrated config loader
    let config_loader = VaultConfigLoader::new(
        vault_client,
        "production".to_string(),
        "trading-service".to_string(),
        true, // Enable fallback to environment variables during transition
    ).await?;

    // Load database configuration from Vault
    let db_config = config_loader.get_database_config("postgresql").await?;
    let pool = sqlx::PgPool::connect(&db_config.url).await?;

    // Load API keys from Vault
    let databento_config = config_loader.get_api_key_config("databento").await?;
    let benzinga_config = config_loader.get_api_key_config("benzinga").await?;

    // Initialize services with Vault-loaded configurations
    // ... rest of service initialization
}

Configuration Module Updates:

// core/src/config/mod.rs - Replace environment variable loading

impl Default for ExternalApiConfig {
    fn default() -> Self {
        // Initialize with Vault loader instead of env::var
        let vault_loader = get_vault_loader(); // Global vault loader instance
        
        Self {
            databento: vault_loader.get_api_key_config("databento").await.ok(),
            benzinga: vault_loader.get_api_key_config("benzinga").await.ok(),
            // ... other configurations
        }
    }
}

3.2 Update Docker Images

Dockerfile Updates:

# Add Vault client dependencies
FROM rust:1.70 as builder
WORKDIR /app
COPY vault-migration/ vault-migration/
COPY services/ services/
RUN cargo build --release --package foxhunt-vault-client
RUN cargo build --release --bin trading_service

FROM debian:bullseye-slim
# Install ca-certificates for HTTPS connections to Vault
RUN apt-get update && apt-get install -y ca-certificates && rm -rf /var/lib/apt/lists/*
COPY --from=builder /app/target/release/trading_service /usr/local/bin/
EXPOSE 50051 8080
CMD ["trading_service"]

Kubernetes Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: trading-service
spec:
  template:
    spec:
      serviceAccountName: trading-service-vault
      containers:
      - name: trading-service
        image: foxhunt/trading-service:vault-migration
        env:
        - name: VAULT_URL
          value: "https://vault.foxhunt.local:8200"
        - name: ENVIRONMENT
          value: "production"
        # Remove old environment variables
        # - name: DATABASE_URL  # Now loaded from Vault
        # - name: DATABENTO_API_KEY  # Now loaded from Vault
        volumeMounts:
        - name: vault-token
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          readOnly: true
      volumes:
      - name: vault-token
        projected:
          sources:
          - serviceAccountToken:
              path: token
              audience: vault

3.3 Gradual Rollout Strategy

Blue-Green Deployment:

# Phase 1: Deploy with fallback enabled
kubectl set env deployment/trading-service VAULT_FALLBACK_ENABLED=true
kubectl rollout restart deployment/trading-service
kubectl rollout status deployment/trading-service

# Phase 2: Verify Vault integration works
kubectl logs -l app=trading-service | grep "Vault"

# Phase 3: Disable fallback after verification
kubectl set env deployment/trading-service VAULT_FALLBACK_ENABLED=false
kubectl rollout restart deployment/trading-service

# Phase 4: Remove environment variables
kubectl patch deployment trading-service -p '{"spec":{"template":{"spec":{"containers":[{"name":"trading-service","env":[]}]}}}}'

Phase 4: Secret Rotation Setup (Week 4)

4.1 Configure Rotation Policies

JWT Signing Keys (30-day rotation):

use vault_migration::SecretRotationManager;
use chrono::Duration as ChronoDuration;

let jwt_policy = RotationPolicy {
    secret_path: "authentication/jwt".to_string(),
    rotation_type: RotationType::JwtSigningKey,
    rotation_interval: ChronoDuration::days(30),
    advance_notice_hours: 24,
    max_versions: 5,
    enable_automatic_rotation: true,
    require_manual_approval: false,
    pre_rotation_hooks: vec![],
    post_rotation_hooks: vec!["restart-services".to_string()],
    rollback_strategy: RollbackStrategy::GracefulFallback { fallback_hours: 2 },
};

rotation_manager.set_rotation_policy(jwt_policy).await?;

API Keys (90-day rotation):

let databento_policy = RotationPolicy {
    secret_path: "api-keys/databento".to_string(),
    rotation_type: RotationType::ApiKey {
        provider_config: ApiKeyProviderConfig {
            provider: "databento".to_string(),
            api_endpoint: "https://api.databento.com".to_string(),
            auth_method: "bearer".to_string(),
            rotation_endpoint: Some("https://api.databento.com/keys/rotate".to_string()),
        },
    },
    rotation_interval: ChronoDuration::days(90),
    advance_notice_hours: 72, // 3 days notice
    max_versions: 3,
    enable_automatic_rotation: true,
    require_manual_approval: true, // API keys require approval
    pre_rotation_hooks: vec!["notify-team".to_string()],
    post_rotation_hooks: vec!["validate-connectivity".to_string()],
    rollback_strategy: RollbackStrategy::GracefulFallback { fallback_hours: 24 },
};

rotation_manager.set_rotation_policy(databento_policy).await?;

4.2 Automation Setup

Kubernetes CronJob for Rotation:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: vault-rotation-scheduler
spec:
  schedule: "0 2 * * *"  # Run daily at 2 AM
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: vault-rotation-service
          containers:
          - name: rotation-scheduler
            image: foxhunt/vault-rotation:latest
            command: ["vault-rotation-scheduler"]
            args: ["--check-and-schedule"]
            env:
            - name: VAULT_URL
              value: "https://vault.foxhunt.local:8200"
          restartPolicy: OnFailure

Monitoring and Alerting:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: vault-rotation-alerts
spec:
  groups:
  - name: vault.rotation
    rules:
    - alert: VaultRotationFailed
      expr: vault_rotation_failures_total > 0
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "Vault secret rotation failed"
        description: "Secret rotation failed for {{ $labels.secret_path }}"
    
    - alert: VaultRotationOverdue
      expr: vault_rotation_overdue_seconds > 86400
      for: 1h
      labels:
        severity: warning
      annotations:
        summary: "Secret rotation overdue"
        description: "Secret {{ $labels.secret_path }} is overdue for rotation by {{ $value }} seconds"

Verification & Testing

4.1 Pre-Migration Testing

Test Script:

#!/bin/bash
set -e

echo "🧪 Running pre-migration tests..."

# Test current environment variable loading
export DATABASE_URL="postgresql://test:test@localhost:5432/test"
export DATABENTO_API_KEY="test-key"

# Run integration tests
cargo test --package core --test config_tests
cargo test --package trading_service --test integration_tests

echo "✅ Pre-migration tests passed"

4.2 Post-Migration Verification

Vault Integration Test:

#[tokio::test]
async fn test_vault_integration() -> Result<()> {
    let config_loader = setup_test_vault_loader().await?;
    
    // Test database config loading
    let db_config = config_loader.get_database_config("postgresql").await?;
    assert!(!db_config.url.is_empty());
    
    // Test API key loading
    let api_config = config_loader.get_api_key_config("databento").await?;
    assert!(!api_config.api_key.is_empty());
    
    // Test JWT config loading
    let jwt_config = config_loader.get_jwt_config().await?;
    assert!(jwt_config.secret.len() >= 32);
    
    Ok(())
}

End-to-End Service Test:

#!/bin/bash
set -e

echo "🔍 Running post-migration verification..."

# Check Vault connectivity
vault status

# Verify secrets are accessible
vault kv get secret/foxhunt/production/trading-service/database/postgresql

# Test service startup with Vault
kubectl scale deployment trading-service --replicas=1
kubectl wait --for=condition=Ready pod -l app=trading-service --timeout=300s

# Run health checks
kubectl exec deployment/trading-service -- /usr/local/bin/health-check

# Run trading system smoke tests
cargo test --package tests --test smoke_tests --features vault-integration

echo "✅ Post-migration verification complete"

Rollback Plan

Emergency Rollback Procedure

Step 1: Immediate Revert (< 5 minutes)

# Revert to previous deployment with environment variables
kubectl rollout undo deployment/trading-service

# Verify rollback
kubectl rollout status deployment/trading-service
kubectl get pods -l app=trading-service

Step 2: Re-enable Environment Variables

# Restore environment variables from backup
kubectl apply -f backup/trading-service-env-vars.yaml

# Restart services
kubectl rollout restart deployment/trading-service

Step 3: Validate System Recovery

# Run health checks
./scripts/health-check.sh

# Verify trading operations
./scripts/trading-smoke-test.sh

Security Considerations

Access Control

Principle of Least Privilege:

• Each service has access only to its required secrets
• Environment-specific isolation (dev/staging/prod)
• Time-limited tokens with automatic renewal

Policy Examples:

# Development environment - broader access for debugging
path "secret/data/foxhunt/development/*" {
  capabilities = ["read", "list"]
}

# Production environment - strict role-based access
path "secret/data/foxhunt/production/trading-service/database/*" {
  capabilities = ["read"]
}

# No access to other services' secrets
path "secret/data/foxhunt/production/ml-service/*" {
  capabilities = ["deny"]
}

Audit and Compliance

Audit Logging:

# Enable audit logging
vault audit enable file file_path=/vault/logs/audit.log

# Monitor secret access
tail -f /vault/logs/audit.log | jq '.request.path' | grep "secret/data/foxhunt"

Compliance Reports:

# Generate monthly access report
vault-audit-analyzer --start-date 2025-01-01 --end-date 2025-01-31 \
    --output compliance-report-2025-01.json

# Check for unauthorized access attempts
vault-audit-analyzer --filter failed-requests --last 24h

Performance Impact Assessment

Latency Analysis

Before Migration (Environment Variables):

• Secret loading: ~0.1ms (cached in memory)
• Service startup: ~2-3 seconds

After Migration (Vault Integration):

• Initial secret loading: ~50-100ms (network + auth)
• Cached secret access: ~0.1ms (same as before)
• Service startup: ~3-4 seconds (+1 second for Vault auth)

Mitigation Strategies:

1. Aggressive Caching: 5-minute cache TTL for non-critical secrets
2. Connection Pooling: Reuse HTTP connections to Vault
3. Background Refresh: Proactively refresh secrets before expiration
4. Health Checks: Monitor Vault connectivity and fallback gracefully

Resource Usage

Additional Memory Usage:

• Vault client library: ~5MB
• Secret cache: ~1MB per service
• Total overhead: <10MB per service

Network Traffic:

• Initial auth: ~1KB
• Secret reads: ~2KB per secret
• Token refresh: ~1KB every hour
• Total: <100KB/hour per service

Monitoring & Maintenance

Key Metrics to Monitor

Vault Health:

vault_up: 1  # Vault cluster availability
vault_sealed: 0  # Vault seal status
vault_leader: 1  # Leader election status

Secret Access:

vault_secret_requests_total  # Total secret requests
vault_secret_request_duration_seconds  # Request latency
vault_secret_cache_hits_total  # Cache hit rate
vault_secret_errors_total  # Error rate

Rotation Status:

vault_rotation_scheduled_total  # Scheduled rotations
vault_rotation_completed_total  # Completed rotations
vault_rotation_failed_total  # Failed rotations
vault_rotation_overdue_total  # Overdue rotations

Maintenance Procedures

Weekly Tasks:

• Review audit logs for unauthorized access
• Check rotation schedule for upcoming secret changes
• Verify backup and disaster recovery procedures

Monthly Tasks:

• Rotate Vault root tokens
• Review and update access policies
• Conduct security audit of secret access patterns
• Update rotation policies based on usage patterns

Quarterly Tasks:

• Full disaster recovery test
• Security penetration testing
• Performance optimization review
• Update documentation and runbooks

Troubleshooting Guide

Common Issues

Issue 1: Service Cannot Connect to Vault

# Check Vault status
vault status

# Verify network connectivity
curl -k https://vault.foxhunt.local:8200/v1/sys/health

# Check service account token
kubectl describe pod -l app=trading-service | grep -A5 "vault-token"

# Test authentication manually
vault auth -method=kubernetes role=trading-service jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"

Issue 2: Secret Not Found

# Verify secret exists
vault kv list secret/foxhunt/production/trading-service

# Check access permissions
vault token capabilities secret/data/foxhunt/production/trading-service/database/postgresql

# Review audit logs
vault audit-reader /vault/logs/audit.log | grep "secret/data/foxhunt"

Issue 3: Token Expired

# Check token status
vault token lookup

# Refresh token
vault token renew

# Check renewal policy
vault read auth/kubernetes/role/trading-service

Issue 4: Rotation Failed

# Check rotation logs
kubectl logs -l app=vault-rotation-scheduler

# Review rotation policy
vault read secret/metadata/foxhunt/production/trading-service/authentication/jwt

# Manual rotation trigger
cargo run --bin manual-rotation -- --secret-path authentication/jwt

Migration Timeline

Week 1: Infrastructure Setup

• Day 1-2: Deploy Vault cluster and basic configuration
• Day 3-4: Set up authentication methods and access policies
• Day 5: Configure secret engines and initial testing

Week 2: Secret Population

• Day 1-2: Run migration scripts and populate all secrets
• Day 3-4: Set up dynamic secret engines for databases
• Day 5: Comprehensive testing and validation

Week 3: Service Integration

• Day 1-2: Update service code and build new images
• Day 3-4: Deploy to staging and run integration tests
• Day 5: Production deployment with fallback enabled

Week 4: Rotation & Cleanup

• Day 1-2: Configure rotation policies and automation
• Day 3-4: Remove environment variables and test full Vault integration
• Day 5: Final verification and documentation updates

Success Criteria

Technical Metrics

• ✅ All 200+ secret references migrated to Vault
• ✅ Zero-downtime deployment achieved
• ✅ Service startup time increase < 2 seconds
• ✅ Secret access latency < 100ms (95th percentile)
• ✅ Automatic rotation working for all critical secrets

Security Improvements

• ✅ All secrets encrypted at rest in Vault
• ✅ Audit logging enabled for all secret access
• ✅ Access control policies enforce least privilege
• ✅ Dynamic credentials for database access
• ✅ Secret rotation policies in place

Operational Benefits

• ✅ Centralized secret management dashboard
• ✅ Automated rotation reduces manual overhead
• ✅ Improved incident response with audit trails
• ✅ Simplified secret distribution for new services
• ✅ Enhanced compliance with security standards

Support & Resources

Documentation Links

• HashiCorp Vault Documentation
• Kubernetes Vault Integration
• Vault API Reference

Internal Resources

• Foxhunt Vault Dashboard: https://vault.foxhunt.local:8200/ui
• Rotation Management UI: https://rotation.foxhunt.local
• Monitoring Dashboards: https://grafana.foxhunt.local/d/vault-secrets

Emergency Contacts

• Platform Team: platform@foxhunt.io
• Security Team: security@foxhunt.io
• On-Call Engineer: +1-555-FOXHUNT

---

This migration guide ensures secure, reliable, and maintainable secret management for the Foxhunt HFT trading system. Follow all procedures carefully and test thoroughly in non-production environments before applying to production systems.