foxhunt/vault-migration/vault-structure.md

HashiCorp Vault Secret Migration Plan for Foxhunt HFT

Vault Organizational Structure

Based on comprehensive codebase analysis identifying 200+ secret references across 5 categories, here's the proposed Vault structure:

Secret Engine Mount Points

/secret/
├── foxhunt/
│   ├── database/
│   ├── api-keys/
│   ├── authentication/
│   ├── brokers/
│   └── certificates/

Detailed Path Structure

1. Database Secrets (/secret/foxhunt/database/)

/secret/foxhunt/database/postgresql
  - url: "postgresql://user:pass@host:port/db"
  - username: "foxhunt_user"
  - password: "secure_password"
  - ssl_mode: "require"

/secret/foxhunt/database/redis
  - url: "redis://user:pass@host:port"
  - password: "redis_password"
  - max_connections: 100

/secret/foxhunt/database/influxdb
  - url: "http://host:8086"
  - token: "influx_token"
  - org: "foxhunt"
  - bucket: "trading_data"

/secret/foxhunt/database/clickhouse
  - url: "http://host:8123"
  - username: "default"
  - password: "clickhouse_password"
  - database: "foxhunt_analytics"

2. API Keys (/secret/foxhunt/api-keys/)

/secret/foxhunt/api-keys/databento
  - api_key: "databento_production_key"
  - endpoint: "wss://gateway.databento.com/v2"
  - rate_limit: 10

/secret/foxhunt/api-keys/benzinga
  - api_key: "benzinga_production_key"
  - endpoint: "wss://api.benzinga.com/api/v1/news/stream"
  - rate_limit: 5

/secret/foxhunt/api-keys/alpha-vantage
  - api_key: "alpha_vantage_key"
  - endpoint: "https://www.alphavantage.co"
  - rate_limit: 1

3. Authentication (/secret/foxhunt/authentication/)

/secret/foxhunt/authentication/jwt
  - secret: "jwt_signing_key_32_chars_min"
  - issuer: "foxhunt-hft"
  - audience: "foxhunt-services"
  - expiration_seconds: 3600

/secret/foxhunt/authentication/encryption
  - primary_key: "aes_256_encryption_key"
  - key_derivation_salt: "random_32_byte_salt"
  - algorithm: "AES256-GCM"

4. Broker Credentials (/secret/foxhunt/brokers/)

/secret/foxhunt/brokers/icmarkets
  - username: "foxhunt_user"
  - password: "icmarkets_password"
  - sender_comp_id: "FOXHUNT"
  - target_comp_id: "ICMARKETS"
  - endpoint: "fix.icmarkets.com:443"

/secret/foxhunt/brokers/interactive-brokers
  - host: "localhost"
  - port: 7497
  - client_id: 1
  - account_id: "DU123456"

5. TLS Certificates (/secret/foxhunt/certificates/)

/secret/foxhunt/certificates/trading-service
  - certificate: "-----BEGIN CERTIFICATE-----..."
  - private_key: "-----BEGIN PRIVATE KEY-----..."
  - ca_certificate: "-----BEGIN CERTIFICATE-----..."

/secret/foxhunt/certificates/client
  - certificate: "-----BEGIN CERTIFICATE-----..."
  - private_key: "-----BEGIN PRIVATE KEY-----..."

Environment-Specific Paths

Each environment gets its own namespace:

/secret/foxhunt/development/...
/secret/foxhunt/staging/...
/secret/foxhunt/production/...

Secret Classification

Critical Secrets (Rotation: 30 days)

• Database passwords
• Broker credentials
• JWT signing keys
• Encryption keys

Standard Secrets (Rotation: 90 days)

• API keys
• TLS private keys

Reference Secrets (Rotation: 365 days)

• Configuration parameters
• Public certificates

Security Considerations

Access Control Policies

1. Service-Level Access

   • trading-service: Read access to all secrets
   • tli-service: Limited read access (no broker credentials)
   • ml-service: Read access to database, API keys only

2. Environment Isolation

   • Production secrets isolated from dev/staging
   • Cross-environment access prohibited

3. Human Access

   • Admin: Full access with audit logging
   • Developer: Development environment only
   • Operator: Read-only production access for troubleshooting

Vault Configuration Requirements

1. Authentication Methods

   • Kubernetes Service Accounts (recommended)
   • AppRole for standalone deployments
   • LDAP/OIDC for human access

2. Secret Engines

   • KV v2 for static secrets
   • Database engine for dynamic database credentials
   • PKI engine for certificate management

3. Audit and Compliance

   • All secret access logged
   • Failed access attempts alerted
   • Regular access reviews

Migration Strategy

Phase 1: Vault Setup and Core Secrets

1. Install and configure Vault cluster
2. Create secret engine mount points
3. Migrate database credentials
4. Migrate JWT/encryption keys

Phase 2: API Keys and Broker Credentials

1. Migrate market data API keys
2. Migrate broker credentials
3. Update configuration loading code

Phase 3: Certificate Management

1. Migrate TLS certificates to Vault
2. Implement certificate rotation
3. Update service startup scripts

Phase 4: Dynamic Secrets

1. Configure database secret engine
2. Implement dynamic database credentials
3. Add secret rotation automation

Implementation Files

This migration requires:

1. vault-client/ - Vault integration library
2. migration-scripts/ - Secret population scripts
3. config-updates/ - Updated configuration files
4. deployment/ - Vault deployment manifests
5. documentation/ - Migration procedures and runbooks