foxhunt/vault-migration/ROTATION_GUIDE.md

Foxhunt Secret Rotation System Guide

Overview

The Foxhunt Secret Rotation System provides automated, policy-driven secret rotation with comprehensive monitoring, notifications, and compliance tracking. The system is built on HashiCorp Vault and provides enterprise-grade security for managing sensitive credentials.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    Foxhunt Secret Rotation System               │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  ┌─────────────────┐    ┌──────────────────┐    ┌─────────────┐ │
│  │ Rotation        │    │ Notification     │    │ Vault       │ │
│  │ Scheduler       │◄──►│ Handler          │◄──►│ Client      │ │
│  │                 │    │                  │    │             │ │
│  └─────────────────┘    └──────────────────┘    └─────────────┘ │
│           │                       │                     │       │
│           ▼                       ▼                     ▼       │
│  ┌─────────────────┐    ┌──────────────────┐    ┌─────────────┐ │
│  │ Policy Engine   │    │ Alert System     │    │ HashiCorp   │ │
│  │                 │    │                  │    │ Vault       │ │
│  └─────────────────┘    └──────────────────┘    └─────────────┘ │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Features

🔄 Automated Rotation

• Policy-based rotation with configurable intervals
• Multiple secret types: API keys, passwords, JWT secrets, encryption keys
• Flexible scheduling with rotation windows
• Retry logic with exponential backoff
• Concurrent rotation with configurable limits

📊 Monitoring & Alerts

• Real-time notifications via Slack, email, webhooks
• Comprehensive audit trail with detailed event logging
• Alert management with acknowledgment system
• Overdue rotation detection with escalation
• Performance metrics and health monitoring

🛡️ Security & Compliance

• Zero-downtime rotations with graceful fallback
• Compliance tracking with regulatory tags
• Secure token management with automatic renewal
• Encrypted communication with Vault
• Role-based access control with AppRole authentication

🎯 Enterprise Features

• Multi-environment support (production, staging, development)
• High availability with multiple instances
• Disaster recovery with backup policies
• Integration APIs for external systems
• Extensible architecture for custom secret types

Secret Types and Policies

1. Database Passwords

Policy: database_password
Rotation Interval: 7 days
Window: 2 hours
Complexity: 24 chars, symbols, no ambiguous chars
Compliance: PCI-DSS, SOX
Notifications: Slack, Email, System Log

2. API Keys

Policy: api_key
Rotation Interval: 30 days
Window: 6 hours  
Complexity: 32 chars, alphanumeric
Compliance: Standard
Notifications: Slack, System Log (failures only)

3. JWT Secrets

Policy: jwt_secret
Rotation Interval: 1 day
Window: 1 hour
Complexity: 64 chars, base64 encoded
Compliance: SOX, GDPR
Notifications: Slack, Webhook, System Log

4. Encryption Keys

Policy: encryption_key
Rotation Interval: 90 days
Window: 4 hours
Complexity: 256-bit AES keys
Compliance: PCI-DSS, SOX, FIPS-140-2
Notifications: Email, Slack, Webhook, System Log

Installation and Setup

Prerequisites

1. HashiCorp Vault (v1.12+) running and accessible
2. PostgreSQL for audit logging and configuration
3. Redis (optional) for caching and rate limiting
4. Rust toolchain (1.70+) for building the service

Step 1: Vault Configuration

# Clone the repository
git clone <repository-url>
cd foxhunt/vault-migration

# Set environment variables
export VAULT_ADDR="https://vault.foxhunt.com:8200"
export VAULT_TOKEN="your-vault-token"
export ENVIRONMENT="production"

# Run the setup script
./rotation/setup-rotation.sh

This script will:

• Enable KV v2 secret engine at foxhunt/
• Create rotation policies for all secret types
• Schedule existing secrets for rotation
• Create service configuration and policies
• Generate AppRole credentials for the rotation service

Step 2: Build the Rotation Service

# Build the rotation service
cd vault-migration
cargo build --release --bin rotation-service

# Create deployment directory
sudo mkdir -p /opt/foxhunt/rotation/bin
sudo cp target/release/rotation-service /opt/foxhunt/rotation/bin/

# Create log directory
sudo mkdir -p /var/log/foxhunt
sudo chown foxhunt:foxhunt /var/log/foxhunt

Step 3: Configure Notifications

Update notification settings in Vault:

# Configure Slack notifications
vault kv put foxhunt/rotation/notifications/slack \
    enabled=true \
    webhook_url="https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK" \
    channel="#security-alerts" \
    severity_filter="info"

# Configure email notifications
vault kv put foxhunt/rotation/notifications/email \
    enabled=true \
    smtp_server="smtp.foxhunt.com" \
    smtp_port=587 \
    from_address="vault-rotator@foxhunt.com" \
    to_addresses="security-team@foxhunt.com"

# Configure webhook notifications
vault kv put foxhunt/rotation/notifications/webhook \
    enabled=true \
    endpoint_url="https://api.foxhunt.com/security/rotation-events" \
    auth_token="your-webhook-auth-token"

Step 4: Deploy as SystemD Service

# Copy service file
sudo cp foxhunt-rotation-service.service /etc/systemd/system/

# Copy credentials file
sudo mkdir -p /opt/foxhunt/rotation
sudo cp .rotation-credentials /opt/foxhunt/rotation/
sudo chown -R foxhunt:foxhunt /opt/foxhunt/rotation
sudo chmod 600 /opt/foxhunt/rotation/.rotation-credentials

# Enable and start service
sudo systemctl daemon-reload
sudo systemctl enable foxhunt-rotation-service
sudo systemctl start foxhunt-rotation-service

# Check status
sudo systemctl status foxhunt-rotation-service
sudo journalctl -u foxhunt-rotation-service -f

Configuration

Environment Variables

The rotation service uses the following environment variables:

# Required - from .rotation-credentials file
FOXHUNT_VAULT_ADDR="https://vault.foxhunt.com:8200"
FOXHUNT_VAULT_ROLE_ID="your-role-id"
FOXHUNT_VAULT_SECRET_ID="your-secret-id"
FOXHUNT_ENVIRONMENT="production"

# Optional - for enhanced functionality
DATABASE_URL="postgresql://user:pass@localhost/foxhunt"
REDIS_URL="redis://localhost:6379"
LOG_LEVEL="info"
METRICS_PORT="9090"

Vault Configuration

All configuration is stored in Vault under the foxhunt/rotation/ path:

foxhunt/
├── rotation/
│   ├── config/
│   │   └── service              # Service configuration
│   ├── policies/
│   │   ├── database_password    # Rotation policy definitions
│   │   ├── api_key
│   │   ├── jwt_secret
│   │   └── encryption_key
│   ├── schedules/
│   │   ├── database_foxhunt_db  # Scheduled rotations
│   │   ├── api_databento
│   │   ├── api_benzinga
│   │   ├── jwt_auth
│   │   └── encryption_primary
│   └── notifications/
│       ├── slack                # Notification configurations
│       ├── email
│       └── webhook

Operations Guide

Monitoring Rotations

Check Service Status

# Service status
sudo systemctl status foxhunt-rotation-service

# View logs
sudo journalctl -u foxhunt-rotation-service -f

# Check for errors
sudo journalctl -u foxhunt-rotation-service -p err

View Rotation History

# List recent rotation events
vault kv get foxhunt/rotation/events/recent

# Get specific rotation details
vault kv get foxhunt/rotation/events/2025-01-23/database_foxhunt_db

Active Alerts

# List active alerts
vault kv get foxhunt/rotation/alerts/active

# Acknowledge alert
vault kv put foxhunt/rotation/alerts/ack/alert-id-here \
    acknowledged_by="admin@foxhunt.com" \
    acknowledged_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)"

Manual Rotation

Trigger Manual Rotation

# Force immediate rotation of a specific secret
vault kv put foxhunt/rotation/manual/trigger \
    secret_path="foxhunt/production/database/foxhunt_db" \
    triggered_by="admin@foxhunt.com" \
    reason="Security incident - immediate rotation required"

Emergency Stop

# Stop all rotations immediately
vault kv put foxhunt/rotation/config/emergency_stop \
    enabled=true \
    reason="Emergency maintenance" \
    stopped_by="admin@foxhunt.com"

# Resume rotations
vault kv delete foxhunt/rotation/config/emergency_stop

Policy Management

Update Rotation Policy

# Change rotation interval for API keys
vault kv patch foxhunt/rotation/policies/api_key \
    rotation_interval_days=14 \
    updated_by="admin@foxhunt.com" \
    updated_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)"

Disable Rotation for Specific Secret

# Disable rotation temporarily
vault kv put foxhunt/rotation/schedules/api_databento \
    enabled=false \
    disabled_reason="Maintenance window" \
    disabled_by="admin@foxhunt.com"

Troubleshooting

Common Issues

1. Rotation Failures

   # Check recent failures
   sudo journalctl -u foxhunt-rotation-service | grep "ERROR"
   
   # Review failed rotation details
   vault kv get foxhunt/rotation/failures/recent
   

2. Authentication Issues

   # Check token validity
   vault auth -method=token
   
   # Renew AppRole secret if expired
   vault write auth/approle/role/foxhunt-rotation-service/secret-id
   

3. Notification Problems

   # Test Slack webhook
   curl -X POST \
     -H 'Content-Type: application/json' \
     -d '{"text":"Test message from Foxhunt Rotation Service"}' \
     "YOUR_SLACK_WEBHOOK_URL"
   

4. Performance Issues

   # Check service metrics
   curl http://localhost:9090/metrics
   
   # Review rotation timing
   vault kv get foxhunt/rotation/metrics/performance
   

Security Considerations

Access Control

• Rotation service runs with minimal required permissions
• AppRole authentication with time-limited tokens
• Separate policies for read/write operations
• Audit logging for all Vault operations

Secret Safety

• Secrets are never logged or exposed in plain text
• Secure generation with cryptographically strong randomness
• Immediate cleanup of temporary files and memory
• Encrypted storage and transmission

Compliance

• All rotations are audited with timestamps and user attribution
• Compliance tags track regulatory requirements
• Retention policies for audit logs and rotation history
• Regular security assessments and penetration testing

API Reference

REST Endpoints

The rotation service exposes the following endpoints:

Health Check

GET /health

Returns service health status and Vault connectivity.

Metrics

GET /metrics

Prometheus-compatible metrics for monitoring.

Manual Rotation

POST /api/v1/rotate
Content-Type: application/json

{
  "secret_path": "foxhunt/production/database/foxhunt_db",
  "reason": "Security incident",
  "triggered_by": "admin@foxhunt.com"
}

Rotation Status

GET /api/v1/status/{secret_path}

Get current rotation status for a specific secret.

Active Alerts

GET /api/v1/alerts

List all active alerts.

Acknowledge Alert

POST /api/v1/alerts/{alert_id}/ack
Content-Type: application/json

{
  "acknowledged_by": "admin@foxhunt.com"
}

Best Practices

1. Regular Monitoring

• Set up dashboard monitoring for rotation health
• Configure alerting for failed rotations
• Review rotation logs weekly
• Monitor Vault token expiration

2. Testing

• Test rotation procedures in staging environment
• Validate notification channels regularly
• Perform disaster recovery drills
• Test manual rotation procedures

3. Security Hygiene

• Rotate rotation service credentials regularly
• Review and update policies quarterly
• Audit access logs monthly
• Keep Vault and service updated

4. Performance Optimization

• Monitor rotation timing and adjust windows
• Use appropriate batch sizes for bulk operations
• Configure rate limiting for external APIs
• Optimize notification delivery

Disaster Recovery

Backup Procedures

# Export rotation policies
vault kv get -format=json foxhunt/rotation/policies/ > rotation-policies-backup.json

# Export schedules
vault kv get -format=json foxhunt/rotation/schedules/ > rotation-schedules-backup.json

# Export configuration
vault kv get -format=json foxhunt/rotation/config/ > rotation-config-backup.json

Recovery Procedures

# Restore policies
vault kv put foxhunt/rotation/policies/@rotation-policies-backup.json

# Restore schedules
vault kv put foxhunt/rotation/schedules/@rotation-schedules-backup.json

# Restore configuration
vault kv put foxhunt/rotation/config/@rotation-config-backup.json

Emergency Response

1. Service Failure: Restart service, check logs, escalate if needed
2. Vault Unavailable: Enable fallback mode, use cached secrets
3. Mass Rotation Failure: Stop all rotations, investigate root cause
4. Security Incident: Emergency rotation of all affected secrets

Support and Maintenance

Regular Tasks

• Weekly: Review rotation logs and metrics
• Monthly: Update rotation policies and test procedures
• Quarterly: Security audit and compliance review
• Annually: Disaster recovery testing and policy updates

Contact Information

• Security Team: security-team@foxhunt.com
• Operations: operations@foxhunt.com
• Compliance: compliance@foxhunt.com
• Emergency: security-incident@foxhunt.com

---

This guide covers the complete setup and operation of the Foxhunt Secret Rotation System. For additional support or custom requirements, contact the security team.