foxhunt/docs/S3_INTEGRATION.md

S3 Integration Guide for Foxhunt HFT System

Overview

This document provides comprehensive guidance for integrating AWS S3 storage with the Foxhunt High-Frequency Trading (HFT) system. S3 is used for cold storage, data archival, and backup functionality across multiple system components.

Architecture Overview

The S3 integration consists of three main components:

1. S3 Archival Service (core/src/storage/s3_archival.rs) - General-purpose archival for historical data
2. S3 Model Storage (services/ml_training_service/src/storage.rs) - ML model artifact storage
3. S3 Checkpoint Storage (ml/src/checkpoint/storage.rs) - ML model checkpoint management

Configuration

Environment Variables

Create a .env file based on .env.example with the following variables:

Required AWS Credentials

# AWS Authentication
AWS_ACCESS_KEY_ID=your_access_key_id
AWS_SECRET_ACCESS_KEY=your_secret_access_key
AWS_REGION=us-east-1

# Optional session token (for temporary credentials)
AWS_SESSION_TOKEN=your_session_token

S3 Bucket Configuration

# Primary buckets for different data types
S3_ARCHIVAL_BUCKET=foxhunt-archives
S3_MODEL_STORAGE_BUCKET=foxhunt-models
S3_CHECKPOINT_BUCKET=foxhunt-checkpoints

# Storage classes for cost optimization
S3_DEFAULT_STORAGE_CLASS=STANDARD_IA
S3_COMPLIANCE_STORAGE_CLASS=GLACIER

Optional Configuration

# Archival settings
S3_ARCHIVAL_RETENTION_DAYS=2555  # 7 years for regulatory compliance
S3_ENABLE_COMPRESSION=true
S3_ENABLE_LIFECYCLE=true
S3_ENABLE_ENCRYPTION=true

# Performance settings
S3_MULTIPART_THRESHOLD=8388608  # 8MB
S3_MAX_CONCURRENT_UPLOADS=4
S3_MAX_RETRIES=3

Vault Integration (Recommended for Production)

For enhanced security, use HashiCorp Vault to store AWS credentials:

# Vault configuration
VAULT_ADDR=http://localhost:8200
VAULT_TOKEN=your_vault_token
VAULT_S3_CREDENTIALS_PATH=secret/data/foxhunt/s3

Store credentials in Vault:

vault kv put secret/foxhunt/s3 \
    access_key_id="your_access_key_id" \
    secret_access_key="your_secret_access_key" \
    bucket_name="foxhunt-models" \
    region="us-east-1"

Usage Examples

1. S3 Archival Service

For archiving historical data, trading logs, and compliance records:

use foxhunt_core::storage::{S3ArchivalService, ArchivalDataType};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize from environment variables
    let archival_service = S3ArchivalService::new().await?;
    
    // Archive trading logs
    let trading_logs = b"trade_data_here";
    let archive_id = archival_service.archive_data(
        &ArchivalDataType::TradingLogs,
        trading_logs,
        Some("Daily trading logs for 2024-01-15".to_string())
    ).await?;
    
    println!("Archived with ID: {}", archive_id);
    Ok(())
}

2. S3 Model Storage

For storing ML model artifacts in the training service:

use foxhunt_ml_training::storage::{S3ModelStorage, StorageManager};
use uuid::Uuid;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize from environment variables
    let s3_storage = S3ModelStorage::from_env().await?;
    
    // Store a model
    let job_id = Uuid::new_v4();
    let model_data = b"serialized_model_data";
    let artifact_path = s3_storage.store_model(job_id, model_data).await?;
    
    println!("Model stored at: {}", artifact_path);
    
    // Retrieve the model
    let retrieved_data = s3_storage.retrieve_model(&artifact_path).await?;
    assert_eq!(model_data, &retrieved_data[..]);
    
    Ok(())
}

3. S3 Checkpoint Storage

For ML model checkpoint management:

use foxhunt_ml::checkpoint::{S3CheckpointStorage, CheckpointMetadata, CheckpointStorage};
use foxhunt_ml::model::ModelType;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize from environment variables
    let checkpoint_storage = S3CheckpointStorage::from_env().await?;
    
    // Create checkpoint metadata
    let metadata = CheckpointMetadata {
        model_type: ModelType::MAMBA,
        model_name: "mamba_v1".to_string(),
        version: "1.0.0".to_string(),
        checkpoint_id: "checkpoint_001".to_string(),
        created_at: chrono::Utc::now(),
        epoch: Some(10),
        step: Some(1000),
        loss: Some(0.0234),
        accuracy: Some(0.9567),
        file_size: 1024 * 1024, // 1MB
        compression: Some("gzip".to_string()),
        metadata: Default::default(),
    };
    
    // Save checkpoint
    let checkpoint_data = b"checkpoint_data_here";
    checkpoint_storage.save_checkpoint(
        "mamba_v1_epoch_10.ckpt",
        checkpoint_data,
        &metadata
    ).await?;
    
    // Load checkpoint
    let loaded_data = checkpoint_storage.load_checkpoint("mamba_v1_epoch_10.ckpt").await?;
    assert_eq!(checkpoint_data, &loaded_data[..]);
    
    Ok(())
}

AWS Credential Management

Option 1: Environment Variables (Development)

Set the required environment variables in your .env file or shell:

export AWS_ACCESS_KEY_ID="your_key"
export AWS_SECRET_ACCESS_KEY="your_secret"
export AWS_REGION="us-east-1"

Option 2: AWS Credential Chain (Production)

The implementation automatically falls back to AWS credential chain:

• IAM roles (recommended for EC2/ECS)
• AWS profiles (~/.aws/credentials)
• Environment variables
• Instance metadata service

Option 3: HashiCorp Vault (Enterprise)

For the ML Training Service, credentials are retrieved from Vault:

let storage_manager = StorageManager::new(config, Some(vault_client)).await?;

Cost Optimization

Storage Classes

The system automatically selects appropriate storage classes:

• STANDARD: Frequently accessed data (trading logs)
• STANDARD_IA: Infrequent access (model checkpoints, historical data)
• GLACIER: Long-term compliance data (7+ years)
• DEEP_ARCHIVE: Very long-term archival (10+ years)

Lifecycle Policies

Automatic transitions are configured when S3_ENABLE_LIFECYCLE=true:

1. Day 0-30: STANDARD storage
2. Day 30-90: STANDARD_IA storage
3. Day 90-365: GLACIER storage
4. Day 365+: DEEP_ARCHIVE storage

Compression

Enable compression to reduce storage costs:

S3_ENABLE_COMPRESSION=true

Supported algorithms:

• gzip: General purpose, good compression ratio
• lz4: Fast compression/decompression
• zstd: Best compression ratio

Security Best Practices

1. Encryption

• Server-side encryption: Enabled by default (AES-256)
• In-transit encryption: HTTPS/TLS for all API calls
• Client-side encryption: Optional for sensitive data

2. Access Control

# Enable versioning for data protection
S3_ENABLE_VERSIONING=true

# Enable MFA delete for critical buckets (production)
S3_ENABLE_MFA_DELETE=true

3. IAM Policies

Minimum required permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucket",
                "s3:HeadBucket"
            ],
            "Resource": [
                "arn:aws:s3:::foxhunt-*",
                "arn:aws:s3:::foxhunt-*/*"
            ]
        }
    ]
}

Monitoring and Logging

CloudWatch Metrics

Monitor the following metrics:

• BucketSizeBytes: Total storage usage
• NumberOfObjects: Object count per bucket
• AllRequests: API request volume
• Errors: Failed requests

CloudTrail Logging

Enable CloudTrail for audit logging:

• All S3 API calls
• Access patterns
• Data lifecycle events

Application Logs

The Rust implementation provides structured logging:

use tracing::{info, warn, error};

// Logs are automatically generated for:
// - Connection establishment
// - Upload/download operations  
// - Error conditions
// - Performance metrics

Troubleshooting

Common Issues

1. Access Denied Errors

Failed to connect to S3 bucket. Check credentials and bucket permissions.

Solution:

• Verify AWS credentials are correct
• Check IAM permissions include required S3 actions
• Ensure bucket exists and is accessible

2. Region Mismatch

The bucket is in this region: us-west-2. Please use this region.

Solution:

• Set correct AWS_REGION environment variable
• Ensure bucket exists in the specified region

3. Large File Upload Failures

Request timeout during multipart upload

Solution:

• Increase S3_MAX_RETRIES value
• Check network connectivity and bandwidth
• Consider using smaller S3_PART_SIZE

4. Credential Chain Issues

Unable to load credentials from any credential source

Solution:

• Verify environment variables are set
• Check AWS CLI configuration: aws configure list
• For EC2/ECS, ensure IAM role is attached

Debug Mode

Enable verbose logging for troubleshooting:

RUST_LOG=debug cargo run --bin your_service

Performance Tuning

Upload Performance

# Multipart upload threshold (8MB default)
S3_MULTIPART_THRESHOLD=8388608

# Concurrent uploads (4 default)
S3_MAX_CONCURRENT_UPLOADS=4

# Part size for multipart uploads (5MB default)
S3_PART_SIZE=5242880

Connection Pooling

The AWS SDK handles connection pooling automatically, but you can tune:

• Request timeout settings
• Connection limits
• Retry strategies

Batch Operations

For high-throughput scenarios:

• Use multipart uploads for large files (>5MB)
• Batch small operations where possible
• Consider S3 Transfer Acceleration for global access

Backup and Disaster Recovery

Cross-Region Replication

Configure replication for critical data:

# Primary region
AWS_REGION=us-east-1

# Backup region for DR
S3_BACKUP_REGION=us-west-2

Backup Strategy

• Daily: Incremental backups of trading data
• Weekly: Full model checkpoint backups
• Monthly: Complete archive snapshots
• Annual: Compliance data verification

Recovery Testing

Regular recovery testing procedures:

1. Test credential rotation
2. Verify backup integrity
3. Practice disaster recovery scenarios
4. Validate cross-region access

Integration with Other Services

PostgreSQL Integration

Backup database dumps to S3:

// Archive PostgreSQL backups
let backup_data = create_database_backup().await?;
archival_service.archive_data(
    &ArchivalDataType::ComplianceData,
    &backup_data,
    Some("PostgreSQL backup".to_string())
).await?;

Redis Integration

Archive Redis snapshots:

// Archive Redis snapshots for replay
let redis_snapshot = create_redis_snapshot().await?;
archival_service.archive_data(
    &ArchivalDataType::TradingLogs,
    &redis_snapshot,
    Some("Redis trading state".to_string())
).await?;

Compliance and Regulatory Requirements

MiFID II Compliance

• Data Retention: 7 years minimum (2555 days configured)
• Data Integrity: Checksums and versioning enabled
• Audit Trail: CloudTrail integration for access logging
• Encryption: At-rest and in-transit encryption

SOX Compliance

• Change Control: Versioned objects with metadata
• Access Control: IAM-based permissions
• Monitoring: CloudWatch metrics and alerting

GDPR Considerations

• Data Location: Specify appropriate AWS regions
• Data Deletion: Implement data retention policies
• Access Logging: Track all data access for audits

Future Enhancements

Planned Features

1. S3 Intelligent Tiering: Automatic cost optimization
2. Cross-Region Sync: Multi-region deployment support
3. Compression Algorithms: Additional compression options
4. Metadata Indexing: Enhanced search capabilities
5. Cost Analytics: Detailed cost breakdown and optimization

Performance Improvements

1. Connection Pooling: Advanced connection management
2. Parallel Processing: Concurrent upload/download operations
3. Caching Layer: Redis-based metadata caching
4. Batch Operations: Bulk operations for efficiency

---

Getting Started Checklist

• Copy .env.example to .env
• Configure AWS credentials (environment variables or IAM role)
• Set up S3 buckets with appropriate names
• Enable feature flags: s3-archival, s3-storage
• Test connection: cargo test s3_connection_test
• Configure lifecycle policies in AWS Console
• Set up CloudWatch monitoring
• Enable CloudTrail for audit logging
• Test backup and recovery procedures

For additional support, refer to the AWS SDK for Rust documentation or contact the development team.