foxhunt/docs/S3_INTEGRATION.md

# S3 Integration Guide for Foxhunt HFT System

## Overview

This document provides comprehensive guidance for integrating AWS S3 storage with the Foxhunt High-Frequency Trading (HFT) system. S3 is used for cold storage, data archival, and backup functionality across multiple system components.

## Architecture Overview

The S3 integration consists of three main components:

1. **S3 Archival Service** (`core/src/storage/s3_archival.rs`) - General-purpose archival for historical data
2. **S3 Model Storage** (`services/ml_training_service/src/storage.rs`) - ML model artifact storage
3. **S3 Checkpoint Storage** (`ml/src/checkpoint/storage.rs`) - ML model checkpoint management

## Configuration

### Environment Variables

Create a `.env` file based on `.env.example` with the following variables:

#### Required AWS Credentials
```bash
# AWS Authentication
AWS_ACCESS_KEY_ID=your_access_key_id
AWS_SECRET_ACCESS_KEY=your_secret_access_key
AWS_REGION=us-east-1

# Optional session token (for temporary credentials)
AWS_SESSION_TOKEN=your_session_token
```

#### S3 Bucket Configuration
```bash
# Primary buckets for different data types
S3_ARCHIVAL_BUCKET=foxhunt-archives
S3_MODEL_STORAGE_BUCKET=foxhunt-models
S3_CHECKPOINT_BUCKET=foxhunt-checkpoints

# Storage classes for cost optimization
S3_DEFAULT_STORAGE_CLASS=STANDARD_IA
S3_COMPLIANCE_STORAGE_CLASS=GLACIER
```

#### Optional Configuration
```bash
# Archival settings
S3_ARCHIVAL_RETENTION_DAYS=2555  # 7 years for regulatory compliance
S3_ENABLE_COMPRESSION=true
S3_ENABLE_LIFECYCLE=true
S3_ENABLE_ENCRYPTION=true

# Performance settings
S3_MULTIPART_THRESHOLD=8388608  # 8MB
S3_MAX_CONCURRENT_UPLOADS=4
S3_MAX_RETRIES=3
```

### Vault Integration (Recommended for Production)

For enhanced security, use HashiCorp Vault to store AWS credentials:

```bash
# Vault configuration
VAULT_ADDR=http://localhost:8200
VAULT_TOKEN=your_vault_token
VAULT_S3_CREDENTIALS_PATH=secret/data/foxhunt/s3
```

Store credentials in Vault:
```bash
vault kv put secret/foxhunt/s3 \
    access_key_id="your_access_key_id" \
    secret_access_key="your_secret_access_key" \
    bucket_name="foxhunt-models" \
    region="us-east-1"
```

## Usage Examples

### 1. S3 Archival Service

For archiving historical data, trading logs, and compliance records:

```rust
use foxhunt_core::storage::{S3ArchivalService, ArchivalDataType};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize from environment variables
    let archival_service = S3ArchivalService::new().await?;
    
    // Archive trading logs
    let trading_logs = b"trade_data_here";
    let archive_id = archival_service.archive_data(
        &ArchivalDataType::TradingLogs,
        trading_logs,
        Some("Daily trading logs for 2024-01-15".to_string())
    ).await?;
    
    println!("Archived with ID: {}", archive_id);
    Ok(())
}
```

### 2. S3 Model Storage

For storing ML model artifacts in the training service:

```rust
use foxhunt_ml_training::storage::{S3ModelStorage, StorageManager};
use uuid::Uuid;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize from environment variables
    let s3_storage = S3ModelStorage::from_env().await?;
    
    // Store a model
    let job_id = Uuid::new_v4();
    let model_data = b"serialized_model_data";
    let artifact_path = s3_storage.store_model(job_id, model_data).await?;
    
    println!("Model stored at: {}", artifact_path);
    
    // Retrieve the model
    let retrieved_data = s3_storage.retrieve_model(&artifact_path).await?;
    assert_eq!(model_data, &retrieved_data[..]);
    
    Ok(())
}
```

### 3. S3 Checkpoint Storage

For ML model checkpoint management:

```rust
use foxhunt_ml::checkpoint::{S3CheckpointStorage, CheckpointMetadata, CheckpointStorage};
use foxhunt_ml::model::ModelType;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize from environment variables
    let checkpoint_storage = S3CheckpointStorage::from_env().await?;
    
    // Create checkpoint metadata
    let metadata = CheckpointMetadata {
        model_type: ModelType::MAMBA,
        model_name: "mamba_v1".to_string(),
        version: "1.0.0".to_string(),
        checkpoint_id: "checkpoint_001".to_string(),
        created_at: chrono::Utc::now(),
        epoch: Some(10),
        step: Some(1000),
        loss: Some(0.0234),
        accuracy: Some(0.9567),
        file_size: 1024 * 1024, // 1MB
        compression: Some("gzip".to_string()),
        metadata: Default::default(),
    };
    
    // Save checkpoint
    let checkpoint_data = b"checkpoint_data_here";
    checkpoint_storage.save_checkpoint(
        "mamba_v1_epoch_10.ckpt",
        checkpoint_data,
        &metadata
    ).await?;
    
    // Load checkpoint
    let loaded_data = checkpoint_storage.load_checkpoint("mamba_v1_epoch_10.ckpt").await?;
    assert_eq!(checkpoint_data, &loaded_data[..]);
    
    Ok(())
}
```

## AWS Credential Management

### Option 1: Environment Variables (Development)
Set the required environment variables in your `.env` file or shell:
```bash
export AWS_ACCESS_KEY_ID="your_key"
export AWS_SECRET_ACCESS_KEY="your_secret"
export AWS_REGION="us-east-1"
```

### Option 2: AWS Credential Chain (Production)
The implementation automatically falls back to AWS credential chain:
- IAM roles (recommended for EC2/ECS)
- AWS profiles (`~/.aws/credentials`)
- Environment variables
- Instance metadata service

### Option 3: HashiCorp Vault (Enterprise)
For the ML Training Service, credentials are retrieved from Vault:
```rust
let storage_manager = StorageManager::new(config, Some(vault_client)).await?;
```

## Cost Optimization

### Storage Classes
The system automatically selects appropriate storage classes:

- **STANDARD**: Frequently accessed data (trading logs)
- **STANDARD_IA**: Infrequent access (model checkpoints, historical data)
- **GLACIER**: Long-term compliance data (7+ years)
- **DEEP_ARCHIVE**: Very long-term archival (10+ years)

### Lifecycle Policies
Automatic transitions are configured when `S3_ENABLE_LIFECYCLE=true`:

1. **Day 0-30**: STANDARD storage
2. **Day 30-90**: STANDARD_IA storage
3. **Day 90-365**: GLACIER storage
4. **Day 365+**: DEEP_ARCHIVE storage

### Compression
Enable compression to reduce storage costs:
```bash
S3_ENABLE_COMPRESSION=true
```

Supported algorithms:
- **gzip**: General purpose, good compression ratio
- **lz4**: Fast compression/decompression
- **zstd**: Best compression ratio

## Security Best Practices

### 1. Encryption
- **Server-side encryption**: Enabled by default (AES-256)
- **In-transit encryption**: HTTPS/TLS for all API calls
- **Client-side encryption**: Optional for sensitive data

### 2. Access Control
```bash
# Enable versioning for data protection
S3_ENABLE_VERSIONING=true

# Enable MFA delete for critical buckets (production)
S3_ENABLE_MFA_DELETE=true
```

### 3. IAM Policies
Minimum required permissions:
```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucket",
                "s3:HeadBucket"
            ],
            "Resource": [
                "arn:aws:s3:::foxhunt-*",
                "arn:aws:s3:::foxhunt-*/*"
            ]
        }
    ]
}
```

## Monitoring and Logging

### CloudWatch Metrics
Monitor the following metrics:
- **BucketSizeBytes**: Total storage usage
- **NumberOfObjects**: Object count per bucket
- **AllRequests**: API request volume
- **Errors**: Failed requests

### CloudTrail Logging
Enable CloudTrail for audit logging:
- All S3 API calls
- Access patterns
- Data lifecycle events

### Application Logs
The Rust implementation provides structured logging:
```rust
use tracing::{info, warn, error};

// Logs are automatically generated for:
// - Connection establishment
// - Upload/download operations  
// - Error conditions
// - Performance metrics
```

## Troubleshooting

### Common Issues

#### 1. Access Denied Errors
```
Failed to connect to S3 bucket. Check credentials and bucket permissions.
```

**Solution**: 
- Verify AWS credentials are correct
- Check IAM permissions include required S3 actions
- Ensure bucket exists and is accessible

#### 2. Region Mismatch
```
The bucket is in this region: us-west-2. Please use this region.
```

**Solution**:
- Set correct `AWS_REGION` environment variable
- Ensure bucket exists in the specified region

#### 3. Large File Upload Failures
```
Request timeout during multipart upload
```

**Solution**:
- Increase `S3_MAX_RETRIES` value
- Check network connectivity and bandwidth
- Consider using smaller `S3_PART_SIZE`

#### 4. Credential Chain Issues
```
Unable to load credentials from any credential source
```

**Solution**:
- Verify environment variables are set
- Check AWS CLI configuration: `aws configure list`
- For EC2/ECS, ensure IAM role is attached

### Debug Mode
Enable verbose logging for troubleshooting:
```bash
RUST_LOG=debug cargo run --bin your_service
```

## Performance Tuning

### Upload Performance
```bash
# Multipart upload threshold (8MB default)
S3_MULTIPART_THRESHOLD=8388608

# Concurrent uploads (4 default)
S3_MAX_CONCURRENT_UPLOADS=4

# Part size for multipart uploads (5MB default)
S3_PART_SIZE=5242880
```

### Connection Pooling
The AWS SDK handles connection pooling automatically, but you can tune:
- Request timeout settings
- Connection limits
- Retry strategies

### Batch Operations
For high-throughput scenarios:
- Use multipart uploads for large files (>5MB)
- Batch small operations where possible
- Consider S3 Transfer Acceleration for global access

## Backup and Disaster Recovery

### Cross-Region Replication
Configure replication for critical data:
```bash
# Primary region
AWS_REGION=us-east-1

# Backup region for DR
S3_BACKUP_REGION=us-west-2
```

### Backup Strategy
- **Daily**: Incremental backups of trading data
- **Weekly**: Full model checkpoint backups
- **Monthly**: Complete archive snapshots
- **Annual**: Compliance data verification

### Recovery Testing
Regular recovery testing procedures:
1. Test credential rotation
2. Verify backup integrity
3. Practice disaster recovery scenarios
4. Validate cross-region access

## Integration with Other Services

### PostgreSQL Integration
Backup database dumps to S3:
```rust
// Archive PostgreSQL backups
let backup_data = create_database_backup().await?;
archival_service.archive_data(
    &ArchivalDataType::ComplianceData,
    &backup_data,
    Some("PostgreSQL backup".to_string())
).await?;
```

### Redis Integration
Archive Redis snapshots:
```rust
// Archive Redis snapshots for replay
let redis_snapshot = create_redis_snapshot().await?;
archival_service.archive_data(
    &ArchivalDataType::TradingLogs,
    &redis_snapshot,
    Some("Redis trading state".to_string())
).await?;
```

## Compliance and Regulatory Requirements

### MiFID II Compliance
- **Data Retention**: 7 years minimum (2555 days configured)
- **Data Integrity**: Checksums and versioning enabled
- **Audit Trail**: CloudTrail integration for access logging
- **Encryption**: At-rest and in-transit encryption

### SOX Compliance
- **Change Control**: Versioned objects with metadata
- **Access Control**: IAM-based permissions
- **Monitoring**: CloudWatch metrics and alerting

### GDPR Considerations
- **Data Location**: Specify appropriate AWS regions
- **Data Deletion**: Implement data retention policies
- **Access Logging**: Track all data access for audits

## Future Enhancements

### Planned Features
1. **S3 Intelligent Tiering**: Automatic cost optimization
2. **Cross-Region Sync**: Multi-region deployment support  
3. **Compression Algorithms**: Additional compression options
4. **Metadata Indexing**: Enhanced search capabilities
5. **Cost Analytics**: Detailed cost breakdown and optimization

### Performance Improvements
1. **Connection Pooling**: Advanced connection management
2. **Parallel Processing**: Concurrent upload/download operations
3. **Caching Layer**: Redis-based metadata caching
4. **Batch Operations**: Bulk operations for efficiency

---

## Getting Started Checklist

- [ ] Copy `.env.example` to `.env`
- [ ] Configure AWS credentials (environment variables or IAM role)
- [ ] Set up S3 buckets with appropriate names
- [ ] Enable feature flags: `s3-archival`, `s3-storage`
- [ ] Test connection: `cargo test s3_connection_test`
- [ ] Configure lifecycle policies in AWS Console
- [ ] Set up CloudWatch monitoring
- [ ] Enable CloudTrail for audit logging
- [ ] Test backup and recovery procedures

For additional support, refer to the AWS SDK for Rust documentation or contact the development team.