foxhunt/storage

Storage Crate

Overview

The storage crate provides robust integration with Amazon S3 for durable and scalable storage of models, large datasets, and other critical artifacts. It includes features for caching, versioning, data integrity verification, and efficient file management.

Features

• S3 Client Integration: Seamlessly integrates with AWS S3 for uploading, downloading, and managing objects.
• Model Caching: Implements a local caching layer for frequently accessed models, reducing latency and S3 API calls.
• Model Versioning: Supports tracking and managing different versions of machine learning models or configuration files stored in S3.
• Checksum Verification: Ensures data integrity by automatically verifying checksums (e.g., MD5, SHA256) during uploads and downloads.
• Compression/Decompression Utilities: Provides built-in support for compressing and decompressing files (e.g., Gzip, Zstd) to optimize storage and transfer costs.
• File Management API: Offers a high-level API for common S3 operations like listing objects, deleting, and managing prefixes.

Usage

use storage::{S3Client, S3Config};
use std::path::PathBuf;

let config = S3Config {
    bucket_name: "foxhunt-models".to_string(),
    region: "us-east-1".to_string(),
    // ... other AWS credentials or profile settings
};

// let client = S3Client::new(config).expect("Failed to create S3 client");

let local_file_path = PathBuf::from("./my_model.bin");
let s3_key = "models/v1/my_model.bin";

// Example: Upload a file to S3
// client.upload_file(&local_file_path, s3_key, true /* with checksum */)
//     .expect("Failed to upload model");
// println!("Model uploaded to s3://{}/{}", config.bucket_name, s3_key);

// Example: Download a file from S3
// let download_path = PathBuf::from("./downloaded_model.bin");
// client.download_file(s3_key, &download_path, true /* with checksum */)
//     .expect("Failed to download model");
// println!("Model downloaded to {:?}", download_path);

Testing

cargo test --package storage

Documentation

Complete API documentation is available at docs.rs/storage.