73b9ca0659
- Added safe_div(), safe_mul(), and safe_add() helper functions - All helpers check for NaN, infinity, and division by zero - Replaced direct float operations with safe wrappers - Fixed percentile calculations (lines 86-89) - Fixed success rate calculation (line 101) - Fixed throughput calculation (line 107) - Fixed all latency metric conversions (lines 133-154) - Fixed P99 latency display (lines 177, 182) - Fixed order quantity/price calculations (lines 215-216) All 17 float_arithmetic warnings in lib.rs now resolved. Part 1/2: 9 warnings requested, 17 actually fixed.
Storage Crate
Overview
The storage crate provides robust integration with Amazon S3 for durable and scalable storage of models, large datasets, and other critical artifacts. It includes features for caching, versioning, data integrity verification, and efficient file management.
Features
- S3 Client Integration: Seamlessly integrates with AWS S3 for uploading, downloading, and managing objects.
- Model Caching: Implements a local caching layer for frequently accessed models, reducing latency and S3 API calls.
- Model Versioning: Supports tracking and managing different versions of machine learning models or configuration files stored in S3.
- Checksum Verification: Ensures data integrity by automatically verifying checksums (e.g., MD5, SHA256) during uploads and downloads.
- Compression/Decompression Utilities: Provides built-in support for compressing and decompressing files (e.g., Gzip, Zstd) to optimize storage and transfer costs.
- File Management API: Offers a high-level API for common S3 operations like listing objects, deleting, and managing prefixes.
Usage
use storage::{S3Client, S3Config};
use std::path::PathBuf;
let config = S3Config {
bucket_name: "foxhunt-models".to_string(),
region: "us-east-1".to_string(),
// ... other AWS credentials or profile settings
};
// let client = S3Client::new(config).expect("Failed to create S3 client");
let local_file_path = PathBuf::from("./my_model.bin");
let s3_key = "models/v1/my_model.bin";
// Example: Upload a file to S3
// client.upload_file(&local_file_path, s3_key, true /* with checksum */)
// .expect("Failed to upload model");
// println!("Model uploaded to s3://{}/{}", config.bucket_name, s3_key);
// Example: Download a file from S3
// let download_path = PathBuf::from("./downloaded_model.bin");
// client.download_file(s3_key, &download_path, true /* with checksum */)
// .expect("Failed to download model");
// println!("Model downloaded to {:?}", download_path);
Testing
cargo test --package storage
Documentation
Complete API documentation is available at docs.rs/storage.