foxhunt/config/README.md

Config Crate

Overview

The config crate provides a centralized, dynamic, and secure configuration management solution for Foxhunt HFT services. It enables hot-reloading of configurations and integrates with robust secret management systems, ensuring operational flexibility and security.

Features

• Centralized PostgreSQL Storage: Stores all application configurations in a PostgreSQL database, providing a single source of truth.
• Dynamic Hot-Reloading: Leverages PostgreSQL's NOTIFY/LISTEN mechanism to push live configuration updates to running services without restarts.
• Secure Secret Management: Integrates with HashiCorp Vault for secure storage and retrieval of sensitive credentials and secrets.
• Schema-Validated Configurations: Enforces structured configuration schemas to prevent malformed or invalid configurations.
• Model Configuration Management: Manages configurations for various trading models, including their parameters and associated S3 asset paths.
• Service-Specific Schemas: Allows defining and validating distinct configuration schemas for each microservice or component.

Architecture

The config crate's architecture comprises:

• Config Store: A PostgreSQL database instance dedicated to storing configuration data.
• Config Loader: Component responsible for fetching configurations from PostgreSQL.
• Vault Client: Interface for securely interacting with HashiCorp Vault to retrieve secrets.
• Notifier/Listener: Utilizes PostgreSQL NOTIFY/LISTEN channels to signal and receive configuration changes for hot-reloading.
• Schema Validator: Ensures that loaded configurations adhere to predefined JSON or YAML schemas.
• Configuration Models: Rust structs that represent the structured configuration data, often deserialized from JSON/YAML stored in the database.

Usage

To load a configuration and listen for live updates:

use config::{
    ConfigManager,
    schema::ServiceConfig,
};
use serde::{Deserialize, Serialize};

#[derive(Debug, Clone, Serialize, Deserialize)]
struct MyServiceSpecificConfig {
    api_key_name: String,
    trade_threshold: f64,
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize ConfigManager with database connection and Vault client
    let config_manager = ConfigManager::new(
        "postgres://user:pass@localhost/foxhunt_config",
        "http://localhost:8200", // Vault address
    ).await?;

    // Load initial configuration for a specific service
    let initial_config: MyServiceSpecificConfig = config_manager
        .get_service_config("my_trading_service")
        .await?;
    println!("Initial config: {:?}", initial_config);

    // Subscribe to updates for this service's configuration
    let mut config_stream = config_manager
        .subscribe_to_service_config::<MyServiceSpecificConfig>("my_trading_service")
        .await?;

    println!("Listening for config updates...");

    tokio::spawn(async move {
        while let Some(updated_config) = config_stream.recv().await {
            println!("Configuration updated: {:?}", updated_config);
            // Apply the new configuration to the running service
        }
    });

    tokio::signal::ctrl_c().await?;
    println!("Shutting down config listener.");

    Ok(())
}

Testing

To run the tests for the config crate:

cargo test --package config

Documentation

Comprehensive API documentation is available at docs.rs/config.