foxhunt/WAVE_66_AGENT_11_MAGIC_NUMBERS_ANALYSIS.md

Wave 66 Agent 11: Magic Numbers and Configuration Centralization Analysis

Executive Summary

Status: Extensive hardcoded configuration values scattered across codebase
Impact: Medium - Affects maintainability, testing, and operational flexibility
Recommendation: Implement 3-tier configuration architecture (compile-time constants, runtime config, database config)

Current State Analysis

1. Hardcoded Values Categories

A. Duration/Timeout Values (200+ instances)

// Examples found:
Duration::from_secs(60)      // 89 instances - auto-recovery, caching, health checks
Duration::from_millis(100)   // 67 instances - batch timeouts, retry delays
Duration::from_secs(30)      // 45 instances - health checks, intervals
Duration::from_secs(300)     // 31 instances - cache TTL, session timeouts
Duration::from_secs(5)       // 28 instances - connection timeouts

Key Problem Areas:

• /home/jgrusewski/Work/foxhunt/risk/src/lib.rs: Lines 327-387 (13 hardcoded durations)
• /home/jgrusewski/Work/foxhunt/ml/src/deployment/validation.rs: 10+ hardcoded timeout values
• /home/jgrusewski/Work/foxhunt/trading_engine/src/events/postgres_writer.rs: Batch and retry timings

B. Numeric Constants (147+ instances)

// Compile-time constants (appropriate for constants modules):
pub const MAX_QUERY_TIMEOUT_MS: u64 = 1000;
pub const MAX_HFT_LATENCY_MICROS: u64 = 50;
pub const PRICE_SCALE: i64 = 1_000_000;
pub const PRECISION_FACTOR: i64 = 100_000_000;

// Should be in database config (runtime-configurable):
const MAX_ORDERS: usize = 100_000;           // trading_engine/src/trading_operations_optimized.rs:36
const RING_BUFFER_SIZE: usize = 4096;        // trading_engine/src/metrics.rs:47
const DEFAULT_MAX_HOLDING_PERIOD_NS: u64 = 3600_000_000_000; // ml/src/labeling/constants.rs:10

C. Percentage Thresholds (100+ instances)

// Risk management thresholds (should be database-configurable):
if breach_percentage >= Decimal::from(120)   // Critical breach at 120%
if breach_percentage >= Decimal::from(100)   // Hard breach at 100%
if breach_percentage >= Decimal::from(90)    // Soft breach at 90%
if utilization >= Decimal::from(85)          // Warning at 85%

// Performance thresholds:
if confidence_level <= 0.90 { 1.282 }
else if confidence_level <= 0.95 { 1.645 }

D. Connection Strings (113+ instances)

// Database URLs (should use environment variables):
"postgresql://localhost/foxhunt"             // 74 instances
"redis://localhost:6379"                     // 37 instances
"postgres://postgres:postgres@127.0.0.1:5432/postgres" // 2 instances

E. Environment Variables (100+ unique keys)

// Critical environment variables found:
std::env::var("DATABASE_URL")                // 15+ instances
std::env::var("REDIS_URL")                   // 12+ instances  
std::env::var("DATABENTO_API_KEY")           // 8+ instances
std::env::var("AWS_REGION")                  // 7+ instances
std::env::var("S3_*")                        // 10+ different S3 vars
std::env::var("IB_*")                        // 6+ Interactive Brokers vars

Detailed Breakdown by Category

Category 1: Breach Thresholds (High Priority for Database Config)

Location: /home/jgrusewski/Work/foxhunt/risk-data/src/limits.rs

// Lines 68-76: Hardcoded breach severity thresholds
if breach_percentage >= Decimal::from(120) {
    BreachSeverity::Critical  // 120% threshold
} else if breach_percentage >= Decimal::from(100) {
    BreachSeverity::Hard      // 100% threshold
} else if breach_percentage >= Decimal::from(90) {
    BreachSeverity::Soft      // 90% threshold
}

Impact: These should be configurable per limit type, not hardcoded.

Category 2: Redis TTL Values (Medium Priority)

Examples:

// risk-data/src/limits.rs:620
.arg(300) // 5 minutes TTL

// risk-data/src/compliance.rs:502
.arg(86_400_i32) // 24 hours TTL

// risk-data/src/var.rs:515
.arg(3600) // 1 hour TTL

Impact: Cannot adjust cache behavior without code changes.

Category 3: Performance Constants (Low Priority - Compile-time OK)

Well-structured examples (already in constants modules):

// common/src/constants.rs - GOOD
pub const MAX_QUERY_TIMEOUT_MS: u64 = 1000;
pub const MAX_HFT_LATENCY_MICROS: u64 = 50;

// ml/src/labeling/constants.rs - GOOD
pub const MAX_GPU_BATCH_SIZE: usize = 8192;
pub const NANOS_PER_SECOND: u64 = 1_000_000_000;

// trading_engine/src/lib.rs - GOOD
pub const MAX_TIMING_LATENCY_NS: u64 = 14;
pub const CACHE_LINE_SIZE: usize = 64;

Category 4: Environment-Specific Config (Critical)

Development vs Production differences:

// risk/src/lib.rs:327-387
Development:
    auto_recovery_delay: Duration::from_secs(60)    // 1 minute
    cache_ttl: Duration::from_secs(30)
    loss_check_interval: Duration::from_secs(30)

Production:
    auto_recovery_delay: Duration::from_secs(1800)  // 30 minutes
    cache_ttl: Duration::from_secs(10)
    loss_check_interval: Duration::from_secs(5)

Problem: Hardcoded in source code, not environment-driven.

Centralization Strategy

Tier 1: Compile-Time Constants (Keep as-is)

Purpose: Values that never change and are performance-critical

Location: Dedicated constants modules

• /home/jgrusewski/Work/foxhunt/common/src/constants.rs ✅ Already good
• /home/jgrusewski/Work/foxhunt/ml/src/labeling/constants.rs ✅ Already good
• /home/jgrusewski/Work/foxhunt/trading_engine/src/lib.rs ✅ Already good

Examples:

// Mathematical constants
pub const NANOS_PER_SECOND: u64 = 1_000_000_000;
pub const BASIS_POINTS_PER_UNIT: u32 = 10_000;
pub const PRICE_SCALE: i64 = 1_000_000;

// Hardware constants
pub const CACHE_LINE_SIZE: usize = 64;
pub const SIMD_ALIGNMENT: usize = 32;

// Protocol constants
pub const MAX_SYMBOL_LENGTH: usize = 12;
pub const MAX_METADATA_ENTRIES: usize = 100;

Tier 2: Runtime Configuration (Environment Variables + Config Crate)

Purpose: Values that change between environments (dev/staging/prod)

Proposed Structure:

// config/src/runtime_config.rs (NEW FILE)

/// Runtime configuration that varies by environment
#[derive(Debug, Clone, Deserialize)]
pub struct RuntimeConfig {
    pub environment: Environment,
    pub timeouts: TimeoutConfig,
    pub performance: PerformanceConfig,
    pub cache: CacheConfig,
    pub monitoring: MonitoringConfig,
}

#[derive(Debug, Clone, Deserialize)]
pub struct TimeoutConfig {
    /// Connection timeout for database operations
    #[serde(default = "default_db_connection_timeout")]
    pub db_connection_timeout_ms: u64,
    
    /// Query timeout for HFT operations
    #[serde(default = "default_db_query_timeout")]
    pub db_query_timeout_ms: u64,
    
    /// gRPC request timeout
    #[serde(default = "default_grpc_timeout")]
    pub grpc_request_timeout_ms: u64,
    
    /// Auto-recovery delay after circuit breaker trip
    #[serde(default = "default_auto_recovery_delay")]
    pub auto_recovery_delay_secs: u64,
}

fn default_db_connection_timeout() -> u64 {
    match std::env::var("ENVIRONMENT").as_deref() {
        Ok("production") => 100,
        Ok("staging") => 200,
        _ => 500, // Development
    }
}

#[derive(Debug, Clone, Deserialize)]
pub struct CacheConfig {
    #[serde(default = "default_cache_ttl")]
    pub default_ttl_secs: u64,
    
    #[serde(default = "default_position_cache_ttl")]
    pub position_cache_ttl_secs: u64,
    
    #[serde(default = "default_var_cache_ttl")]
    pub var_cache_ttl_secs: u64,
}

Environment Variable Mapping:

# .env.development
ENVIRONMENT=development
DB_CONNECTION_TIMEOUT_MS=500
DB_QUERY_TIMEOUT_MS=1000
AUTO_RECOVERY_DELAY_SECS=60
CACHE_DEFAULT_TTL_SECS=30

# .env.production
ENVIRONMENT=production
DB_CONNECTION_TIMEOUT_MS=100
DB_QUERY_TIMEOUT_MS=800
AUTO_RECOVERY_DELAY_SECS=1800
CACHE_DEFAULT_TTL_SECS=10

Tier 3: Database Configuration (Hot-Reloadable)

Purpose: Values that operators need to adjust at runtime

Schema Extension (add to existing config tables):

-- database/schemas/005_runtime_config.sql

CREATE TABLE IF NOT EXISTS runtime_thresholds (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    config_key VARCHAR(255) NOT NULL UNIQUE,
    config_category VARCHAR(100) NOT NULL, -- 'risk', 'performance', 'cache', etc.
    config_value JSONB NOT NULL,
    description TEXT,
    min_value DECIMAL,
    max_value DECIMAL,
    value_type VARCHAR(50) NOT NULL, -- 'duration_ms', 'percentage', 'count', etc.
    environment VARCHAR(50), -- NULL = all environments
    is_active BOOLEAN DEFAULT true,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    created_by VARCHAR(255),
    updated_by VARCHAR(255)
);

-- Breach severity thresholds
INSERT INTO runtime_thresholds (config_key, config_category, config_value, description, value_type) VALUES
('breach.warning_threshold_pct', 'risk', '{"value": 80}', 'Warning threshold percentage', 'percentage'),
('breach.soft_threshold_pct', 'risk', '{"value": 90}', 'Soft breach threshold percentage', 'percentage'),
('breach.hard_threshold_pct', 'risk', '{"value": 100}', 'Hard breach threshold percentage', 'percentage'),
('breach.critical_threshold_pct', 'risk', '{"value": 120}', 'Critical breach threshold percentage', 'percentage');

-- Cache TTL values
INSERT INTO runtime_thresholds (config_key, config_category, config_value, description, value_type) VALUES
('cache.position_ttl_secs', 'performance', '{"value": 60}', 'Position cache TTL in seconds', 'duration_secs'),
('cache.var_calculation_ttl_secs', 'performance', '{"value": 3600}', 'VaR calculation cache TTL', 'duration_secs'),
('cache.compliance_check_ttl_secs', 'performance', '{"value": 86400}', 'Compliance check cache TTL', 'duration_secs');

-- Risk calculation intervals
INSERT INTO runtime_thresholds (config_key, config_category, config_value, description, value_type) VALUES
('risk.loss_check_interval_secs', 'risk', '{"value": 10}', 'How often to check for losses', 'duration_secs'),
('risk.position_check_interval_secs', 'risk', '{"value": 5}', 'Position limit check interval', 'duration_secs'),
('risk.concentration_check_interval_secs', 'risk', '{"value": 30}', 'Concentration risk check interval', 'duration_secs');

-- VaR confidence thresholds
INSERT INTO runtime_thresholds (config_key, config_category, config_value, description, value_type) VALUES
('var.p95_z_score', 'risk', '{"value": 1.645}', 'Z-score for 95% confidence VaR', 'decimal'),
('var.p99_z_score', 'risk', '{"value": 2.326}', 'Z-score for 99% confidence VaR', 'decimal'),
('var.p99_9_z_score', 'risk', '{"value": 3.09}', 'Z-score for 99.9% confidence VaR', 'decimal');

CREATE INDEX idx_runtime_thresholds_key ON runtime_thresholds(config_key);
CREATE INDEX idx_runtime_thresholds_category ON runtime_thresholds(config_category);
CREATE INDEX idx_runtime_thresholds_active ON runtime_thresholds(is_active) WHERE is_active = true;

Implementation Plan

Phase 1: Create Constants Consolidation Module (Week 1)

New File: /home/jgrusewski/Work/foxhunt/common/src/thresholds.rs

//! Centralized threshold constants for the Foxhunt system
//!
//! This module consolidates all hardcoded threshold values that were
//! previously scattered throughout the codebase.

/// Risk management thresholds
pub mod risk_thresholds {
    use rust_decimal::Decimal;
    
    /// Breach severity warning threshold (percentage of limit)
    pub const BREACH_WARNING_PCT: Decimal = Decimal::from_parts_raw(80, 0, 0, false, 0);
    
    /// Breach severity soft threshold (percentage of limit)
    pub const BREACH_SOFT_PCT: Decimal = Decimal::from_parts_raw(90, 0, 0, false, 0);
    
    /// Breach severity hard threshold (percentage of limit)
    pub const BREACH_HARD_PCT: Decimal = Decimal::from_parts_raw(100, 0, 0, false, 0);
    
    /// Breach severity critical threshold (percentage of limit)
    pub const BREACH_CRITICAL_PCT: Decimal = Decimal::from_parts_raw(120, 0, 0, false, 0);
    
    /// Minimum capital adequacy ratio (Basel III standard)
    pub const MIN_CAPITAL_ADEQUACY_RATIO: f64 = 0.08;
    
    /// Minimum leverage ratio (Basel III standard)
    pub const MIN_LEVERAGE_RATIO: f64 = 0.03;
}

/// Performance and timing constants
pub mod performance {
    use std::time::Duration;
    
    /// Maximum latency for HFT critical path operations
    pub const MAX_CRITICAL_PATH_LATENCY_NS: u64 = 14;
    
    /// Maximum acceptable latency for risk checks (microseconds)
    pub const MAX_RISK_CHECK_LATENCY_US: u64 = 50;
    
    /// Default batch processing size
    pub const DEFAULT_BATCH_SIZE: usize = 100;
    
    /// Ring buffer size for lock-free operations
    pub const RING_BUFFER_SIZE: usize = 4096;
    
    /// Small batch size for SIMD operations
    pub const SIMD_BATCH_SIZE: usize = 8;
}

/// Cache TTL defaults (can be overridden by runtime config)
pub mod cache_defaults {
    use std::time::Duration;
    
    /// Default TTL for position cache entries
    pub const POSITION_CACHE_TTL: Duration = Duration::from_secs(60);
    
    /// Default TTL for VaR calculation cache
    pub const VAR_CACHE_TTL: Duration = Duration::from_secs(3600);
    
    /// Default TTL for compliance check cache
    pub const COMPLIANCE_CACHE_TTL: Duration = Duration::from_secs(86400);
}

/// Database operation defaults
pub mod database_defaults {
    use std::time::Duration;
    
    /// Default query timeout for standard operations
    pub const QUERY_TIMEOUT: Duration = Duration::from_millis(1000);
    
    /// Default connection timeout
    pub const CONNECTION_TIMEOUT: Duration = Duration::from_millis(100);
    
    /// Default pool size
    pub const DEFAULT_POOL_SIZE: u32 = 20;
    
    /// Maximum pool size
    pub const MAX_POOL_SIZE: u32 = 100;
}

Phase 2: Extend Config Crate with Runtime Configuration (Week 1-2)

File: /home/jgrusewski/Work/foxhunt/config/src/runtime.rs (NEW)

//! Runtime configuration management
//!
//! Loads configuration from environment variables and provides
//! type-safe access to runtime configuration values.

use serde::{Deserialize, Serialize};
use std::time::Duration;

#[derive(Debug, Clone, PartialEq, Eq, Deserialize, Serialize)]
#[serde(rename_all = "lowercase")]
pub enum Environment {
    Development,
    Staging,
    Production,
}

#[derive(Debug, Clone, Deserialize)]
pub struct RuntimeConfig {
    #[serde(default = "Environment::default")]
    pub environment: Environment,
    
    #[serde(default)]
    pub timeouts: TimeoutConfig,
    
    #[serde(default)]
    pub cache: CacheConfig,
    
    #[serde(default)]
    pub performance: PerformanceConfig,
}

impl RuntimeConfig {
    /// Load runtime configuration from environment variables
    pub fn from_env() -> Result<Self, ConfigError> {
        envy::from_env().map_err(|e| ConfigError::InvalidConfiguration {
            reason: format!("Failed to load runtime config from environment: {}", e),
        })
    }
    
    /// Get configuration value with environment-specific defaults
    pub fn get_timeout(&self, key: &str) -> Duration {
        match key {
            "db_connection" => Duration::from_millis(self.timeouts.db_connection_timeout_ms),
            "db_query" => Duration::from_millis(self.timeouts.db_query_timeout_ms),
            "grpc_request" => Duration::from_millis(self.timeouts.grpc_request_timeout_ms),
            "auto_recovery" => Duration::from_secs(self.timeouts.auto_recovery_delay_secs),
            _ => Duration::from_secs(30), // Default fallback
        }
    }
}

#[derive(Debug, Clone, Deserialize)]
pub struct TimeoutConfig {
    #[serde(default = "TimeoutConfig::default_db_connection_timeout")]
    pub db_connection_timeout_ms: u64,
    
    #[serde(default = "TimeoutConfig::default_db_query_timeout")]
    pub db_query_timeout_ms: u64,
    
    #[serde(default = "TimeoutConfig::default_grpc_timeout")]
    pub grpc_request_timeout_ms: u64,
    
    #[serde(default = "TimeoutConfig::default_auto_recovery_delay")]
    pub auto_recovery_delay_secs: u64,
}

impl TimeoutConfig {
    fn default_db_connection_timeout() -> u64 {
        match std::env::var("ENVIRONMENT").as_deref() {
            Ok("production") => 100,
            Ok("staging") => 200,
            _ => 500,
        }
    }
    
    fn default_db_query_timeout() -> u64 {
        match std::env::var("ENVIRONMENT").as_deref() {
            Ok("production") => 800,
            Ok("staging") => 1000,
            _ => 2000,
        }
    }
    
    fn default_grpc_timeout() -> u64 {
        10000 // 10 seconds default
    }
    
    fn default_auto_recovery_delay() -> u64 {
        match std::env::var("ENVIRONMENT").as_deref() {
            Ok("production") => 1800, // 30 minutes
            Ok("staging") => 600,     // 10 minutes
            _ => 60,                  // 1 minute
        }
    }
}

impl Default for TimeoutConfig {
    fn default() -> Self {
        Self {
            db_connection_timeout_ms: Self::default_db_connection_timeout(),
            db_query_timeout_ms: Self::default_db_query_timeout(),
            grpc_request_timeout_ms: Self::default_grpc_timeout(),
            auto_recovery_delay_secs: Self::default_auto_recovery_delay(),
        }
    }
}

Phase 3: Database Configuration Schema (Week 2)

File: /home/jgrusewski/Work/foxhunt/database/schemas/005_runtime_config.sql (NEW)

See "Tier 3" section above for full schema.

Phase 4: Migration Guide (Week 3)

Priority Order for Refactoring:

1. High Priority (Week 3):

   • Breach thresholds in risk-data/src/limits.rs
   • Environment-specific durations in risk/src/lib.rs
   • Cache TTL values in all *-data crates

2. Medium Priority (Week 4):

   • Database connection strings (migrate to env vars)
   • Redis TTL values
   • Timeout values in services

3. Low Priority (Week 5):

   • Test-only constants
   • Performance tuning constants (can remain compile-time)

Migration Example

Before (Hardcoded):

// risk-data/src/limits.rs:376-382
if breach_percentage >= Decimal::from(120) {
    BreachSeverity::Critical
} else if breach_percentage >= Decimal::from(100) {
    BreachSeverity::Hard
} else if breach_percentage >= Decimal::from(90) {
    BreachSeverity::Soft
}

After (Database-Configured):

// risk-data/src/limits.rs
use config::RuntimeThresholds;

impl PositionLimitRepository {
    async fn determine_breach_severity(
        &self,
        breach_percentage: Decimal,
    ) -> RiskDataResult<BreachSeverity> {
        // Load thresholds from database (cached)
        let thresholds = self.config_loader
            .get_runtime_thresholds("breach")
            .await?;
        
        let critical = thresholds.get_decimal("critical_threshold_pct")?;
        let hard = thresholds.get_decimal("hard_threshold_pct")?;
        let soft = thresholds.get_decimal("soft_threshold_pct")?;
        
        if breach_percentage >= critical {
            Ok(BreachSeverity::Critical)
        } else if breach_percentage >= hard {
            Ok(BreachSeverity::Hard)
        } else if breach_percentage >= soft {
            Ok(BreachSeverity::Soft)
        } else {
            Ok(BreachSeverity::Warning)
        }
    }
}

Testing Strategy

1. Unit Tests with Config Overrides

#[test]
fn test_breach_severity_custom_thresholds() {
    let config = TestRuntimeThresholds::builder()
        .set("breach.critical_threshold_pct", 150.0)
        .set("breach.hard_threshold_pct", 125.0)
        .set("breach.soft_threshold_pct", 100.0)
        .build();
    
    let repo = PositionLimitRepository::with_config(config);
    
    assert_eq!(
        repo.determine_breach_severity(Decimal::from(130)).await?,
        BreachSeverity::Hard
    );
}

2. Integration Tests with Environment Configs

#[tokio::test]
async fn test_environment_specific_timeouts() {
    std::env::set_var("ENVIRONMENT", "production");
    let config = RuntimeConfig::from_env()?;
    
    assert_eq!(config.timeouts.db_connection_timeout_ms, 100);
    assert_eq!(config.timeouts.auto_recovery_delay_secs, 1800);
}

Deliverables Summary

1. New Files to Create:

• /home/jgrusewski/Work/foxhunt/common/src/thresholds.rs
• /home/jgrusewski/Work/foxhunt/config/src/runtime.rs
• /home/jgrusewski/Work/foxhunt/database/schemas/005_runtime_config.sql
• /home/jgrusewski/Work/foxhunt/.env.development.example
• /home/jgrusewski/Work/foxhunt/.env.production.example
• /home/jgrusewski/Work/foxhunt/docs/CONFIGURATION_GUIDE.md

2. Files to Modify:

• risk-data/src/limits.rs - Use runtime thresholds for breach severity
• risk/src/lib.rs - Migrate environment-specific durations
• risk-data/src/compliance.rs - Migrate Redis TTL values
• risk-data/src/var.rs - Migrate cache TTL values
• All service main.rs files - Load RuntimeConfig

3. Documentation:

• Configuration guide for operators
• Migration guide for developers
• Environment variable reference
• Database configuration reference

Risk Assessment

Low Risk:

• Adding new constants modules (non-breaking)
• Creating database schema for runtime config (additive)

Medium Risk:

• Migrating hardcoded values to environment variables (requires deployment coordination)
• Changing breach threshold logic (requires thorough testing)

Mitigation:

• Gradual migration with feature flags
• Comprehensive test coverage before migration
• Fallback to hardcoded defaults if config unavailable
• Detailed logging of configuration sources

Operational Benefits

1. Flexibility: Adjust thresholds without code deployment
2. Environment Parity: Consistent config management across dev/staging/prod
3. Auditability: Track configuration changes in database
4. Testing: Override configs easily in tests
5. Debugging: Clear source of truth for configuration values

Conclusion

The codebase has extensive hardcoded configuration scattered across 100+ files. Implementing the 3-tier architecture (compile-time constants, runtime config, database config) will significantly improve maintainability and operational flexibility while maintaining the performance-critical nature of HFT operations.

Estimated Effort: 3-4 weeks for complete migration Priority: Medium (improves maintainability but not blocking production) Dependencies: None (can be done incrementally)