Files
foxhunt/common/src/thresholds.rs
jgrusewski a2d1eacce6 🚀 Wave 66: Production Readiness - 12 Parallel Agents Complete
## Overview
Deployed 12 parallel agents to resolve critical production blockers across authentication,
configuration, ML pipeline, testing, and system optimization. All core objectives achieved.

## 🔐 Authentication & Security (Agents 1-2)
### Agent 1: Tonic 0.14 Authentication Compatibility 
- Migrated from Tower Service middleware to Tonic's native Interceptor
- Fixed Error = Infallible incompatibility with Tonic 0.14
- Re-enabled authentication across all gRPC services
- Maintains JWT, mTLS, rate limiting, RBAC, and audit trails
- Files: trading_service/src/{auth_interceptor.rs, main.rs}

### Agent 2: Postgres Feature Flag 
- Added missing 'postgres' feature to adaptive-strategy/Cargo.toml
- Resolved 9 warnings about unexpected cfg conditions
- Properly gated all postgres-dependent code
- Files: adaptive-strategy/{Cargo.toml, src/database_loader.rs, src/lib.rs}

## 🤖 ML & Data Pipeline (Agents 3, 5, 7)
### Agent 3: ML Performance Monitoring Foundation 
- Created ml_metrics.rs with 12 Prometheus metrics
- Designed integration plan for MLPerformanceMonitor and MLFallbackManager
- Added prometheus dependency to trading_service
- Files: trading_service/src/{lib.rs, ml_metrics.rs}, Cargo.toml
- Docs: WAVE_66_AGENT_3_IMPLEMENTATION.md

### Agent 5: Mock Data Feature Removal 
- Fixed module import issues in ml_training_service
- Removed mock-data from default features (production uses real data)
- Updated README with feature flag documentation
- Files: ml_training_service/{Cargo.toml, src/main.rs, README.md}

### Agent 7: Advanced Feature Extraction 
- Implemented technical indicators (RSI, MACD, EMA, Bollinger, ATR)
- Created stateful TechnicalIndicatorCalculator (566 lines)
- Integrated with data_loader for real ML features
- Unblocked ML training pipeline
- Files: ml_training_service/src/{technical_indicators.rs, data_loader.rs, lib.rs}

## ⚙️ Configuration & Testing (Agents 4, 6, 11, 12)
### Agent 4: E2E Test Proto Fixes 
- Fixed namespace collision from wildcard proto imports
- Resolved 9 compilation errors (5 ambiguity + 4 API mismatches)
- Updated for Tonic 0.14 API changes
- Files: tests/e2e/src/workflows.rs

### Agent 6: Config Phase 4 - Integration Tests 
- Created 25 comprehensive integration tests
- Hot-reload verification with PostgreSQL NOTIFY/LISTEN
- ACID transaction testing (atomicity, consistency, isolation, durability)
- Concurrent update handling and performance benchmarks
- Files: adaptive-strategy/tests/hot_reload_integration.rs
- Docs: adaptive-strategy/{PHASE4_COMPLETION.md, docs/hot_reload_testing.md}

### Agent 11: Magic Numbers Centralization 
- Analyzed 500+ hardcoded values across 100+ files
- Created centralized thresholds module (450 lines, 15 sub-modules)
- Environment configuration templates (.env.{development,production}.example)
- 3-tier configuration architecture designed
- Files: common/src/thresholds.rs, .env.*.example
- Docs: WAVE_66_AGENT_11_{ANALYSIS,DELIVERABLES,SUMMARY}.md
- Docs: docs/CONFIGURATION_QUICK_REFERENCE.md

### Agent 12: Test Suite Execution 
- Executed 418 core tests with 100% pass rate
- Verified trading_engine (281 tests), adaptive-strategy (69 tests), common (68 tests)
- Production readiness assessment completed
- Fixed test compilation issues in data/tests/comprehensive_coverage_tests.rs
- Docs: docs/wave66_agent12_test_report.md

## 📊 System Optimization (Agents 8-10)
### Agent 8: Database Pooling Analysis 
- Identified critical 30s timeout in ML training service
- Inconsistent pool sizing across services
- Insufficient statement cache (backtesting 100 → 500)
- HFT-optimized configurations designed
- Comprehensive analysis documented (no code changes - design phase)

### Agent 9: gRPC Streaming Analysis 
- Critical HTTP/2 optimization opportunities identified
- tcp_nodelay(true) for -40ms latency reduction
- Stream-specific buffer sizing (1K → 100K for market data)
- Backpressure monitoring design
- 4-week implementation roadmap created

### Agent 10: Metrics Aggregation Analysis 
- Critical cardinality explosion identified (100K+ potential time series)
- Unbounded memory growth in HDR histograms
- Asset class bucketing strategy designed (99% cardinality reduction)
- LRU caching for bounded memory
- 5-phase optimization plan documented

## 📈 Impact Summary
-  Authentication fully operational with Tonic 0.14
-  ML training pipeline unblocked (real features, not mock data)
-  Configuration hot-reload fully tested (25 integration tests)
-  418 core tests passing (100% pass rate)
-  Production deployment foundation complete
-  Comprehensive optimization roadmaps for Waves 67-70

## 🔧 Files Changed (29 total)
Modified: 17 files across services, crates, and tests
Created: 12 new files (modules, tests, documentation)

## 🎯 Next Steps (Wave 67+)
- Implement Agent 8-10 optimization plans
- Complete ML monitoring integration (Agent 3)
- Execute configuration centralization migration
- Performance validation and load testing

🤖 Generated with Claude Code
Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-03 08:09:52 +02:00

489 lines
15 KiB
Rust

//! Centralized threshold constants for the Foxhunt HFT system
//!
//! This module consolidates all hardcoded threshold values that were
//! previously scattered throughout the codebase. Constants here are
//! compile-time values for performance-critical operations.
//!
//! For runtime-configurable values, see the `config` crate's runtime module.
use rust_decimal::Decimal;
use std::time::Duration;
/// Risk management thresholds
pub mod risk {
use super::*;
/// Breach severity warning threshold (percentage of limit)
/// Used when position is at 80-90% of limit
pub const BREACH_WARNING_PCT: u8 = 80;
/// Breach severity soft threshold (percentage of limit)
/// Used when position is at 90-100% of limit
pub const BREACH_SOFT_PCT: u8 = 90;
/// Breach severity hard threshold (percentage of limit)
/// Used when position is at 100-120% of limit
pub const BREACH_HARD_PCT: u8 = 100;
/// Breach severity critical threshold (percentage of limit)
/// Used when position exceeds 120% of limit
pub const BREACH_CRITICAL_PCT: u8 = 120;
/// Minimum capital adequacy ratio (Basel III standard)
pub const MIN_CAPITAL_ADEQUACY_RATIO: f64 = 0.08;
/// Minimum leverage ratio (Basel III standard)
pub const MIN_LEVERAGE_RATIO: f64 = 0.03;
/// Default VaR confidence level (95%)
pub const DEFAULT_VAR_CONFIDENCE: f64 = 0.95;
/// High VaR confidence level (99%)
pub const HIGH_VAR_CONFIDENCE: f64 = 0.99;
/// Maximum drawdown warning threshold (percentage)
pub const MAX_DRAWDOWN_WARNING_PCT: u8 = 15;
/// Maximum drawdown critical threshold (percentage)
pub const MAX_DRAWDOWN_CRITICAL_PCT: u8 = 25;
}
/// VaR calculation constants
pub mod var {
/// Z-score for 90% confidence level
pub const Z_SCORE_P90: f64 = 1.282;
/// Z-score for 95% confidence level
pub const Z_SCORE_P95: f64 = 1.645;
/// Z-score for 97.5% confidence level
pub const Z_SCORE_P97_5: f64 = 1.96;
/// Z-score for 99% confidence level
pub const Z_SCORE_P99: f64 = 2.326;
/// Z-score for 99.9% confidence level
pub const Z_SCORE_P99_9: f64 = 3.09;
/// Default lookback period for historical VaR (trading days)
pub const DEFAULT_LOOKBACK_DAYS: usize = 252;
/// Minimum data quality score for VaR calculation
pub const MIN_DATA_QUALITY_SCORE: f64 = 0.6;
}
/// Performance and timing constants
pub mod performance {
use super::Duration;
/// Maximum latency for HFT critical path operations (nanoseconds)
pub const MAX_CRITICAL_PATH_LATENCY_NS: u64 = 14;
/// Maximum acceptable latency for risk checks (microseconds)
pub const MAX_RISK_CHECK_LATENCY_US: u64 = 50;
/// Maximum latency for ML inference (microseconds)
pub const MAX_ML_INFERENCE_LATENCY_US: u64 = 100;
/// Default batch processing size
pub const DEFAULT_BATCH_SIZE: usize = 100;
/// Ring buffer size for lock-free operations
pub const RING_BUFFER_SIZE: usize = 4096;
/// Small batch size for SIMD operations
pub const SIMD_BATCH_SIZE: usize = 8;
/// Maximum small batch size
pub const MAX_SMALL_BATCH_SIZE: usize = 10;
/// Default worker thread count (adjusted based on CPU cores at runtime)
pub const DEFAULT_WORKER_THREADS: usize = 4;
/// Default queue capacity for async operations
pub const DEFAULT_QUEUE_CAPACITY: usize = 10000;
}
/// Cache TTL defaults (can be overridden by runtime config)
pub mod cache {
use super::Duration;
/// Default TTL for position cache entries (1 minute)
pub const POSITION_CACHE_TTL: Duration = Duration::from_secs(60);
/// Default TTL for VaR calculation cache (1 hour)
pub const VAR_CACHE_TTL: Duration = Duration::from_secs(3600);
/// Default TTL for compliance check cache (24 hours)
pub const COMPLIANCE_CACHE_TTL: Duration = Duration::from_secs(86400);
/// Default TTL for market data cache (5 minutes)
pub const MARKET_DATA_CACHE_TTL: Duration = Duration::from_secs(300);
/// Default TTL for model predictions cache (1 minute)
pub const MODEL_PREDICTION_CACHE_TTL: Duration = Duration::from_secs(60);
/// Redis key TTL for position limits (5 minutes)
pub const REDIS_POSITION_LIMIT_TTL_SECS: i32 = 300;
/// Redis key TTL for compliance checks (24 hours)
pub const REDIS_COMPLIANCE_TTL_SECS: i32 = 86400;
/// Redis key TTL for VaR calculations (1 hour)
pub const REDIS_VAR_TTL_SECS: i32 = 3600;
}
/// Database operation defaults
pub mod database {
use super::Duration;
/// Default query timeout for standard operations
pub const QUERY_TIMEOUT: Duration = Duration::from_millis(1000);
/// Default connection timeout
pub const CONNECTION_TIMEOUT: Duration = Duration::from_millis(100);
/// Default pool acquire timeout
pub const ACQUIRE_TIMEOUT: Duration = Duration::from_millis(50);
/// Default connection lifetime (1 hour)
pub const CONNECTION_LIFETIME: Duration = Duration::from_secs(3600);
/// Default idle timeout (5 minutes)
pub const IDLE_TIMEOUT: Duration = Duration::from_secs(300);
/// Default pool size
pub const DEFAULT_POOL_SIZE: u32 = 20;
/// Maximum pool size
pub const MAX_POOL_SIZE: u32 = 100;
/// Maximum query result limit
pub const MAX_QUERY_LIMIT: i64 = 1000;
}
/// Network and gRPC defaults
pub mod network {
use super::Duration;
/// Default connect timeout for gRPC clients
pub const GRPC_CONNECT_TIMEOUT: Duration = Duration::from_secs(5);
/// Default request timeout for gRPC
pub const GRPC_REQUEST_TIMEOUT: Duration = Duration::from_secs(10);
/// Default keep-alive interval
pub const KEEP_ALIVE_INTERVAL: Duration = Duration::from_secs(30);
/// Keep-alive timeout
pub const KEEP_ALIVE_TIMEOUT: Duration = Duration::from_secs(5);
/// Maximum concurrent connections
pub const MAX_CONCURRENT_CONNECTIONS: u32 = 100;
/// HTTP/2 initial stream window size
pub const INITIAL_STREAM_WINDOW_SIZE: u32 = 65535;
/// HTTP/2 initial connection window size
pub const INITIAL_CONNECTION_WINDOW_SIZE: u32 = 1048576;
}
/// Retry and recovery defaults
pub mod retry {
use super::Duration;
/// Initial delay for exponential backoff
pub const INITIAL_RETRY_DELAY: Duration = Duration::from_millis(100);
/// Maximum delay for exponential backoff
pub const MAX_RETRY_DELAY: Duration = Duration::from_secs(30);
/// Maximum retry attempts for critical operations
pub const MAX_RETRY_ATTEMPTS: u32 = 3;
/// Backoff multiplier for exponential backoff
pub const BACKOFF_MULTIPLIER: f32 = 1.5;
/// Maximum total duration for retry attempts
pub const MAX_TOTAL_RETRY_DURATION: Duration = Duration::from_secs(60);
}
/// Health check and monitoring intervals
pub mod monitoring {
use super::Duration;
/// Default health check interval
pub const HEALTH_CHECK_INTERVAL: Duration = Duration::from_secs(30);
/// Default metrics collection interval
pub const METRICS_COLLECTION_INTERVAL: Duration = Duration::from_secs(10);
/// Default log flush interval
pub const LOG_FLUSH_INTERVAL: Duration = Duration::from_secs(5);
/// Circuit breaker check interval
pub const CIRCUIT_BREAKER_CHECK_INTERVAL: Duration = Duration::from_millis(100);
/// Kill switch session timeout (5 minutes)
pub const KILL_SWITCH_SESSION_TIMEOUT: Duration = Duration::from_secs(300);
}
/// Event processing defaults
pub mod events {
use super::Duration;
/// Event batch timeout
pub const BATCH_TIMEOUT: Duration = Duration::from_millis(100);
/// Event batch size
pub const BATCH_SIZE: usize = 100;
/// Event retry delay
pub const RETRY_DELAY: Duration = Duration::from_millis(50);
/// Maximum event backlog before applying backpressure
pub const MAX_EVENT_BACKLOG: usize = 10000;
/// Maximum span buffer size for tracing
pub const MAX_SPAN_BUFFER_SIZE: usize = 100_000;
/// Span export batch size
pub const SPAN_EXPORT_BATCH_SIZE: usize = 1000;
}
/// ML model constants
pub mod ml {
use super::Duration;
/// Maximum GPU batch size
pub const MAX_GPU_BATCH_SIZE: usize = 8192;
/// Maximum CPU batch size
pub const MAX_CPU_BATCH_SIZE: usize = 1024;
/// Default model cache cleanup interval (1 hour)
pub const MODEL_CACHE_CLEANUP_INTERVAL: Duration = Duration::from_secs(3600);
/// Default model health check interval (30 seconds)
pub const MODEL_HEALTH_CHECK_INTERVAL: Duration = Duration::from_secs(30);
/// Model deployment stage timeout (5 minutes)
pub const DEPLOYMENT_STAGE_TIMEOUT: Duration = Duration::from_secs(300);
/// Model deployment total timeout (30 minutes)
pub const DEPLOYMENT_TOTAL_TIMEOUT: Duration = Duration::from_secs(1800);
/// Model validation scan timeout (10 minutes)
pub const VALIDATION_SCAN_TIMEOUT: Duration = Duration::from_secs(600);
/// Canary deployment duration (5 minutes)
pub const CANARY_DURATION: Duration = Duration::from_secs(300);
/// Model rollback timeout (1 minute)
pub const ROLLBACK_TIMEOUT: Duration = Duration::from_secs(60);
/// Drift detection check interval (5 minutes)
pub const DRIFT_CHECK_INTERVAL: Duration = Duration::from_secs(300);
/// Drift detection warning threshold
pub const DRIFT_WARNING_THRESHOLD: f64 = 0.05;
/// Maximum recommendation age for Kelly sizing (1 minute)
pub const MAX_KELLY_RECOMMENDATION_AGE: Duration = Duration::from_secs(60);
/// Kelly sizing cache TTL (5 minutes)
pub const KELLY_CACHE_TTL: Duration = Duration::from_secs(300);
}
/// Safety system defaults
pub mod safety {
use super::Duration;
/// Safety check timeout for production (5ms)
pub const PRODUCTION_SAFETY_CHECK_TIMEOUT: Duration = Duration::from_millis(5);
/// Safety check timeout for development (50ms)
pub const DEVELOPMENT_SAFETY_CHECK_TIMEOUT: Duration = Duration::from_millis(50);
/// Auto-recovery delay for production (30 minutes)
pub const PRODUCTION_AUTO_RECOVERY_DELAY: Duration = Duration::from_secs(1800);
/// Auto-recovery delay for development (1 minute)
pub const DEVELOPMENT_AUTO_RECOVERY_DELAY: Duration = Duration::from_secs(60);
/// Loss check interval for production (5 seconds)
pub const PRODUCTION_LOSS_CHECK_INTERVAL: Duration = Duration::from_secs(5);
/// Loss check interval for development (30 seconds)
pub const DEVELOPMENT_LOSS_CHECK_INTERVAL: Duration = Duration::from_secs(30);
/// Position check interval for production (2 seconds)
pub const PRODUCTION_POSITION_CHECK_INTERVAL: Duration = Duration::from_secs(2);
/// Position check interval for development (15 seconds)
pub const DEVELOPMENT_POSITION_CHECK_INTERVAL: Duration = Duration::from_secs(15);
/// Memory check interval
pub const MEMORY_CHECK_INTERVAL: Duration = Duration::from_secs(1);
/// Circuit breaker trip cooldown (30 seconds)
pub const CIRCUIT_BREAKER_COOLDOWN: Duration = Duration::from_secs(30);
}
/// Time conversion constants
pub mod time {
/// Nanoseconds per microsecond
pub const NANOS_PER_MICRO: u64 = 1_000;
/// Nanoseconds per millisecond
pub const NANOS_PER_MILLI: u64 = 1_000_000;
/// Nanoseconds per second
pub const NANOS_PER_SECOND: u64 = 1_000_000_000;
/// Microseconds per second
pub const MICROS_PER_SECOND: u64 = 1_000_000;
/// Milliseconds per second
pub const MILLIS_PER_SECOND: u64 = 1_000;
/// Seconds per minute
pub const SECONDS_PER_MINUTE: u64 = 60;
/// Seconds per hour
pub const SECONDS_PER_HOUR: u64 = 3600;
/// Seconds per day
pub const SECONDS_PER_DAY: u64 = 86400;
/// Trading days per year
pub const TRADING_DAYS_PER_YEAR: usize = 252;
}
/// Financial constants
pub mod financial {
/// Basis points per unit
pub const BASIS_POINTS_PER_UNIT: u32 = 10_000;
/// Cents per dollar
pub const CENTS_PER_DOLLAR: u32 = 100;
/// Default profit target in basis points (1%)
pub const DEFAULT_PROFIT_TARGET_BPS: u32 = 100;
/// Default stop loss in basis points (0.5%)
pub const DEFAULT_STOP_LOSS_BPS: u32 = 50;
/// Minimum return threshold in basis points
pub const MIN_RETURN_THRESHOLD_BPS: i32 = 5;
/// Price scaling factor (6 decimal places)
pub const PRICE_SCALE: i64 = 1_000_000;
/// Quantity scaling factor (6 decimal places)
pub const QUANTITY_SCALE: i64 = 1_000_000;
/// Money scaling factor (6 decimal places)
pub const MONEY_SCALE: i64 = 1_000_000;
/// Unified scaling factor for all financial operations
pub const UNIFIED_SCALE_FACTOR: i64 = 1_000_000;
/// ML precision factor (8 decimal places)
pub const PRECISION_FACTOR: i64 = 100_000_000;
/// VPIN precision factor (4 decimal places)
pub const VPIN_PRECISION_FACTOR: i64 = 10_000;
}
/// Validation limits
pub mod limits {
/// Maximum symbol length
pub const MAX_SYMBOL_LENGTH: usize = 12;
/// Maximum account ID length
pub const MAX_ACCOUNT_ID_LENGTH: usize = 32;
/// Maximum description length
pub const MAX_DESCRIPTION_LENGTH: usize = 256;
/// Maximum metadata key length
pub const MAX_METADATA_KEY_LENGTH: usize = 64;
/// Maximum metadata value length
pub const MAX_METADATA_VALUE_LENGTH: usize = 512;
/// Maximum metadata entries
pub const MAX_METADATA_ENTRIES: usize = 100;
/// Maximum price value
pub const MAX_PRICE: f64 = 1_000_000.0;
/// Minimum price value
pub const MIN_PRICE: f64 = 0.000_001;
/// Maximum quantity value
pub const MAX_QUANTITY: f64 = 1_000_000_000.0;
/// Minimum quantity value
pub const MIN_QUANTITY: f64 = 0.000_001;
/// Maximum leverage
pub const MAX_LEVERAGE: f64 = 1000.0;
/// Minimum leverage
pub const MIN_LEVERAGE: f64 = 0.1;
/// Maximum allocation size (1GB)
pub const MAX_ALLOCATION_SIZE: usize = 1024 * 1024 * 1024;
/// Maximum duration in milliseconds (24 hours)
pub const MAX_DURATION_MILLIS: u64 = 24 * 60 * 60 * 1000;
}
/// Hardware alignment constants
pub mod hardware {
/// CPU cache line size
pub const CACHE_LINE_SIZE: usize = 64;
/// SIMD alignment for AVX2
pub const SIMD_ALIGNMENT: usize = 32;
/// Page size (4KB)
pub const PAGE_SIZE: usize = 4096;
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_breach_thresholds_ordered() {
assert!(risk::BREACH_WARNING_PCT < risk::BREACH_SOFT_PCT);
assert!(risk::BREACH_SOFT_PCT < risk::BREACH_HARD_PCT);
assert!(risk::BREACH_HARD_PCT < risk::BREACH_CRITICAL_PCT);
}
#[test]
fn test_var_z_scores_ordered() {
assert!(var::Z_SCORE_P90 < var::Z_SCORE_P95);
assert!(var::Z_SCORE_P95 < var::Z_SCORE_P97_5);
assert!(var::Z_SCORE_P97_5 < var::Z_SCORE_P99);
assert!(var::Z_SCORE_P99 < var::Z_SCORE_P99_9);
}
#[test]
fn test_time_conversions() {
assert_eq!(time::NANOS_PER_MICRO * 1000, time::NANOS_PER_MILLI);
assert_eq!(time::NANOS_PER_MILLI * 1000, time::NANOS_PER_SECOND);
assert_eq!(time::MICROS_PER_SECOND * 1000, time::NANOS_PER_SECOND);
}
#[test]
fn test_financial_scales_consistent() {
assert_eq!(financial::PRICE_SCALE, financial::UNIFIED_SCALE_FACTOR);
assert_eq!(financial::QUANTITY_SCALE, financial::UNIFIED_SCALE_FACTOR);
assert_eq!(financial::MONEY_SCALE, financial::UNIFIED_SCALE_FACTOR);
}
}