Files
foxhunt/common/src/thresholds.rs
jgrusewski 3799c04064 🎯 Wave 159: Fix ML Training Infrastructure (22 Parallel Agents)
Critical Discovery: Training scripts used benchmark tool instead of trainers
- No .safetensors model files were being saved
- Fixed by creating real training examples with checkpoint callbacks

## Training Infrastructure Fixed (Agents 1-24)

### Root Cause Identified (Agent 1-2)
- scripts/train_all_models_full.sh used gpu_training_benchmark (benchmark only)
- Benchmarks measure performance but DO NOT save models
- Created 4 new training examples with proper model persistence

### Module Exports Fixed (Agents 3-6)
- ml/src/trainers/mod.rs: Added DQN module export
- All trainer types now accessible: DQNTrainer, PPOTrainer, Mamba2Trainer, TFTTrainer

### Training Examples Created (Agents 7-14)
- ml/examples/train_dqn.rs (170 lines) - DQN with Experience replay
- ml/examples/train_ppo.rs (140 lines) - PPO with GAE
- ml/examples/train_mamba2.rs (210 lines) - MAMBA-2 with state space
- ml/examples/train_tft.rs (250 lines) - TFT with temporal fusion

### Trainer Bugs Fixed (Agents 11, 23)
- ml/src/trainers/dqn.rs: Fixed Experience initialization (timestamp, type conversions)
- ml/src/trainers/ppo.rs: Fixed tensor shape mismatches (flatten before scalar)
- ml/src/trainers/dqn.rs: Fixed epsilon type conversion (f64 → f32 cast)

### E2E Test Infrastructure (Agents 15-18, TDD Approach)
- tests/e2e/tests/dqn_training_test.rs (369 lines) - 2/2 passing
- tests/e2e/tests/ppo_training_test.rs (512 lines) - Comprehensive validation
- tests/e2e/tests/mamba2_training_test.rs (459 lines) - gRPC integration
- tests/e2e/tests/tft_training_test.rs (616 lines) - Progress streaming

### Scripts & Validation (Agents 19-20)
- scripts/train_all_models_fixed.sh - Uses real trainers
- scripts/validate_training.sh (268 lines) - Quick validation
- scripts/test_dqn_training.sh - Individual model testing

### API Documentation (Agents 7-10)
- TRAINING_GUIDE.md - Comprehensive training guide
- docs/AGENT_19_TRAINING_SCRIPT_VALIDATION.md - Script validation
- 200+ pages of trainer API documentation

## Technical Achievements

### Performance
- DQN Experience constructor: Proper type handling
- PPO tensor operations: .flatten_all()?.to_vec1::<f32>()?[0]
- GPU memory optimization: Batch size limits for RTX 3050 Ti (4GB)

### Architecture
- Checkpoint callbacks: |epoch, model_data| → .safetensors files
- Real-time progress streaming: tokio::sync::mpsc channels
- E2E testing: Fast iteration without Docker rebuilds

### Production Readiness
- Module exports: 100% 
- Training examples: 100%  (all compile and run)
- E2E tests: 100%  (4 comprehensive test suites)
- Build status: 100%  (zero compilation errors)

## Files Modified: 50+
- Core trainers: dqn.rs, ppo.rs, mamba2.rs, tft.rs
- Module exports: mod.rs
- Training examples: 4 new files (770 lines total)
- E2E tests: 4 new files (1956 lines total)
- Scripts: 5 new validation scripts
- Documentation: 7 new docs (100K+ words)

## Tests Created: 8 E2E Tests
- DQN: Checkpoint creation, model loading
- PPO: Training metrics, convergence
- MAMBA-2: State space validation, gRPC
- TFT: Temporal fusion, progress streaming

Status:  Ready for model training (500 epochs per model)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-14 09:06:37 +02:00

494 lines
15 KiB
Rust

//! Centralized threshold constants for the Foxhunt HFT system
//!
//! This module consolidates all hardcoded threshold values that were
//! previously scattered throughout the codebase. Constants here are
//! compile-time values for performance-critical operations.
//!
//! For runtime-configurable values, see the `config` crate's runtime module.
use std::time::Duration;
/// Risk management thresholds
pub mod risk {
/// Breach severity warning threshold (percentage of limit)
///
/// Used when position is at 80-90% of limit
pub const BREACH_WARNING_PCT: u8 = 80;
/// Breach severity soft threshold (percentage of limit)
///
/// Used when position is at 90-100% of limit
pub const BREACH_SOFT_PCT: u8 = 90;
/// Breach severity hard threshold (percentage of limit)
///
/// Used when position is at 100-120% of limit
pub const BREACH_HARD_PCT: u8 = 100;
/// Breach severity critical threshold (percentage of limit)
///
/// Used when position exceeds 120% of limit
pub const BREACH_CRITICAL_PCT: u8 = 120;
/// Minimum capital adequacy ratio (Basel III standard)
pub const MIN_CAPITAL_ADEQUACY_RATIO: f64 = 0.08;
/// Minimum leverage ratio (Basel III standard)
pub const MIN_LEVERAGE_RATIO: f64 = 0.03;
/// Default `VaR` confidence level (95%)
pub const DEFAULT_VAR_CONFIDENCE: f64 = 0.95;
/// High `VaR` confidence level (99%)
pub const HIGH_VAR_CONFIDENCE: f64 = 0.99;
/// Maximum drawdown warning threshold (percentage)
pub const MAX_DRAWDOWN_WARNING_PCT: u8 = 15;
/// Maximum drawdown critical threshold (percentage)
pub const MAX_DRAWDOWN_CRITICAL_PCT: u8 = 25;
}
/// `VaR` calculation constants
pub mod var {
/// Z-score for 90% confidence level
pub const Z_SCORE_P90: f64 = 1.282;
/// Z-score for 95% confidence level
pub const Z_SCORE_P95: f64 = 1.645;
/// Z-score for 97.5% confidence level
pub const Z_SCORE_P97_5: f64 = 1.96;
/// Z-score for 99% confidence level
pub const Z_SCORE_P99: f64 = 2.326;
/// Z-score for 99.9% confidence level
pub const Z_SCORE_P99_9: f64 = 3.09;
/// Default lookback period for historical `VaR` (trading days)
pub const DEFAULT_LOOKBACK_DAYS: usize = 252;
/// Minimum data quality score for `VaR` calculation
pub const MIN_DATA_QUALITY_SCORE: f64 = 0.6;
}
/// Performance and timing constants
pub mod performance {
/// Maximum latency for HFT critical path operations (nanoseconds)
pub const MAX_CRITICAL_PATH_LATENCY_NS: u64 = 14;
/// Maximum acceptable latency for risk checks (microseconds)
pub const MAX_RISK_CHECK_LATENCY_US: u64 = 50;
/// Maximum latency for ML inference (microseconds)
pub const MAX_ML_INFERENCE_LATENCY_US: u64 = 100;
/// Default batch processing size
pub const DEFAULT_BATCH_SIZE: usize = 100;
/// Ring buffer size for lock-free operations
pub const RING_BUFFER_SIZE: usize = 4096;
/// Small batch size for SIMD operations
pub const SIMD_BATCH_SIZE: usize = 8;
/// Maximum small batch size
pub const MAX_SMALL_BATCH_SIZE: usize = 10;
/// Default worker thread count (adjusted based on CPU cores at runtime)
pub const DEFAULT_WORKER_THREADS: usize = 4;
/// Default queue capacity for async operations
pub const DEFAULT_QUEUE_CAPACITY: usize = 10000;
}
/// Cache TTL defaults (can be overridden by runtime config)
pub mod cache {
use super::Duration;
/// Default TTL for position cache entries (1 minute)
pub const POSITION_CACHE_TTL: Duration = Duration::from_secs(60);
/// Default TTL for `VaR` calculation cache (1 hour)
pub const VAR_CACHE_TTL: Duration = Duration::from_secs(3600);
/// Default TTL for compliance check cache (24 hours)
pub const COMPLIANCE_CACHE_TTL: Duration = Duration::from_secs(86400);
/// Default TTL for market data cache (5 minutes)
pub const MARKET_DATA_CACHE_TTL: Duration = Duration::from_secs(300);
/// Default TTL for model predictions cache (1 minute)
pub const MODEL_PREDICTION_CACHE_TTL: Duration = Duration::from_secs(60);
/// Redis key TTL for position limits (5 minutes)
pub const REDIS_POSITION_LIMIT_TTL_SECS: i32 = 300;
/// Redis key TTL for compliance checks (24 hours)
pub const REDIS_COMPLIANCE_TTL_SECS: i32 = 86400;
/// Redis key TTL for `VaR` calculations (1 hour)
pub const REDIS_VAR_TTL_SECS: i32 = 3600;
}
/// Database operation defaults
pub mod database {
use super::Duration;
/// Default query timeout for standard operations
pub const QUERY_TIMEOUT: Duration = Duration::from_millis(1000);
/// Default connection timeout
pub const CONNECTION_TIMEOUT: Duration = Duration::from_millis(100);
/// Default pool acquire timeout
pub const ACQUIRE_TIMEOUT: Duration = Duration::from_millis(50);
/// Default connection lifetime (1 hour)
pub const CONNECTION_LIFETIME: Duration = Duration::from_secs(3600);
/// Default idle timeout (5 minutes)
pub const IDLE_TIMEOUT: Duration = Duration::from_secs(300);
/// Default pool size
pub const DEFAULT_POOL_SIZE: u32 = 20;
/// Maximum pool size
pub const MAX_POOL_SIZE: u32 = 100;
/// Maximum query result limit
pub const MAX_QUERY_LIMIT: i64 = 1000;
}
/// Network and gRPC defaults
pub mod network {
use super::Duration;
/// Default connect timeout for gRPC clients
pub const GRPC_CONNECT_TIMEOUT: Duration = Duration::from_secs(5);
/// Default request timeout for gRPC
pub const GRPC_REQUEST_TIMEOUT: Duration = Duration::from_secs(10);
/// Default keep-alive interval
pub const KEEP_ALIVE_INTERVAL: Duration = Duration::from_secs(30);
/// Keep-alive timeout
pub const KEEP_ALIVE_TIMEOUT: Duration = Duration::from_secs(5);
/// Maximum concurrent connections
pub const MAX_CONCURRENT_CONNECTIONS: u32 = 100;
/// HTTP/2 initial stream window size
pub const INITIAL_STREAM_WINDOW_SIZE: u32 = 65535;
/// HTTP/2 initial connection window size
pub const INITIAL_CONNECTION_WINDOW_SIZE: u32 = 1048576;
}
/// Retry and recovery defaults
pub mod retry {
use super::Duration;
/// Initial delay for exponential backoff
pub const INITIAL_RETRY_DELAY: Duration = Duration::from_millis(100);
/// Maximum delay for exponential backoff
pub const MAX_RETRY_DELAY: Duration = Duration::from_secs(30);
/// Maximum retry attempts for critical operations
pub const MAX_RETRY_ATTEMPTS: u32 = 3;
/// Backoff multiplier for exponential backoff
pub const BACKOFF_MULTIPLIER: f32 = 1.5;
/// Maximum total duration for retry attempts
pub const MAX_TOTAL_RETRY_DURATION: Duration = Duration::from_secs(60);
}
/// Health check and monitoring intervals
pub mod monitoring {
use super::Duration;
/// Default health check interval
pub const HEALTH_CHECK_INTERVAL: Duration = Duration::from_secs(30);
/// Default metrics collection interval
pub const METRICS_COLLECTION_INTERVAL: Duration = Duration::from_secs(10);
/// Default log flush interval
pub const LOG_FLUSH_INTERVAL: Duration = Duration::from_secs(5);
/// Circuit breaker check interval
pub const CIRCUIT_BREAKER_CHECK_INTERVAL: Duration = Duration::from_millis(100);
/// Kill switch session timeout (5 minutes)
pub const KILL_SWITCH_SESSION_TIMEOUT: Duration = Duration::from_secs(300);
}
/// Event processing defaults
pub mod events {
use super::Duration;
/// Event batch timeout
pub const BATCH_TIMEOUT: Duration = Duration::from_millis(100);
/// Event batch size
pub const BATCH_SIZE: usize = 100;
/// Event retry delay
pub const RETRY_DELAY: Duration = Duration::from_millis(50);
/// Maximum event backlog before applying backpressure
pub const MAX_EVENT_BACKLOG: usize = 10000;
/// Maximum span buffer size for tracing
pub const MAX_SPAN_BUFFER_SIZE: usize = 100_000;
/// Span export batch size
pub const SPAN_EXPORT_BATCH_SIZE: usize = 1000;
}
/// ML model constants
pub mod ml {
use super::Duration;
/// Maximum GPU batch size
pub const MAX_GPU_BATCH_SIZE: usize = 8192;
/// Maximum CPU batch size
pub const MAX_CPU_BATCH_SIZE: usize = 1024;
/// Default model cache cleanup interval (1 hour)
pub const MODEL_CACHE_CLEANUP_INTERVAL: Duration = Duration::from_secs(3600);
/// Default model health check interval (30 seconds)
pub const MODEL_HEALTH_CHECK_INTERVAL: Duration = Duration::from_secs(30);
/// Model deployment stage timeout (5 minutes)
pub const DEPLOYMENT_STAGE_TIMEOUT: Duration = Duration::from_secs(300);
/// Model deployment total timeout (30 minutes)
pub const DEPLOYMENT_TOTAL_TIMEOUT: Duration = Duration::from_secs(1800);
/// Model validation scan timeout (10 minutes)
pub const VALIDATION_SCAN_TIMEOUT: Duration = Duration::from_secs(600);
/// Canary deployment duration (5 minutes)
pub const CANARY_DURATION: Duration = Duration::from_secs(300);
/// Model rollback timeout (1 minute)
pub const ROLLBACK_TIMEOUT: Duration = Duration::from_secs(60);
/// Drift detection check interval (5 minutes)
pub const DRIFT_CHECK_INTERVAL: Duration = Duration::from_secs(300);
/// Drift detection warning threshold
pub const DRIFT_WARNING_THRESHOLD: f64 = 0.05;
/// Maximum recommendation age for Kelly sizing (1 minute)
pub const MAX_KELLY_RECOMMENDATION_AGE: Duration = Duration::from_secs(60);
/// Kelly sizing cache TTL (5 minutes)
pub const KELLY_CACHE_TTL: Duration = Duration::from_secs(300);
}
/// Safety system defaults
pub mod safety {
use super::Duration;
/// Safety check timeout for production (5ms)
pub const PRODUCTION_SAFETY_CHECK_TIMEOUT: Duration = Duration::from_millis(5);
/// Safety check timeout for development (50ms)
pub const DEVELOPMENT_SAFETY_CHECK_TIMEOUT: Duration = Duration::from_millis(50);
/// Auto-recovery delay for production (30 minutes)
pub const PRODUCTION_AUTO_RECOVERY_DELAY: Duration = Duration::from_secs(1800);
/// Auto-recovery delay for development (1 minute)
pub const DEVELOPMENT_AUTO_RECOVERY_DELAY: Duration = Duration::from_secs(60);
/// Loss check interval for production (5 seconds)
pub const PRODUCTION_LOSS_CHECK_INTERVAL: Duration = Duration::from_secs(5);
/// Loss check interval for development (30 seconds)
pub const DEVELOPMENT_LOSS_CHECK_INTERVAL: Duration = Duration::from_secs(30);
/// Position check interval for production (2 seconds)
pub const PRODUCTION_POSITION_CHECK_INTERVAL: Duration = Duration::from_secs(2);
/// Position check interval for development (15 seconds)
pub const DEVELOPMENT_POSITION_CHECK_INTERVAL: Duration = Duration::from_secs(15);
/// Memory check interval
pub const MEMORY_CHECK_INTERVAL: Duration = Duration::from_secs(1);
/// Circuit breaker trip cooldown (30 seconds)
pub const CIRCUIT_BREAKER_COOLDOWN: Duration = Duration::from_secs(30);
}
/// Time conversion constants
pub mod time {
/// Nanoseconds per microsecond
pub const NANOS_PER_MICRO: u64 = 1_000;
/// Nanoseconds per millisecond
pub const NANOS_PER_MILLI: u64 = 1_000_000;
/// Nanoseconds per second
pub const NANOS_PER_SECOND: u64 = 1_000_000_000;
/// Microseconds per second
pub const MICROS_PER_SECOND: u64 = 1_000_000;
/// Milliseconds per second
pub const MILLIS_PER_SECOND: u64 = 1_000;
/// Seconds per minute
pub const SECONDS_PER_MINUTE: u64 = 60;
/// Seconds per hour
pub const SECONDS_PER_HOUR: u64 = 3600;
/// Seconds per day
pub const SECONDS_PER_DAY: u64 = 86400;
/// Trading days per year
pub const TRADING_DAYS_PER_YEAR: usize = 252;
}
/// Financial constants
pub mod financial {
/// Basis points per unit
pub const BASIS_POINTS_PER_UNIT: u32 = 10_000;
/// Cents per dollar
pub const CENTS_PER_DOLLAR: u32 = 100;
/// Default profit target in basis points (1%)
pub const DEFAULT_PROFIT_TARGET_BPS: u32 = 100;
/// Default stop loss in basis points (0.5%)
pub const DEFAULT_STOP_LOSS_BPS: u32 = 50;
/// Minimum return threshold in basis points
pub const MIN_RETURN_THRESHOLD_BPS: i32 = 5;
/// Price scaling factor (6 decimal places)
pub const PRICE_SCALE: i64 = 1_000_000;
/// Quantity scaling factor (6 decimal places)
pub const QUANTITY_SCALE: i64 = 1_000_000;
/// Money scaling factor (6 decimal places)
pub const MONEY_SCALE: i64 = 1_000_000;
/// Unified scaling factor for all financial operations
pub const UNIFIED_SCALE_FACTOR: i64 = 1_000_000;
/// ML precision factor (8 decimal places)
pub const PRECISION_FACTOR: i64 = 100_000_000;
/// VPIN precision factor (4 decimal places)
pub const VPIN_PRECISION_FACTOR: i64 = 10_000;
}
/// Validation limits
pub mod limits {
/// Maximum symbol length
pub const MAX_SYMBOL_LENGTH: usize = 12;
/// Maximum account ID length
pub const MAX_ACCOUNT_ID_LENGTH: usize = 32;
/// Maximum description length
pub const MAX_DESCRIPTION_LENGTH: usize = 256;
/// Maximum metadata key length
pub const MAX_METADATA_KEY_LENGTH: usize = 64;
/// Maximum metadata value length
pub const MAX_METADATA_VALUE_LENGTH: usize = 512;
/// Maximum metadata entries
pub const MAX_METADATA_ENTRIES: usize = 100;
/// Maximum price value
pub const MAX_PRICE: f64 = 1_000_000.0;
/// Minimum price value
pub const MIN_PRICE: f64 = 0.000_001;
/// Maximum quantity value
pub const MAX_QUANTITY: f64 = 1_000_000_000.0;
/// Minimum quantity value
pub const MIN_QUANTITY: f64 = 0.000_001;
/// Maximum leverage
pub const MAX_LEVERAGE: f64 = 1000.0;
/// Minimum leverage
pub const MIN_LEVERAGE: f64 = 0.1;
/// Maximum allocation size (1GB)
pub const MAX_ALLOCATION_SIZE: usize = 1024 * 1024 * 1024;
/// Maximum duration in milliseconds (24 hours)
pub const MAX_DURATION_MILLIS: u64 = 24 * 60 * 60 * 1000;
}
/// Hardware alignment constants
pub mod hardware {
/// CPU cache line size
pub const CACHE_LINE_SIZE: usize = 64;
/// SIMD alignment for AVX2
pub const SIMD_ALIGNMENT: usize = 32;
/// Page size (4KB)
pub const PAGE_SIZE: usize = 4096;
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
#[allow(clippy::assertions_on_constants)]
fn test_breach_thresholds_ordered() {
assert!(risk::BREACH_WARNING_PCT < risk::BREACH_SOFT_PCT);
assert!(risk::BREACH_SOFT_PCT < risk::BREACH_HARD_PCT);
assert!(risk::BREACH_HARD_PCT < risk::BREACH_CRITICAL_PCT);
}
#[test]
#[allow(clippy::assertions_on_constants)]
fn test_var_z_scores_ordered() {
assert!(var::Z_SCORE_P90 < var::Z_SCORE_P95);
assert!(var::Z_SCORE_P95 < var::Z_SCORE_P97_5);
assert!(var::Z_SCORE_P97_5 < var::Z_SCORE_P99);
assert!(var::Z_SCORE_P99 < var::Z_SCORE_P99_9);
}
#[test]
fn test_time_conversions() {
assert_eq!(time::NANOS_PER_MICRO * 1000, time::NANOS_PER_MILLI);
assert_eq!(time::NANOS_PER_MILLI * 1000, time::NANOS_PER_SECOND);
assert_eq!(time::MICROS_PER_SECOND * 1000, time::NANOS_PER_SECOND);
}
#[test]
fn test_financial_scales_consistent() {
assert_eq!(financial::PRICE_SCALE, financial::UNIFIED_SCALE_FACTOR);
assert_eq!(financial::QUANTITY_SCALE, financial::UNIFIED_SCALE_FACTOR);
assert_eq!(financial::MONEY_SCALE, financial::UNIFIED_SCALE_FACTOR);
}
}