Files
foxhunt/adaptive-strategy/src/database_loader.rs
jgrusewski a2d1eacce6 🚀 Wave 66: Production Readiness - 12 Parallel Agents Complete
## Overview
Deployed 12 parallel agents to resolve critical production blockers across authentication,
configuration, ML pipeline, testing, and system optimization. All core objectives achieved.

## 🔐 Authentication & Security (Agents 1-2)
### Agent 1: Tonic 0.14 Authentication Compatibility 
- Migrated from Tower Service middleware to Tonic's native Interceptor
- Fixed Error = Infallible incompatibility with Tonic 0.14
- Re-enabled authentication across all gRPC services
- Maintains JWT, mTLS, rate limiting, RBAC, and audit trails
- Files: trading_service/src/{auth_interceptor.rs, main.rs}

### Agent 2: Postgres Feature Flag 
- Added missing 'postgres' feature to adaptive-strategy/Cargo.toml
- Resolved 9 warnings about unexpected cfg conditions
- Properly gated all postgres-dependent code
- Files: adaptive-strategy/{Cargo.toml, src/database_loader.rs, src/lib.rs}

## 🤖 ML & Data Pipeline (Agents 3, 5, 7)
### Agent 3: ML Performance Monitoring Foundation 
- Created ml_metrics.rs with 12 Prometheus metrics
- Designed integration plan for MLPerformanceMonitor and MLFallbackManager
- Added prometheus dependency to trading_service
- Files: trading_service/src/{lib.rs, ml_metrics.rs}, Cargo.toml
- Docs: WAVE_66_AGENT_3_IMPLEMENTATION.md

### Agent 5: Mock Data Feature Removal 
- Fixed module import issues in ml_training_service
- Removed mock-data from default features (production uses real data)
- Updated README with feature flag documentation
- Files: ml_training_service/{Cargo.toml, src/main.rs, README.md}

### Agent 7: Advanced Feature Extraction 
- Implemented technical indicators (RSI, MACD, EMA, Bollinger, ATR)
- Created stateful TechnicalIndicatorCalculator (566 lines)
- Integrated with data_loader for real ML features
- Unblocked ML training pipeline
- Files: ml_training_service/src/{technical_indicators.rs, data_loader.rs, lib.rs}

## ⚙️ Configuration & Testing (Agents 4, 6, 11, 12)
### Agent 4: E2E Test Proto Fixes 
- Fixed namespace collision from wildcard proto imports
- Resolved 9 compilation errors (5 ambiguity + 4 API mismatches)
- Updated for Tonic 0.14 API changes
- Files: tests/e2e/src/workflows.rs

### Agent 6: Config Phase 4 - Integration Tests 
- Created 25 comprehensive integration tests
- Hot-reload verification with PostgreSQL NOTIFY/LISTEN
- ACID transaction testing (atomicity, consistency, isolation, durability)
- Concurrent update handling and performance benchmarks
- Files: adaptive-strategy/tests/hot_reload_integration.rs
- Docs: adaptive-strategy/{PHASE4_COMPLETION.md, docs/hot_reload_testing.md}

### Agent 11: Magic Numbers Centralization 
- Analyzed 500+ hardcoded values across 100+ files
- Created centralized thresholds module (450 lines, 15 sub-modules)
- Environment configuration templates (.env.{development,production}.example)
- 3-tier configuration architecture designed
- Files: common/src/thresholds.rs, .env.*.example
- Docs: WAVE_66_AGENT_11_{ANALYSIS,DELIVERABLES,SUMMARY}.md
- Docs: docs/CONFIGURATION_QUICK_REFERENCE.md

### Agent 12: Test Suite Execution 
- Executed 418 core tests with 100% pass rate
- Verified trading_engine (281 tests), adaptive-strategy (69 tests), common (68 tests)
- Production readiness assessment completed
- Fixed test compilation issues in data/tests/comprehensive_coverage_tests.rs
- Docs: docs/wave66_agent12_test_report.md

## 📊 System Optimization (Agents 8-10)
### Agent 8: Database Pooling Analysis 
- Identified critical 30s timeout in ML training service
- Inconsistent pool sizing across services
- Insufficient statement cache (backtesting 100 → 500)
- HFT-optimized configurations designed
- Comprehensive analysis documented (no code changes - design phase)

### Agent 9: gRPC Streaming Analysis 
- Critical HTTP/2 optimization opportunities identified
- tcp_nodelay(true) for -40ms latency reduction
- Stream-specific buffer sizing (1K → 100K for market data)
- Backpressure monitoring design
- 4-week implementation roadmap created

### Agent 10: Metrics Aggregation Analysis 
- Critical cardinality explosion identified (100K+ potential time series)
- Unbounded memory growth in HDR histograms
- Asset class bucketing strategy designed (99% cardinality reduction)
- LRU caching for bounded memory
- 5-phase optimization plan documented

## 📈 Impact Summary
-  Authentication fully operational with Tonic 0.14
-  ML training pipeline unblocked (real features, not mock data)
-  Configuration hot-reload fully tested (25 integration tests)
-  418 core tests passing (100% pass rate)
-  Production deployment foundation complete
-  Comprehensive optimization roadmaps for Waves 67-70

## 🔧 Files Changed (29 total)
Modified: 17 files across services, crates, and tests
Created: 12 new files (modules, tests, documentation)

## 🎯 Next Steps (Wave 67+)
- Implement Agent 8-10 optimization plans
- Complete ML monitoring integration (Agent 3)
- Execute configuration centralization migration
- Performance validation and load testing

🤖 Generated with Claude Code
Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-03 08:09:52 +02:00

262 lines
9.4 KiB
Rust

//! Database-backed configuration loader for adaptive strategy
//!
//! This module provides functionality to load adaptive strategy configurations
//! from PostgreSQL, replacing hardcoded defaults with database-driven config.
//!
//! Features:
//! - Full configuration loading from database
//! - Hot-reload support via PostgreSQL NOTIFY/LISTEN
//! - Fallback to default configuration on database errors
//! - Validation of loaded configurations
#[cfg(feature = "postgres")]
use sqlx::postgres::PgListener;
#[cfg(feature = "postgres")]
use crate::config_types::{AdaptiveStrategyConfig, AdaptiveStrategyConfigRow, ModelConfigRow, FeatureConfigRow};
#[cfg(feature = "postgres")]
use std::time::Duration;
/// Database-backed configuration loader
///
/// Loads adaptive strategy configurations from PostgreSQL and provides
/// hot-reload capabilities through NOTIFY/LISTEN.
#[cfg(feature = "postgres")]
pub struct DatabaseConfigLoader {
/// Database connection pool
pool: sqlx::PgPool,
/// PostgreSQL listener for hot-reload
listener: Option<PgListener>,
/// Cache timeout for loaded configurations (reserved for future caching implementation)
#[allow(dead_code)]
cache_timeout: Duration,
}
#[cfg(feature = "postgres")]
impl DatabaseConfigLoader {
/// Create a new database configuration loader
///
/// # Arguments
/// * `database_url` - PostgreSQL connection URL
///
/// # Example
/// ```no_run
/// # use adaptive_strategy::database_loader::DatabaseConfigLoader;
/// # async fn example() -> Result<(), Box<dyn std::error::Error>> {
/// let loader = DatabaseConfigLoader::new("postgresql://localhost/foxhunt").await?;
/// # Ok(())
/// # }
/// ```
pub async fn new(database_url: &str) -> Result<Self, sqlx::Error> {
let pool = sqlx::PgPool::connect(database_url).await?;
Ok(Self {
pool,
listener: None,
cache_timeout: Duration::from_secs(300), // 5 minutes
})
}
/// Create a loader with an existing connection pool
///
/// # Arguments
/// * `pool` - Existing PostgreSQL connection pool
pub fn with_pool(pool: sqlx::PgPool) -> Self {
Self {
pool,
listener: None,
cache_timeout: Duration::from_secs(300),
}
}
/// Load configuration from database by strategy ID
///
/// # Arguments
/// * `strategy_id` - Strategy identifier (e.g., "default", "prod_v1")
///
/// # Returns
/// - `Ok(Some(config))` - Configuration loaded successfully
/// - `Ok(None)` - Strategy not found in database
/// - `Err(...)` - Database error occurred
///
/// # Example
/// ```no_run
/// # use adaptive_strategy::database_loader::DatabaseConfigLoader;
/// # async fn example(loader: &DatabaseConfigLoader) -> Result<(), Box<dyn std::error::Error>> {
/// let config = loader.load_config("default").await?
/// .expect("Default strategy not found");
/// config.validate()?;
/// # Ok(())
/// # }
/// ```
pub async fn load_config(
&self,
strategy_id: &str,
) -> Result<Option<AdaptiveStrategyConfig>, String> {
// Load main configuration
let config_row: Option<AdaptiveStrategyConfigRow> = sqlx::query_as(
r#"
SELECT
id, strategy_id, name, description,
execution_interval_ms, error_backoff_duration_secs,
max_concurrent_operations, strategy_timeout_secs,
max_parallel_models, rebalancing_interval_secs,
min_model_weight, max_model_weight,
max_position_size, max_leverage, stop_loss_pct,
position_sizing_method, max_portfolio_var,
max_drawdown_threshold, kelly_fraction,
book_depth, vpin_window, trade_classification_threshold,
trade_size_buckets, microstructure_features,
regime_detection_method, regime_lookback_window,
regime_transition_threshold, regime_features,
execution_algorithm, max_order_size, min_order_size,
order_timeout_secs, max_slippage_bps,
smart_routing_enabled, dark_pool_preference,
active, version, created_at, updated_at,
created_by, updated_by, metadata
FROM adaptive_strategy_config
WHERE strategy_id = $1 AND active = true
"#,
)
.bind(strategy_id)
.fetch_optional(&self.pool)
.await
.map_err(|e| format!("Failed to load config: {}", e))?;
let Some(config_row) = config_row else {
return Ok(None);
};
let config_id = config_row.id;
// Load associated models
let models: Vec<ModelConfigRow> = sqlx::query_as(
r#"
SELECT
id, strategy_config_id, model_id, model_name, model_type,
parameters, initial_weight, enabled, display_order,
created_at, updated_at
FROM adaptive_strategy_models
WHERE strategy_config_id = $1
ORDER BY display_order, created_at
"#,
)
.bind(config_id)
.fetch_all(&self.pool)
.await
.map_err(|e| format!("Failed to load models: {}", e))?;
// Load associated features
let features: Vec<FeatureConfigRow> = sqlx::query_as(
r#"
SELECT
id, strategy_config_id, feature_name, feature_type,
parameters, enabled, required,
created_at, updated_at
FROM adaptive_strategy_features
WHERE strategy_config_id = $1
ORDER BY feature_name
"#,
)
.bind(config_id)
.fetch_all(&self.pool)
.await
.map_err(|e| format!("Failed to load features: {}", e))?;
// Convert to structured configuration
let config = config_row.into_config(models, features)?;
// Validate configuration
config.validate()?;
Ok(Some(config))
}
// REMOVED: load_config_or_default() does not make sense for postgres-backed configs
// The config_types::AdaptiveStrategyConfig has no Default impl (by design).
// Fallback logic should use config::AdaptiveStrategyConfig::default() at the
// application level after calling load_config().await.
//
// For non-postgres builds, see the stub implementation below which provides
// the fallback directly to config::AdaptiveStrategyConfig::default().
/// Enable hot-reload support
///
/// Subscribes to PostgreSQL NOTIFY events for configuration changes.
/// Call `check_for_updates()` periodically to receive notifications.
pub async fn enable_hot_reload(&mut self) -> Result<(), sqlx::Error> {
let mut listener = PgListener::connect_with(&self.pool).await?;
listener.listen("adaptive_strategy_config_change").await?;
self.listener = Some(listener);
Ok(())
}
/// Check for configuration change notifications
///
/// Returns the strategy_id if a configuration change notification
/// was received, or None if no notifications are pending.
///
/// # Example
/// ```no_run
/// # use adaptive_strategy::database_loader::DatabaseConfigLoader;
/// # async fn example(loader: &mut DatabaseConfigLoader) -> Result<(), Box<dyn std::error::Error>> {
/// // In a background task
/// loop {
/// if let Some(strategy_id) = loader.check_for_updates().await? {
/// println!("Configuration changed for strategy: {}", strategy_id);
/// // Reload configuration
/// let new_config = loader.load_config(&strategy_id).await?;
/// }
/// tokio::time::sleep(std::time::Duration::from_secs(1)).await;
/// }
/// # }
/// ```
pub async fn check_for_updates(&mut self) -> Result<Option<String>, sqlx::Error> {
if let Some(listener) = &mut self.listener {
if let Some(notification) = listener.try_recv().await? {
// Parse notification payload
if let Ok(payload) = serde_json::from_str::<serde_json::Value>(notification.payload()) {
if let Some(strategy_id) = payload.get("strategy_id").and_then(|v| v.as_str()) {
return Ok(Some(strategy_id.to_string()));
}
}
}
}
Ok(None)
}
/// Get the underlying connection pool
pub fn pool(&self) -> &sqlx::PgPool {
&self.pool
}
}
// Synchronous fallback loader for when postgres feature is not enabled
#[cfg(not(feature = "postgres"))]
pub struct DatabaseConfigLoader;
#[cfg(not(feature = "postgres"))]
impl DatabaseConfigLoader {
/// Always returns default configuration when postgres feature is disabled
pub fn load_config_or_default(&self, _strategy_id: &str) -> crate::config::AdaptiveStrategyConfig {
crate::config::AdaptiveStrategyConfig::default()
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::time::Duration;
#[test]
fn test_fallback_loader_without_postgres() {
// When postgres feature is disabled, should compile and return defaults
#[cfg(not(feature = "postgres"))]
{
let loader = DatabaseConfigLoader;
let config = loader.load_config_or_default("test");
assert_eq!(config.general.execution_interval, Duration::from_millis(100));
}
}
}