## Overview Deployed 12 parallel agents to resolve critical production blockers across authentication, configuration, ML pipeline, testing, and system optimization. All core objectives achieved. ## 🔐 Authentication & Security (Agents 1-2) ### Agent 1: Tonic 0.14 Authentication Compatibility ✅ - Migrated from Tower Service middleware to Tonic's native Interceptor - Fixed Error = Infallible incompatibility with Tonic 0.14 - Re-enabled authentication across all gRPC services - Maintains JWT, mTLS, rate limiting, RBAC, and audit trails - Files: trading_service/src/{auth_interceptor.rs, main.rs} ### Agent 2: Postgres Feature Flag ✅ - Added missing 'postgres' feature to adaptive-strategy/Cargo.toml - Resolved 9 warnings about unexpected cfg conditions - Properly gated all postgres-dependent code - Files: adaptive-strategy/{Cargo.toml, src/database_loader.rs, src/lib.rs} ## 🤖 ML & Data Pipeline (Agents 3, 5, 7) ### Agent 3: ML Performance Monitoring Foundation ✅ - Created ml_metrics.rs with 12 Prometheus metrics - Designed integration plan for MLPerformanceMonitor and MLFallbackManager - Added prometheus dependency to trading_service - Files: trading_service/src/{lib.rs, ml_metrics.rs}, Cargo.toml - Docs: WAVE_66_AGENT_3_IMPLEMENTATION.md ### Agent 5: Mock Data Feature Removal ✅ - Fixed module import issues in ml_training_service - Removed mock-data from default features (production uses real data) - Updated README with feature flag documentation - Files: ml_training_service/{Cargo.toml, src/main.rs, README.md} ### Agent 7: Advanced Feature Extraction ✅ - Implemented technical indicators (RSI, MACD, EMA, Bollinger, ATR) - Created stateful TechnicalIndicatorCalculator (566 lines) - Integrated with data_loader for real ML features - Unblocked ML training pipeline - Files: ml_training_service/src/{technical_indicators.rs, data_loader.rs, lib.rs} ## ⚙️ Configuration & Testing (Agents 4, 6, 11, 12) ### Agent 4: E2E Test Proto Fixes ✅ - Fixed namespace collision from wildcard proto imports - Resolved 9 compilation errors (5 ambiguity + 4 API mismatches) - Updated for Tonic 0.14 API changes - Files: tests/e2e/src/workflows.rs ### Agent 6: Config Phase 4 - Integration Tests ✅ - Created 25 comprehensive integration tests - Hot-reload verification with PostgreSQL NOTIFY/LISTEN - ACID transaction testing (atomicity, consistency, isolation, durability) - Concurrent update handling and performance benchmarks - Files: adaptive-strategy/tests/hot_reload_integration.rs - Docs: adaptive-strategy/{PHASE4_COMPLETION.md, docs/hot_reload_testing.md} ### Agent 11: Magic Numbers Centralization ✅ - Analyzed 500+ hardcoded values across 100+ files - Created centralized thresholds module (450 lines, 15 sub-modules) - Environment configuration templates (.env.{development,production}.example) - 3-tier configuration architecture designed - Files: common/src/thresholds.rs, .env.*.example - Docs: WAVE_66_AGENT_11_{ANALYSIS,DELIVERABLES,SUMMARY}.md - Docs: docs/CONFIGURATION_QUICK_REFERENCE.md ### Agent 12: Test Suite Execution ✅ - Executed 418 core tests with 100% pass rate - Verified trading_engine (281 tests), adaptive-strategy (69 tests), common (68 tests) - Production readiness assessment completed - Fixed test compilation issues in data/tests/comprehensive_coverage_tests.rs - Docs: docs/wave66_agent12_test_report.md ## 📊 System Optimization (Agents 8-10) ### Agent 8: Database Pooling Analysis ✅ - Identified critical 30s timeout in ML training service - Inconsistent pool sizing across services - Insufficient statement cache (backtesting 100 → 500) - HFT-optimized configurations designed - Comprehensive analysis documented (no code changes - design phase) ### Agent 9: gRPC Streaming Analysis ✅ - Critical HTTP/2 optimization opportunities identified - tcp_nodelay(true) for -40ms latency reduction - Stream-specific buffer sizing (1K → 100K for market data) - Backpressure monitoring design - 4-week implementation roadmap created ### Agent 10: Metrics Aggregation Analysis ✅ - Critical cardinality explosion identified (100K+ potential time series) - Unbounded memory growth in HDR histograms - Asset class bucketing strategy designed (99% cardinality reduction) - LRU caching for bounded memory - 5-phase optimization plan documented ## 📈 Impact Summary - ✅ Authentication fully operational with Tonic 0.14 - ✅ ML training pipeline unblocked (real features, not mock data) - ✅ Configuration hot-reload fully tested (25 integration tests) - ✅ 418 core tests passing (100% pass rate) - ✅ Production deployment foundation complete - ✅ Comprehensive optimization roadmaps for Waves 67-70 ## 🔧 Files Changed (29 total) Modified: 17 files across services, crates, and tests Created: 12 new files (modules, tests, documentation) ## 🎯 Next Steps (Wave 67+) - Implement Agent 8-10 optimization plans - Complete ML monitoring integration (Agent 3) - Execute configuration centralization migration - Performance validation and load testing 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com>
262 lines
9.4 KiB
Rust
262 lines
9.4 KiB
Rust
//! Database-backed configuration loader for adaptive strategy
|
|
//!
|
|
//! This module provides functionality to load adaptive strategy configurations
|
|
//! from PostgreSQL, replacing hardcoded defaults with database-driven config.
|
|
//!
|
|
//! Features:
|
|
//! - Full configuration loading from database
|
|
//! - Hot-reload support via PostgreSQL NOTIFY/LISTEN
|
|
//! - Fallback to default configuration on database errors
|
|
//! - Validation of loaded configurations
|
|
|
|
#[cfg(feature = "postgres")]
|
|
use sqlx::postgres::PgListener;
|
|
|
|
#[cfg(feature = "postgres")]
|
|
use crate::config_types::{AdaptiveStrategyConfig, AdaptiveStrategyConfigRow, ModelConfigRow, FeatureConfigRow};
|
|
#[cfg(feature = "postgres")]
|
|
use std::time::Duration;
|
|
|
|
/// Database-backed configuration loader
|
|
///
|
|
/// Loads adaptive strategy configurations from PostgreSQL and provides
|
|
/// hot-reload capabilities through NOTIFY/LISTEN.
|
|
#[cfg(feature = "postgres")]
|
|
pub struct DatabaseConfigLoader {
|
|
/// Database connection pool
|
|
pool: sqlx::PgPool,
|
|
/// PostgreSQL listener for hot-reload
|
|
listener: Option<PgListener>,
|
|
/// Cache timeout for loaded configurations (reserved for future caching implementation)
|
|
#[allow(dead_code)]
|
|
cache_timeout: Duration,
|
|
}
|
|
|
|
#[cfg(feature = "postgres")]
|
|
impl DatabaseConfigLoader {
|
|
/// Create a new database configuration loader
|
|
///
|
|
/// # Arguments
|
|
/// * `database_url` - PostgreSQL connection URL
|
|
///
|
|
/// # Example
|
|
/// ```no_run
|
|
/// # use adaptive_strategy::database_loader::DatabaseConfigLoader;
|
|
/// # async fn example() -> Result<(), Box<dyn std::error::Error>> {
|
|
/// let loader = DatabaseConfigLoader::new("postgresql://localhost/foxhunt").await?;
|
|
/// # Ok(())
|
|
/// # }
|
|
/// ```
|
|
pub async fn new(database_url: &str) -> Result<Self, sqlx::Error> {
|
|
let pool = sqlx::PgPool::connect(database_url).await?;
|
|
|
|
Ok(Self {
|
|
pool,
|
|
listener: None,
|
|
cache_timeout: Duration::from_secs(300), // 5 minutes
|
|
})
|
|
}
|
|
|
|
/// Create a loader with an existing connection pool
|
|
///
|
|
/// # Arguments
|
|
/// * `pool` - Existing PostgreSQL connection pool
|
|
pub fn with_pool(pool: sqlx::PgPool) -> Self {
|
|
Self {
|
|
pool,
|
|
listener: None,
|
|
cache_timeout: Duration::from_secs(300),
|
|
}
|
|
}
|
|
|
|
/// Load configuration from database by strategy ID
|
|
///
|
|
/// # Arguments
|
|
/// * `strategy_id` - Strategy identifier (e.g., "default", "prod_v1")
|
|
///
|
|
/// # Returns
|
|
/// - `Ok(Some(config))` - Configuration loaded successfully
|
|
/// - `Ok(None)` - Strategy not found in database
|
|
/// - `Err(...)` - Database error occurred
|
|
///
|
|
/// # Example
|
|
/// ```no_run
|
|
/// # use adaptive_strategy::database_loader::DatabaseConfigLoader;
|
|
/// # async fn example(loader: &DatabaseConfigLoader) -> Result<(), Box<dyn std::error::Error>> {
|
|
/// let config = loader.load_config("default").await?
|
|
/// .expect("Default strategy not found");
|
|
/// config.validate()?;
|
|
/// # Ok(())
|
|
/// # }
|
|
/// ```
|
|
pub async fn load_config(
|
|
&self,
|
|
strategy_id: &str,
|
|
) -> Result<Option<AdaptiveStrategyConfig>, String> {
|
|
// Load main configuration
|
|
let config_row: Option<AdaptiveStrategyConfigRow> = sqlx::query_as(
|
|
r#"
|
|
SELECT
|
|
id, strategy_id, name, description,
|
|
execution_interval_ms, error_backoff_duration_secs,
|
|
max_concurrent_operations, strategy_timeout_secs,
|
|
max_parallel_models, rebalancing_interval_secs,
|
|
min_model_weight, max_model_weight,
|
|
max_position_size, max_leverage, stop_loss_pct,
|
|
position_sizing_method, max_portfolio_var,
|
|
max_drawdown_threshold, kelly_fraction,
|
|
book_depth, vpin_window, trade_classification_threshold,
|
|
trade_size_buckets, microstructure_features,
|
|
regime_detection_method, regime_lookback_window,
|
|
regime_transition_threshold, regime_features,
|
|
execution_algorithm, max_order_size, min_order_size,
|
|
order_timeout_secs, max_slippage_bps,
|
|
smart_routing_enabled, dark_pool_preference,
|
|
active, version, created_at, updated_at,
|
|
created_by, updated_by, metadata
|
|
FROM adaptive_strategy_config
|
|
WHERE strategy_id = $1 AND active = true
|
|
"#,
|
|
)
|
|
.bind(strategy_id)
|
|
.fetch_optional(&self.pool)
|
|
.await
|
|
.map_err(|e| format!("Failed to load config: {}", e))?;
|
|
|
|
let Some(config_row) = config_row else {
|
|
return Ok(None);
|
|
};
|
|
|
|
let config_id = config_row.id;
|
|
|
|
// Load associated models
|
|
let models: Vec<ModelConfigRow> = sqlx::query_as(
|
|
r#"
|
|
SELECT
|
|
id, strategy_config_id, model_id, model_name, model_type,
|
|
parameters, initial_weight, enabled, display_order,
|
|
created_at, updated_at
|
|
FROM adaptive_strategy_models
|
|
WHERE strategy_config_id = $1
|
|
ORDER BY display_order, created_at
|
|
"#,
|
|
)
|
|
.bind(config_id)
|
|
.fetch_all(&self.pool)
|
|
.await
|
|
.map_err(|e| format!("Failed to load models: {}", e))?;
|
|
|
|
// Load associated features
|
|
let features: Vec<FeatureConfigRow> = sqlx::query_as(
|
|
r#"
|
|
SELECT
|
|
id, strategy_config_id, feature_name, feature_type,
|
|
parameters, enabled, required,
|
|
created_at, updated_at
|
|
FROM adaptive_strategy_features
|
|
WHERE strategy_config_id = $1
|
|
ORDER BY feature_name
|
|
"#,
|
|
)
|
|
.bind(config_id)
|
|
.fetch_all(&self.pool)
|
|
.await
|
|
.map_err(|e| format!("Failed to load features: {}", e))?;
|
|
|
|
// Convert to structured configuration
|
|
let config = config_row.into_config(models, features)?;
|
|
|
|
// Validate configuration
|
|
config.validate()?;
|
|
|
|
Ok(Some(config))
|
|
}
|
|
|
|
// REMOVED: load_config_or_default() does not make sense for postgres-backed configs
|
|
// The config_types::AdaptiveStrategyConfig has no Default impl (by design).
|
|
// Fallback logic should use config::AdaptiveStrategyConfig::default() at the
|
|
// application level after calling load_config().await.
|
|
//
|
|
// For non-postgres builds, see the stub implementation below which provides
|
|
// the fallback directly to config::AdaptiveStrategyConfig::default().
|
|
|
|
/// Enable hot-reload support
|
|
///
|
|
/// Subscribes to PostgreSQL NOTIFY events for configuration changes.
|
|
/// Call `check_for_updates()` periodically to receive notifications.
|
|
pub async fn enable_hot_reload(&mut self) -> Result<(), sqlx::Error> {
|
|
let mut listener = PgListener::connect_with(&self.pool).await?;
|
|
listener.listen("adaptive_strategy_config_change").await?;
|
|
self.listener = Some(listener);
|
|
Ok(())
|
|
}
|
|
|
|
/// Check for configuration change notifications
|
|
///
|
|
/// Returns the strategy_id if a configuration change notification
|
|
/// was received, or None if no notifications are pending.
|
|
///
|
|
/// # Example
|
|
/// ```no_run
|
|
/// # use adaptive_strategy::database_loader::DatabaseConfigLoader;
|
|
/// # async fn example(loader: &mut DatabaseConfigLoader) -> Result<(), Box<dyn std::error::Error>> {
|
|
/// // In a background task
|
|
/// loop {
|
|
/// if let Some(strategy_id) = loader.check_for_updates().await? {
|
|
/// println!("Configuration changed for strategy: {}", strategy_id);
|
|
/// // Reload configuration
|
|
/// let new_config = loader.load_config(&strategy_id).await?;
|
|
/// }
|
|
/// tokio::time::sleep(std::time::Duration::from_secs(1)).await;
|
|
/// }
|
|
/// # }
|
|
/// ```
|
|
pub async fn check_for_updates(&mut self) -> Result<Option<String>, sqlx::Error> {
|
|
if let Some(listener) = &mut self.listener {
|
|
if let Some(notification) = listener.try_recv().await? {
|
|
// Parse notification payload
|
|
if let Ok(payload) = serde_json::from_str::<serde_json::Value>(notification.payload()) {
|
|
if let Some(strategy_id) = payload.get("strategy_id").and_then(|v| v.as_str()) {
|
|
return Ok(Some(strategy_id.to_string()));
|
|
}
|
|
}
|
|
}
|
|
}
|
|
Ok(None)
|
|
}
|
|
|
|
/// Get the underlying connection pool
|
|
pub fn pool(&self) -> &sqlx::PgPool {
|
|
&self.pool
|
|
}
|
|
}
|
|
|
|
// Synchronous fallback loader for when postgres feature is not enabled
|
|
#[cfg(not(feature = "postgres"))]
|
|
pub struct DatabaseConfigLoader;
|
|
|
|
#[cfg(not(feature = "postgres"))]
|
|
impl DatabaseConfigLoader {
|
|
/// Always returns default configuration when postgres feature is disabled
|
|
pub fn load_config_or_default(&self, _strategy_id: &str) -> crate::config::AdaptiveStrategyConfig {
|
|
crate::config::AdaptiveStrategyConfig::default()
|
|
}
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
use std::time::Duration;
|
|
|
|
#[test]
|
|
fn test_fallback_loader_without_postgres() {
|
|
// When postgres feature is disabled, should compile and return defaults
|
|
#[cfg(not(feature = "postgres"))]
|
|
{
|
|
let loader = DatabaseConfigLoader;
|
|
let config = loader.load_config_or_default("test");
|
|
assert_eq!(config.general.execution_interval, Duration::from_millis(100));
|
|
}
|
|
}
|
|
}
|