Adds the convergence guardrail: every per-epoch HEALTH_DIAG metric is
checked against the bands in config/metric-bands.toml; N consecutive
warn-band epochs emit a tracing::warn; 2N consecutive error-band epochs
return Err(CommonError::RegressionDetected{...}) cleanly from the
training loop, which propagates to the train_baseline_rl subprocess
exit code (no libc::raise — clean Rust error path).
Wire-points:
- New module: crates/ml/src/trainers/dqn/trainer/monitoring.rs
- MetricBands {warn_low, warn_high, error_low, error_high}
- BandSettings {consecutive_epochs_for_warn, consecutive_epochs_for_error}
- MetricBandsRegistry: load_from_toml + update_and_check
- TerminationReason {RegressionWarn, RegressionError}
- NaN treated as out-of-band (consecutive++; never resets streak)
- Unknown metrics return None (silent OK per Invariant 7 audit)
- crates/common/src/error.rs: new CommonError::RegressionDetected variant
carrying {metric, value, band, consecutive}
- crates/ml/src/trainers/dqn/trainer/constructor.rs: load
config/metric-bands.toml at trainer init; warn-only on missing file
(backward compat for environments without the config)
- crates/ml/src/trainers/dqn/trainer/training_loop.rs: harvest per-epoch
metrics (parallel emit alongside HEALTH_DIAG), feed each through
registry.update_and_check; on Some(TerminationReason::RegressionError)
emit final HEALTH_DIAG[N]: TERMINATED_BY_REGRESSION line and return Err
- services/trading_service/src/error.rs: minimal handler for the new
CommonError variant (existing pattern)
Validation:
- 8 unit tests in monitoring::tests pass (band logic, NaN, warn-only
behaviour, error-streak threshold, unknown-metric, invalid TOML)
- regression_detection GPU smoke (3.19s): trainer with intentionally
narrow train_loss error band [0, 1e-9] self-terminates at epoch 5
after 6 consecutive error-band epochs; final HEALTH_DIAG line emits
TERMINATED_BY_REGRESSION with metric/value/consecutive/band fields
- multi_fold_convergence smoke (650s, --release): all 3 folds train
to completion, all 3 checkpoints saved, no false-positive
termination on the populated metric bands. Per-fold best train
Sharpe: F0=-9.7831 (bit-baseline), F1=25.8272, F2=39.2687. F1/F2
on the lower end of observed noise distribution
({74.56, 61.10, 71.53, 25.83} for F1; {88.20, 61.57, 65.96, 39.27}
for F2) but training healthy throughout: aux clauses fire every
epoch, sharpe_ema recovers from F0 collapse (-9.78 → +14.8 by start
of F2), no regression detection trips.
config/metric-bands.toml populated for the metrics emitted by
HEALTH_DIAG today (avg_q_value, train_loss, val_sharpe, train_sharpe,
aux_next_bar_mse, aux_regime_ce, isv_* slot EMAs, sharpe_ema, etc.).
Bands derived from current cleanroom smoke + permissive defaults
where only one sample exists; populate-metric-bands-from-runs.py will
tighten them after Plan 5 Task 5's multi-seed pass produces real
distributions.
Constraints honoured: GPU-only in hot path (band check is CPU-side
post-HEALTH_DIAG, off the captured graph); no atomicAdd; no stubs;
no // ok: band-aids; no tuned constants beyond the toml-loaded bands;
no .unwrap() introduced; cargo check clean at 11 warnings (workspace
baseline preserved, plus ml-dqn pre-existing 1 warning).
Audit doc: new row added documenting monitoring.rs module, the
CommonError variant, the training_loop wire-point, and the design
choice that band-checks run AFTER HEALTH_DIAG emit (not before) so
the diag log already reflects the metric values that triggered any
termination.
Plan 5 T1 (multi-seed harness) landed at c6634254e+47c8b783c; T2
(this) gives the regression hard-stop that the multi-seed final
pass (T5) consumes to bail out early on bad seeds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
410 lines
15 KiB
Rust
410 lines
15 KiB
Rust
//! Common error types and utilities
|
||
//!
|
||
//! This module provides shared error types and utilities used across
|
||
//! all Foxhunt services.
|
||
|
||
use serde::{Deserialize, Serialize};
|
||
use std::fmt;
|
||
use std::time::Duration;
|
||
use thiserror::Error;
|
||
|
||
/// Common error type for all Foxhunt services
|
||
#[derive(Debug, Error)]
|
||
pub enum CommonError {
|
||
/// Database operation failed - wraps database-specific errors
|
||
#[error("Database error: {0}")]
|
||
Database(#[from] crate::database::DatabaseError),
|
||
/// Configuration is invalid or missing required parameters
|
||
#[error("Configuration error: {0}")]
|
||
Configuration(String),
|
||
/// Network communication error occurred
|
||
#[error("Network error: {0}")]
|
||
Network(String),
|
||
/// Service-specific error with categorization for metrics
|
||
#[error("Service error: {category} - {message}")]
|
||
Service {
|
||
/// Error category for classification
|
||
category: ErrorCategory,
|
||
/// Descriptive error message
|
||
message: String,
|
||
},
|
||
/// Input validation failed
|
||
#[error("Validation error: {0}")]
|
||
Validation(String),
|
||
/// Operation exceeded maximum allowed execution time
|
||
#[error("Timeout error: operation took {actual_ms}ms, max allowed {max_ms}ms")]
|
||
Timeout {
|
||
/// Actual execution time in milliseconds
|
||
actual_ms: u64,
|
||
/// Maximum allowed execution time in milliseconds
|
||
max_ms: u64,
|
||
},
|
||
/// Plan 5 Task 2 — A.4 Regression Detection Hard-Stop.
|
||
///
|
||
/// Fired by `MetricBandsRegistry::update_and_check` after `2N` consecutive
|
||
/// epochs of an out-of-band HEALTH_DIAG metric (per `config/metric-bands.toml`).
|
||
/// `band_*` fields are flat doubles (not the `MetricBands` struct) so this
|
||
/// variant has no upward dependency on the `ml` crate. Trainer consumes the
|
||
/// variant by returning it from `train_with_data_full_loop_slices`; the call
|
||
/// stack unwinds cleanly. There is no SIGTERM raise — the error itself is
|
||
/// the termination signal.
|
||
#[error(
|
||
"Regression detected: metric `{metric}` value {value:.6e} out of error band \
|
||
[{band_error_low:.6e}, {band_error_high:.6e}] for {consecutive} consecutive epochs"
|
||
)]
|
||
RegressionDetected {
|
||
/// HEALTH_DIAG metric name (matches a key in `config/metric-bands.toml`).
|
||
metric: String,
|
||
/// Value at the terminating epoch.
|
||
value: f64,
|
||
/// Warn-band lower bound (`mean - 5σ` per Task 2.5).
|
||
band_warn_low: f64,
|
||
/// Warn-band upper bound (`mean + 5σ` per Task 2.5).
|
||
band_warn_high: f64,
|
||
/// Error-band lower bound (`mean - 10σ` per Task 2.5).
|
||
band_error_low: f64,
|
||
/// Error-band upper bound (`mean + 10σ` per Task 2.5).
|
||
band_error_high: f64,
|
||
/// Consecutive epochs out-of-error-band at termination
|
||
/// (always equals `2 * settings.consecutive_epochs_for_warn`).
|
||
consecutive: u32,
|
||
},
|
||
}
|
||
|
||
/// Error categories for classification and metrics
|
||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
|
||
#[allow(clippy::module_name_repetitions)]
|
||
pub enum ErrorCategory {
|
||
/// Market data related errors
|
||
MarketData,
|
||
/// Trading and order management errors
|
||
Trading,
|
||
/// Network and communication errors
|
||
Network,
|
||
/// System and infrastructure errors
|
||
System,
|
||
/// Configuration errors
|
||
Configuration,
|
||
/// Validation errors
|
||
Validation,
|
||
/// Critical errors requiring immediate attention
|
||
Critical,
|
||
/// Connection errors (data providers)
|
||
Connection,
|
||
/// Authentication errors
|
||
Authentication,
|
||
/// Rate limiting errors
|
||
RateLimit,
|
||
/// Data parsing errors
|
||
Parse,
|
||
/// Subscription errors
|
||
Subscription,
|
||
/// Financial safety and calculation errors
|
||
FinancialSafety,
|
||
/// Risk management and circuit breakers
|
||
RiskManagement,
|
||
/// Database and persistence layer
|
||
Database,
|
||
/// Broker connectivity and execution
|
||
Broker,
|
||
/// Machine learning and AI errors
|
||
MachineLearning,
|
||
/// Security and authentication errors
|
||
Security,
|
||
/// Business logic errors
|
||
BusinessLogic,
|
||
/// Resource errors (not found, conflicts)
|
||
Resource,
|
||
/// Development and testing errors
|
||
Development,
|
||
/// Risk management errors
|
||
Risk,
|
||
/// Machine learning errors (alias for `MachineLearning`)
|
||
ML,
|
||
/// Unknown/other errors
|
||
Other,
|
||
}
|
||
|
||
impl fmt::Display for ErrorCategory {
|
||
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
|
||
match self {
|
||
Self::MarketData => write!(f, "MARKET_DATA"),
|
||
Self::Trading => write!(f, "TRADING"),
|
||
Self::Network => write!(f, "NETWORK"),
|
||
Self::System => write!(f, "SYSTEM"),
|
||
Self::Configuration => write!(f, "CONFIGURATION"),
|
||
Self::Validation => write!(f, "VALIDATION"),
|
||
Self::Critical => write!(f, "CRITICAL"),
|
||
Self::Connection => write!(f, "CONNECTION"),
|
||
Self::Authentication => write!(f, "AUTHENTICATION"),
|
||
Self::RateLimit => write!(f, "RATE_LIMIT"),
|
||
Self::Parse => write!(f, "PARSE"),
|
||
Self::Subscription => write!(f, "SUBSCRIPTION"),
|
||
Self::FinancialSafety => write!(f, "FINANCIAL_SAFETY"),
|
||
Self::RiskManagement => write!(f, "RISK_MANAGEMENT"),
|
||
Self::Database => write!(f, "DATABASE"),
|
||
Self::Broker => write!(f, "BROKER"),
|
||
Self::MachineLearning => write!(f, "MACHINE_LEARNING"),
|
||
Self::Security => write!(f, "SECURITY"),
|
||
Self::BusinessLogic => write!(f, "BUSINESS_LOGIC"),
|
||
Self::Resource => write!(f, "RESOURCE"),
|
||
Self::Development => write!(f, "DEVELOPMENT"),
|
||
Self::Risk => write!(f, "RISK"),
|
||
Self::ML => write!(f, "ML"),
|
||
Self::Other => write!(f, "OTHER"),
|
||
}
|
||
}
|
||
}
|
||
|
||
/// Error severity levels for prioritization and alerting
|
||
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)]
|
||
#[allow(clippy::module_name_repetitions)]
|
||
pub enum ErrorSeverity {
|
||
/// Debug level - for development and troubleshooting
|
||
Debug,
|
||
/// Low severity - informational errors (alias for Info)
|
||
Low,
|
||
/// Info level - informational messages
|
||
Info,
|
||
/// Medium severity - recoverable errors (alias for Warn)
|
||
Medium,
|
||
/// Warning level - potentially problematic situations
|
||
Warn,
|
||
/// High severity - significant errors requiring attention (alias for Error)
|
||
High,
|
||
/// Error level - error conditions that should be addressed
|
||
Error,
|
||
/// Critical level - serious error conditions requiring immediate attention
|
||
Critical,
|
||
}
|
||
|
||
impl fmt::Display for ErrorSeverity {
|
||
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
|
||
match self {
|
||
Self::Debug => write!(f, "DEBUG"),
|
||
Self::Low => write!(f, "LOW"),
|
||
Self::Info => write!(f, "INFO"),
|
||
Self::Medium => write!(f, "MEDIUM"),
|
||
Self::Warn => write!(f, "WARN"),
|
||
Self::High => write!(f, "HIGH"),
|
||
Self::Error => write!(f, "ERROR"),
|
||
Self::Critical => write!(f, "CRITICAL"),
|
||
}
|
||
}
|
||
}
|
||
|
||
/// Retry strategies for error recovery
|
||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
|
||
pub enum RetryStrategy {
|
||
/// Do not retry - error is permanent
|
||
NoRetry,
|
||
/// Retry immediately without delay
|
||
Immediate,
|
||
/// Linear backoff with fixed intervals
|
||
Linear {
|
||
/// Base delay in milliseconds between retries
|
||
base_delay_ms: u64,
|
||
},
|
||
/// Exponential backoff with jitter
|
||
Exponential {
|
||
/// Base delay in milliseconds for exponential backoff
|
||
base_delay_ms: u64,
|
||
/// Maximum delay cap in milliseconds
|
||
max_delay_ms: u64,
|
||
},
|
||
/// Wait for circuit breaker to close
|
||
CircuitBreaker,
|
||
}
|
||
|
||
impl RetryStrategy {
|
||
/// Calculate delay for retry attempt
|
||
#[must_use]
|
||
pub fn calculate_delay(&self, attempt: u32) -> Option<Duration> {
|
||
match self {
|
||
Self::NoRetry => None,
|
||
Self::Immediate => Some(Duration::from_millis(0)),
|
||
Self::Linear { base_delay_ms } => Some(Duration::from_millis(
|
||
base_delay_ms.saturating_mul(u64::from(attempt)),
|
||
)),
|
||
Self::Exponential {
|
||
base_delay_ms,
|
||
max_delay_ms,
|
||
} => {
|
||
let delay_ms = base_delay_ms.saturating_mul(2_u64.saturating_pow(attempt.min(10)));
|
||
let capped_delay = delay_ms.min(*max_delay_ms);
|
||
|
||
// Add simple jitter (±10%)
|
||
#[allow(clippy::integer_division)]
|
||
let jitter_ms = capped_delay / 10;
|
||
#[allow(clippy::integer_division)]
|
||
let final_delay = capped_delay.saturating_sub(jitter_ms / 2);
|
||
|
||
Some(Duration::from_millis(final_delay))
|
||
},
|
||
Self::CircuitBreaker => Some(Duration::from_secs(30)),
|
||
}
|
||
}
|
||
|
||
/// Get maximum recommended retry attempts
|
||
#[must_use]
|
||
pub const fn max_attempts(&self) -> Option<u32> {
|
||
match self {
|
||
Self::NoRetry => Some(0),
|
||
Self::Immediate => Some(3),
|
||
Self::Linear { .. } => Some(5),
|
||
Self::Exponential { .. } => Some(7),
|
||
Self::CircuitBreaker => Some(1),
|
||
}
|
||
}
|
||
}
|
||
|
||
/// Convenience functions for creating common errors
|
||
impl CommonError {
|
||
/// Create a configuration error
|
||
pub fn config<S: Into<String>>(message: S) -> Self {
|
||
Self::Configuration(message.into())
|
||
}
|
||
|
||
/// Create a network error
|
||
pub fn network<S: Into<String>>(message: S) -> Self {
|
||
Self::Network(message.into())
|
||
}
|
||
|
||
/// Create a service error with category
|
||
pub fn service<S: Into<String>>(category: ErrorCategory, message: S) -> Self {
|
||
Self::Service {
|
||
category,
|
||
message: message.into(),
|
||
}
|
||
}
|
||
|
||
/// Create a validation error
|
||
pub fn validation<S: Into<String>>(message: S) -> Self {
|
||
Self::Validation(message.into())
|
||
}
|
||
|
||
/// Create a timeout error
|
||
pub const fn timeout(actual_ms: u64, max_ms: u64) -> Self {
|
||
Self::Timeout { actual_ms, max_ms }
|
||
}
|
||
|
||
/// Create a machine learning specific service error
|
||
pub fn ml<S: Into<String>, M: Into<String>>(model_name: S, message: M) -> Self {
|
||
Self::Service {
|
||
category: ErrorCategory::MachineLearning,
|
||
message: format!("{}: {}", model_name.into(), message.into()),
|
||
}
|
||
}
|
||
|
||
/// Create a serialization error
|
||
pub fn serialization<S: Into<String>>(message: S) -> Self {
|
||
Self::Service {
|
||
category: ErrorCategory::Parse,
|
||
message: format!("Serialization error: {}", message.into()),
|
||
}
|
||
}
|
||
|
||
/// Create an internal error
|
||
pub fn internal<S: Into<String>>(message: S) -> Self {
|
||
Self::Service {
|
||
category: ErrorCategory::System,
|
||
message: format!("Internal error: {}", message.into()),
|
||
}
|
||
}
|
||
|
||
/// Create a resource exhausted error
|
||
pub fn resource_exhausted<S: Into<String>>(resource: S) -> Self {
|
||
Self::Service {
|
||
category: ErrorCategory::Resource,
|
||
message: format!("Resource exhausted: {}", resource.into()),
|
||
}
|
||
}
|
||
|
||
/// Get the error category for classification and metrics
|
||
pub const fn category(&self) -> ErrorCategory {
|
||
match self {
|
||
Self::Database(_) => ErrorCategory::Database,
|
||
Self::Configuration(_) => ErrorCategory::Configuration,
|
||
Self::Network(_) => ErrorCategory::Network,
|
||
Self::Service { category, .. } => *category,
|
||
Self::Validation(_) => ErrorCategory::Validation,
|
||
Self::Timeout { .. } => ErrorCategory::System,
|
||
Self::RegressionDetected { .. } => ErrorCategory::MachineLearning,
|
||
}
|
||
}
|
||
|
||
/// Get error severity level
|
||
pub const fn severity(&self) -> ErrorSeverity {
|
||
match self {
|
||
Self::Database(_) => ErrorSeverity::Critical,
|
||
Self::Configuration(_) => ErrorSeverity::Critical,
|
||
Self::Network(_) => ErrorSeverity::Error,
|
||
Self::Service { category, .. } => match category {
|
||
ErrorCategory::Critical
|
||
| ErrorCategory::FinancialSafety
|
||
| ErrorCategory::Authentication => ErrorSeverity::Critical,
|
||
ErrorCategory::Trading
|
||
| ErrorCategory::RiskManagement
|
||
| ErrorCategory::Database => ErrorSeverity::Error,
|
||
_ => ErrorSeverity::Warn,
|
||
},
|
||
Self::Validation(_) => ErrorSeverity::Warn,
|
||
Self::Timeout { .. } => ErrorSeverity::Error,
|
||
Self::RegressionDetected { .. } => ErrorSeverity::Critical,
|
||
}
|
||
}
|
||
|
||
/// Check if the error is retryable
|
||
pub const fn is_retryable(&self) -> bool {
|
||
match self {
|
||
Self::Database(_) => true, // Database operations can be retried
|
||
Self::Configuration(_) => false, // Configuration errors are permanent
|
||
Self::Network(_) => true, // Network errors are often transient
|
||
Self::Service { category, .. } => !matches!(
|
||
category,
|
||
ErrorCategory::Authentication
|
||
| ErrorCategory::Configuration
|
||
| ErrorCategory::Validation
|
||
),
|
||
Self::Validation(_) => false, // Validation errors are permanent
|
||
Self::Timeout { .. } => true, // Timeouts can be retried
|
||
// A regression hard-stop terminates the run intentionally; retry
|
||
// requires human diagnosis of the regression first, not an automatic
|
||
// re-attempt that would just trip the same band again.
|
||
Self::RegressionDetected { .. } => false,
|
||
}
|
||
}
|
||
|
||
/// Get retry strategy for this error
|
||
pub const fn retry_strategy(&self) -> RetryStrategy {
|
||
if !self.is_retryable() {
|
||
return RetryStrategy::NoRetry;
|
||
}
|
||
|
||
match self {
|
||
Self::Database(_) => RetryStrategy::Exponential {
|
||
base_delay_ms: 1000,
|
||
max_delay_ms: 10000,
|
||
},
|
||
Self::Network(_) => RetryStrategy::Linear { base_delay_ms: 500 },
|
||
Self::Service { category, .. } => match category {
|
||
ErrorCategory::Network | ErrorCategory::Connection => {
|
||
RetryStrategy::Linear { base_delay_ms: 500 }
|
||
},
|
||
ErrorCategory::RateLimit => RetryStrategy::Exponential {
|
||
base_delay_ms: 5000,
|
||
max_delay_ms: 60000,
|
||
},
|
||
_ => RetryStrategy::Immediate,
|
||
},
|
||
Self::Timeout { .. } => RetryStrategy::Linear {
|
||
base_delay_ms: 1000,
|
||
},
|
||
_ => RetryStrategy::NoRetry,
|
||
}
|
||
}
|
||
}
|
||
|
||
/// Result type for common operations
|
||
pub type CommonResult<T> = Result<T, CommonError>;
|