Files
foxhunt/crates/common/src/error.rs
jgrusewski 6cdfbff8d6 plan5(task2): A.4 regression-detection hard-stop on 2N consecutive error-band
Adds the convergence guardrail: every per-epoch HEALTH_DIAG metric is
checked against the bands in config/metric-bands.toml; N consecutive
warn-band epochs emit a tracing::warn; 2N consecutive error-band epochs
return Err(CommonError::RegressionDetected{...}) cleanly from the
training loop, which propagates to the train_baseline_rl subprocess
exit code (no libc::raise — clean Rust error path).

Wire-points:
- New module: crates/ml/src/trainers/dqn/trainer/monitoring.rs
  - MetricBands {warn_low, warn_high, error_low, error_high}
  - BandSettings {consecutive_epochs_for_warn, consecutive_epochs_for_error}
  - MetricBandsRegistry: load_from_toml + update_and_check
  - TerminationReason {RegressionWarn, RegressionError}
  - NaN treated as out-of-band (consecutive++; never resets streak)
  - Unknown metrics return None (silent OK per Invariant 7 audit)
- crates/common/src/error.rs: new CommonError::RegressionDetected variant
  carrying {metric, value, band, consecutive}
- crates/ml/src/trainers/dqn/trainer/constructor.rs: load
  config/metric-bands.toml at trainer init; warn-only on missing file
  (backward compat for environments without the config)
- crates/ml/src/trainers/dqn/trainer/training_loop.rs: harvest per-epoch
  metrics (parallel emit alongside HEALTH_DIAG), feed each through
  registry.update_and_check; on Some(TerminationReason::RegressionError)
  emit final HEALTH_DIAG[N]: TERMINATED_BY_REGRESSION line and return Err
- services/trading_service/src/error.rs: minimal handler for the new
  CommonError variant (existing pattern)

Validation:
- 8 unit tests in monitoring::tests pass (band logic, NaN, warn-only
  behaviour, error-streak threshold, unknown-metric, invalid TOML)
- regression_detection GPU smoke (3.19s): trainer with intentionally
  narrow train_loss error band [0, 1e-9] self-terminates at epoch 5
  after 6 consecutive error-band epochs; final HEALTH_DIAG line emits
  TERMINATED_BY_REGRESSION with metric/value/consecutive/band fields
- multi_fold_convergence smoke (650s, --release): all 3 folds train
  to completion, all 3 checkpoints saved, no false-positive
  termination on the populated metric bands. Per-fold best train
  Sharpe: F0=-9.7831 (bit-baseline), F1=25.8272, F2=39.2687. F1/F2
  on the lower end of observed noise distribution
  ({74.56, 61.10, 71.53, 25.83} for F1; {88.20, 61.57, 65.96, 39.27}
  for F2) but training healthy throughout: aux clauses fire every
  epoch, sharpe_ema recovers from F0 collapse (-9.78 → +14.8 by start
  of F2), no regression detection trips.

config/metric-bands.toml populated for the metrics emitted by
HEALTH_DIAG today (avg_q_value, train_loss, val_sharpe, train_sharpe,
aux_next_bar_mse, aux_regime_ce, isv_* slot EMAs, sharpe_ema, etc.).
Bands derived from current cleanroom smoke + permissive defaults
where only one sample exists; populate-metric-bands-from-runs.py will
tighten them after Plan 5 Task 5's multi-seed pass produces real
distributions.

Constraints honoured: GPU-only in hot path (band check is CPU-side
post-HEALTH_DIAG, off the captured graph); no atomicAdd; no stubs;
no // ok: band-aids; no tuned constants beyond the toml-loaded bands;
no .unwrap() introduced; cargo check clean at 11 warnings (workspace
baseline preserved, plus ml-dqn pre-existing 1 warning).

Audit doc: new row added documenting monitoring.rs module, the
CommonError variant, the training_loop wire-point, and the design
choice that band-checks run AFTER HEALTH_DIAG emit (not before) so
the diag log already reflects the metric values that triggered any
termination.

Plan 5 T1 (multi-seed harness) landed at c6634254e+47c8b783c; T2
(this) gives the regression hard-stop that the multi-seed final
pass (T5) consumes to bail out early on bad seeds.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-26 11:49:14 +02:00

410 lines
15 KiB
Rust
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
//! Common error types and utilities
//!
//! This module provides shared error types and utilities used across
//! all Foxhunt services.
use serde::{Deserialize, Serialize};
use std::fmt;
use std::time::Duration;
use thiserror::Error;
/// Common error type for all Foxhunt services
#[derive(Debug, Error)]
pub enum CommonError {
/// Database operation failed - wraps database-specific errors
#[error("Database error: {0}")]
Database(#[from] crate::database::DatabaseError),
/// Configuration is invalid or missing required parameters
#[error("Configuration error: {0}")]
Configuration(String),
/// Network communication error occurred
#[error("Network error: {0}")]
Network(String),
/// Service-specific error with categorization for metrics
#[error("Service error: {category} - {message}")]
Service {
/// Error category for classification
category: ErrorCategory,
/// Descriptive error message
message: String,
},
/// Input validation failed
#[error("Validation error: {0}")]
Validation(String),
/// Operation exceeded maximum allowed execution time
#[error("Timeout error: operation took {actual_ms}ms, max allowed {max_ms}ms")]
Timeout {
/// Actual execution time in milliseconds
actual_ms: u64,
/// Maximum allowed execution time in milliseconds
max_ms: u64,
},
/// Plan 5 Task 2 — A.4 Regression Detection Hard-Stop.
///
/// Fired by `MetricBandsRegistry::update_and_check` after `2N` consecutive
/// epochs of an out-of-band HEALTH_DIAG metric (per `config/metric-bands.toml`).
/// `band_*` fields are flat doubles (not the `MetricBands` struct) so this
/// variant has no upward dependency on the `ml` crate. Trainer consumes the
/// variant by returning it from `train_with_data_full_loop_slices`; the call
/// stack unwinds cleanly. There is no SIGTERM raise — the error itself is
/// the termination signal.
#[error(
"Regression detected: metric `{metric}` value {value:.6e} out of error band \
[{band_error_low:.6e}, {band_error_high:.6e}] for {consecutive} consecutive epochs"
)]
RegressionDetected {
/// HEALTH_DIAG metric name (matches a key in `config/metric-bands.toml`).
metric: String,
/// Value at the terminating epoch.
value: f64,
/// Warn-band lower bound (`mean - 5σ` per Task 2.5).
band_warn_low: f64,
/// Warn-band upper bound (`mean + 5σ` per Task 2.5).
band_warn_high: f64,
/// Error-band lower bound (`mean - 10σ` per Task 2.5).
band_error_low: f64,
/// Error-band upper bound (`mean + 10σ` per Task 2.5).
band_error_high: f64,
/// Consecutive epochs out-of-error-band at termination
/// (always equals `2 * settings.consecutive_epochs_for_warn`).
consecutive: u32,
},
}
/// Error categories for classification and metrics
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[allow(clippy::module_name_repetitions)]
pub enum ErrorCategory {
/// Market data related errors
MarketData,
/// Trading and order management errors
Trading,
/// Network and communication errors
Network,
/// System and infrastructure errors
System,
/// Configuration errors
Configuration,
/// Validation errors
Validation,
/// Critical errors requiring immediate attention
Critical,
/// Connection errors (data providers)
Connection,
/// Authentication errors
Authentication,
/// Rate limiting errors
RateLimit,
/// Data parsing errors
Parse,
/// Subscription errors
Subscription,
/// Financial safety and calculation errors
FinancialSafety,
/// Risk management and circuit breakers
RiskManagement,
/// Database and persistence layer
Database,
/// Broker connectivity and execution
Broker,
/// Machine learning and AI errors
MachineLearning,
/// Security and authentication errors
Security,
/// Business logic errors
BusinessLogic,
/// Resource errors (not found, conflicts)
Resource,
/// Development and testing errors
Development,
/// Risk management errors
Risk,
/// Machine learning errors (alias for `MachineLearning`)
ML,
/// Unknown/other errors
Other,
}
impl fmt::Display for ErrorCategory {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
Self::MarketData => write!(f, "MARKET_DATA"),
Self::Trading => write!(f, "TRADING"),
Self::Network => write!(f, "NETWORK"),
Self::System => write!(f, "SYSTEM"),
Self::Configuration => write!(f, "CONFIGURATION"),
Self::Validation => write!(f, "VALIDATION"),
Self::Critical => write!(f, "CRITICAL"),
Self::Connection => write!(f, "CONNECTION"),
Self::Authentication => write!(f, "AUTHENTICATION"),
Self::RateLimit => write!(f, "RATE_LIMIT"),
Self::Parse => write!(f, "PARSE"),
Self::Subscription => write!(f, "SUBSCRIPTION"),
Self::FinancialSafety => write!(f, "FINANCIAL_SAFETY"),
Self::RiskManagement => write!(f, "RISK_MANAGEMENT"),
Self::Database => write!(f, "DATABASE"),
Self::Broker => write!(f, "BROKER"),
Self::MachineLearning => write!(f, "MACHINE_LEARNING"),
Self::Security => write!(f, "SECURITY"),
Self::BusinessLogic => write!(f, "BUSINESS_LOGIC"),
Self::Resource => write!(f, "RESOURCE"),
Self::Development => write!(f, "DEVELOPMENT"),
Self::Risk => write!(f, "RISK"),
Self::ML => write!(f, "ML"),
Self::Other => write!(f, "OTHER"),
}
}
}
/// Error severity levels for prioritization and alerting
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)]
#[allow(clippy::module_name_repetitions)]
pub enum ErrorSeverity {
/// Debug level - for development and troubleshooting
Debug,
/// Low severity - informational errors (alias for Info)
Low,
/// Info level - informational messages
Info,
/// Medium severity - recoverable errors (alias for Warn)
Medium,
/// Warning level - potentially problematic situations
Warn,
/// High severity - significant errors requiring attention (alias for Error)
High,
/// Error level - error conditions that should be addressed
Error,
/// Critical level - serious error conditions requiring immediate attention
Critical,
}
impl fmt::Display for ErrorSeverity {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
Self::Debug => write!(f, "DEBUG"),
Self::Low => write!(f, "LOW"),
Self::Info => write!(f, "INFO"),
Self::Medium => write!(f, "MEDIUM"),
Self::Warn => write!(f, "WARN"),
Self::High => write!(f, "HIGH"),
Self::Error => write!(f, "ERROR"),
Self::Critical => write!(f, "CRITICAL"),
}
}
}
/// Retry strategies for error recovery
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub enum RetryStrategy {
/// Do not retry - error is permanent
NoRetry,
/// Retry immediately without delay
Immediate,
/// Linear backoff with fixed intervals
Linear {
/// Base delay in milliseconds between retries
base_delay_ms: u64,
},
/// Exponential backoff with jitter
Exponential {
/// Base delay in milliseconds for exponential backoff
base_delay_ms: u64,
/// Maximum delay cap in milliseconds
max_delay_ms: u64,
},
/// Wait for circuit breaker to close
CircuitBreaker,
}
impl RetryStrategy {
/// Calculate delay for retry attempt
#[must_use]
pub fn calculate_delay(&self, attempt: u32) -> Option<Duration> {
match self {
Self::NoRetry => None,
Self::Immediate => Some(Duration::from_millis(0)),
Self::Linear { base_delay_ms } => Some(Duration::from_millis(
base_delay_ms.saturating_mul(u64::from(attempt)),
)),
Self::Exponential {
base_delay_ms,
max_delay_ms,
} => {
let delay_ms = base_delay_ms.saturating_mul(2_u64.saturating_pow(attempt.min(10)));
let capped_delay = delay_ms.min(*max_delay_ms);
// Add simple jitter (±10%)
#[allow(clippy::integer_division)]
let jitter_ms = capped_delay / 10;
#[allow(clippy::integer_division)]
let final_delay = capped_delay.saturating_sub(jitter_ms / 2);
Some(Duration::from_millis(final_delay))
},
Self::CircuitBreaker => Some(Duration::from_secs(30)),
}
}
/// Get maximum recommended retry attempts
#[must_use]
pub const fn max_attempts(&self) -> Option<u32> {
match self {
Self::NoRetry => Some(0),
Self::Immediate => Some(3),
Self::Linear { .. } => Some(5),
Self::Exponential { .. } => Some(7),
Self::CircuitBreaker => Some(1),
}
}
}
/// Convenience functions for creating common errors
impl CommonError {
/// Create a configuration error
pub fn config<S: Into<String>>(message: S) -> Self {
Self::Configuration(message.into())
}
/// Create a network error
pub fn network<S: Into<String>>(message: S) -> Self {
Self::Network(message.into())
}
/// Create a service error with category
pub fn service<S: Into<String>>(category: ErrorCategory, message: S) -> Self {
Self::Service {
category,
message: message.into(),
}
}
/// Create a validation error
pub fn validation<S: Into<String>>(message: S) -> Self {
Self::Validation(message.into())
}
/// Create a timeout error
pub const fn timeout(actual_ms: u64, max_ms: u64) -> Self {
Self::Timeout { actual_ms, max_ms }
}
/// Create a machine learning specific service error
pub fn ml<S: Into<String>, M: Into<String>>(model_name: S, message: M) -> Self {
Self::Service {
category: ErrorCategory::MachineLearning,
message: format!("{}: {}", model_name.into(), message.into()),
}
}
/// Create a serialization error
pub fn serialization<S: Into<String>>(message: S) -> Self {
Self::Service {
category: ErrorCategory::Parse,
message: format!("Serialization error: {}", message.into()),
}
}
/// Create an internal error
pub fn internal<S: Into<String>>(message: S) -> Self {
Self::Service {
category: ErrorCategory::System,
message: format!("Internal error: {}", message.into()),
}
}
/// Create a resource exhausted error
pub fn resource_exhausted<S: Into<String>>(resource: S) -> Self {
Self::Service {
category: ErrorCategory::Resource,
message: format!("Resource exhausted: {}", resource.into()),
}
}
/// Get the error category for classification and metrics
pub const fn category(&self) -> ErrorCategory {
match self {
Self::Database(_) => ErrorCategory::Database,
Self::Configuration(_) => ErrorCategory::Configuration,
Self::Network(_) => ErrorCategory::Network,
Self::Service { category, .. } => *category,
Self::Validation(_) => ErrorCategory::Validation,
Self::Timeout { .. } => ErrorCategory::System,
Self::RegressionDetected { .. } => ErrorCategory::MachineLearning,
}
}
/// Get error severity level
pub const fn severity(&self) -> ErrorSeverity {
match self {
Self::Database(_) => ErrorSeverity::Critical,
Self::Configuration(_) => ErrorSeverity::Critical,
Self::Network(_) => ErrorSeverity::Error,
Self::Service { category, .. } => match category {
ErrorCategory::Critical
| ErrorCategory::FinancialSafety
| ErrorCategory::Authentication => ErrorSeverity::Critical,
ErrorCategory::Trading
| ErrorCategory::RiskManagement
| ErrorCategory::Database => ErrorSeverity::Error,
_ => ErrorSeverity::Warn,
},
Self::Validation(_) => ErrorSeverity::Warn,
Self::Timeout { .. } => ErrorSeverity::Error,
Self::RegressionDetected { .. } => ErrorSeverity::Critical,
}
}
/// Check if the error is retryable
pub const fn is_retryable(&self) -> bool {
match self {
Self::Database(_) => true, // Database operations can be retried
Self::Configuration(_) => false, // Configuration errors are permanent
Self::Network(_) => true, // Network errors are often transient
Self::Service { category, .. } => !matches!(
category,
ErrorCategory::Authentication
| ErrorCategory::Configuration
| ErrorCategory::Validation
),
Self::Validation(_) => false, // Validation errors are permanent
Self::Timeout { .. } => true, // Timeouts can be retried
// A regression hard-stop terminates the run intentionally; retry
// requires human diagnosis of the regression first, not an automatic
// re-attempt that would just trip the same band again.
Self::RegressionDetected { .. } => false,
}
}
/// Get retry strategy for this error
pub const fn retry_strategy(&self) -> RetryStrategy {
if !self.is_retryable() {
return RetryStrategy::NoRetry;
}
match self {
Self::Database(_) => RetryStrategy::Exponential {
base_delay_ms: 1000,
max_delay_ms: 10000,
},
Self::Network(_) => RetryStrategy::Linear { base_delay_ms: 500 },
Self::Service { category, .. } => match category {
ErrorCategory::Network | ErrorCategory::Connection => {
RetryStrategy::Linear { base_delay_ms: 500 }
},
ErrorCategory::RateLimit => RetryStrategy::Exponential {
base_delay_ms: 5000,
max_delay_ms: 60000,
},
_ => RetryStrategy::Immediate,
},
Self::Timeout { .. } => RetryStrategy::Linear {
base_delay_ms: 1000,
},
_ => RetryStrategy::NoRetry,
}
}
}
/// Result type for common operations
pub type CommonResult<T> = Result<T, CommonError>;