Files
foxhunt/services/trading_service/src/error.rs
jgrusewski 6cdfbff8d6 plan5(task2): A.4 regression-detection hard-stop on 2N consecutive error-band
Adds the convergence guardrail: every per-epoch HEALTH_DIAG metric is
checked against the bands in config/metric-bands.toml; N consecutive
warn-band epochs emit a tracing::warn; 2N consecutive error-band epochs
return Err(CommonError::RegressionDetected{...}) cleanly from the
training loop, which propagates to the train_baseline_rl subprocess
exit code (no libc::raise — clean Rust error path).

Wire-points:
- New module: crates/ml/src/trainers/dqn/trainer/monitoring.rs
  - MetricBands {warn_low, warn_high, error_low, error_high}
  - BandSettings {consecutive_epochs_for_warn, consecutive_epochs_for_error}
  - MetricBandsRegistry: load_from_toml + update_and_check
  - TerminationReason {RegressionWarn, RegressionError}
  - NaN treated as out-of-band (consecutive++; never resets streak)
  - Unknown metrics return None (silent OK per Invariant 7 audit)
- crates/common/src/error.rs: new CommonError::RegressionDetected variant
  carrying {metric, value, band, consecutive}
- crates/ml/src/trainers/dqn/trainer/constructor.rs: load
  config/metric-bands.toml at trainer init; warn-only on missing file
  (backward compat for environments without the config)
- crates/ml/src/trainers/dqn/trainer/training_loop.rs: harvest per-epoch
  metrics (parallel emit alongside HEALTH_DIAG), feed each through
  registry.update_and_check; on Some(TerminationReason::RegressionError)
  emit final HEALTH_DIAG[N]: TERMINATED_BY_REGRESSION line and return Err
- services/trading_service/src/error.rs: minimal handler for the new
  CommonError variant (existing pattern)

Validation:
- 8 unit tests in monitoring::tests pass (band logic, NaN, warn-only
  behaviour, error-streak threshold, unknown-metric, invalid TOML)
- regression_detection GPU smoke (3.19s): trainer with intentionally
  narrow train_loss error band [0, 1e-9] self-terminates at epoch 5
  after 6 consecutive error-band epochs; final HEALTH_DIAG line emits
  TERMINATED_BY_REGRESSION with metric/value/consecutive/band fields
- multi_fold_convergence smoke (650s, --release): all 3 folds train
  to completion, all 3 checkpoints saved, no false-positive
  termination on the populated metric bands. Per-fold best train
  Sharpe: F0=-9.7831 (bit-baseline), F1=25.8272, F2=39.2687. F1/F2
  on the lower end of observed noise distribution
  ({74.56, 61.10, 71.53, 25.83} for F1; {88.20, 61.57, 65.96, 39.27}
  for F2) but training healthy throughout: aux clauses fire every
  epoch, sharpe_ema recovers from F0 collapse (-9.78 → +14.8 by start
  of F2), no regression detection trips.

config/metric-bands.toml populated for the metrics emitted by
HEALTH_DIAG today (avg_q_value, train_loss, val_sharpe, train_sharpe,
aux_next_bar_mse, aux_regime_ce, isv_* slot EMAs, sharpe_ema, etc.).
Bands derived from current cleanroom smoke + permissive defaults
where only one sample exists; populate-metric-bands-from-runs.py will
tighten them after Plan 5 Task 5's multi-seed pass produces real
distributions.

Constraints honoured: GPU-only in hot path (band check is CPU-side
post-HEALTH_DIAG, off the captured graph); no atomicAdd; no stubs;
no // ok: band-aids; no tuned constants beyond the toml-loaded bands;
no .unwrap() introduced; cargo check clean at 11 warnings (workspace
baseline preserved, plus ml-dqn pre-existing 1 warning).

Audit doc: new row added documenting monitoring.rs module, the
CommonError variant, the training_loop wire-point, and the design
choice that band-checks run AFTER HEALTH_DIAG emit (not before) so
the diag log already reflects the metric values that triggered any
termination.

Plan 5 T1 (multi-seed harness) landed at c6634254e+47c8b783c; T2
(this) gives the regression hard-stop that the multi-seed final
pass (T5) consumes to bail out early on bad seeds.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-26 11:49:14 +02:00

164 lines
6.8 KiB
Rust

//! Error types for the Trading Service - Using Shared Library Types
// REMOVED: All pub use statements eliminated per cleanup requirements
// Use direct import: common::error::CommonError
/// Trading service specific error extensions
///
/// For cases where we need domain-specific error information
#[derive(Debug, thiserror::Error)]
pub enum TradingServiceError {
/// Shared library error with context
#[error("Trading service error: {0}")]
Common(#[from] common::error::CommonError),
/// `Order` validation failed with specific trading context
#[error("Order validation failed: {reason}")]
OrderValidation { reason: String },
/// Risk management violation with trading-specific details
#[error("Risk violation: {violation_type} - {message}")]
RiskViolation {
violation_type: String,
message: String,
},
/// ML model error with model context
#[error("ML model error: {model_name} - {message}")]
MLModel { model_name: String, message: String },
/// Database operation error
#[error("Database error: {source}")]
DatabaseError {
source: Box<dyn std::error::Error + Send + Sync>,
},
/// Configuration error
#[error("Configuration error: {message}")]
ConfigurationError { message: String },
/// Rate limit exceeded error
#[error("Rate limit exceeded: {message}")]
RateLimitExceeded { message: String },
/// Subscription timeout error
#[error("Subscription timeout: {message}")]
SubscriptionTimeout { message: String },
/// Subscription closed error
#[error("Subscription closed: {message}")]
SubscriptionClosed { message: String },
/// Internal service error
#[error("Internal error: {message}")]
Internal { message: String },
/// Timestamp conversion error
#[error("Invalid timestamp: {timestamp} - cannot convert to DateTime")]
TimestampConversion { timestamp: i64 },
/// Validation error
#[error("Validation error: {message}")]
ValidationError { message: String },
}
/// Result type for trading service operations
pub type TradingServiceResult<T> = std::result::Result<T, TradingServiceError>;
/// Convenience type alias using common result
pub type Result<T> = TradingServiceResult<T>;
/// Convert TradingServiceError to tonic::Status for gRPC responses
impl From<TradingServiceError> for tonic::Status {
fn from(err: TradingServiceError) -> Self {
match err {
TradingServiceError::Common(common_err) => {
// Leverage shared error to gRPC status conversion
match common_err {
common::error::CommonError::Validation(ref msg) => {
tonic::Status::invalid_argument(format!("Validation error: {}", msg))
},
common::error::CommonError::Configuration(ref msg) => {
tonic::Status::invalid_argument(format!("Configuration error: {}", msg))
},
common::error::CommonError::Network(ref msg) => {
tonic::Status::unavailable(format!("Network error: {}", msg))
},
common::error::CommonError::Database(ref db_err) => {
tonic::Status::internal(format!("Database error: {}", db_err))
},
common::error::CommonError::Service { category, message } => match category {
common::error::ErrorCategory::Authentication => {
tonic::Status::unauthenticated(message)
},
common::error::ErrorCategory::Resource => {
tonic::Status::not_found(message)
},
common::error::ErrorCategory::Validation => {
tonic::Status::invalid_argument(message)
},
_ => tonic::Status::internal(format!("{}: {}", category, message)),
},
common::error::CommonError::Timeout { actual_ms, max_ms } => {
tonic::Status::deadline_exceeded(format!(
"Operation timed out: {}ms (max: {}ms)",
actual_ms, max_ms
))
},
common::error::CommonError::RegressionDetected {
ref metric,
value,
consecutive,
..
} => tonic::Status::failed_precondition(format!(
"ML training regression detected: metric={} value={:.6e} \
consecutive_epochs={}",
metric, value, consecutive
)),
}
},
TradingServiceError::OrderValidation { reason } => {
tonic::Status::invalid_argument(format!("Order validation failed: {}", reason))
},
TradingServiceError::RiskViolation {
violation_type,
message,
} => tonic::Status::failed_precondition(format!(
"Risk violation {}: {}",
violation_type, message
)),
TradingServiceError::MLModel {
model_name,
message,
} => tonic::Status::internal(format!("ML model {} error: {}", model_name, message)),
TradingServiceError::DatabaseError { source } => {
tonic::Status::internal(format!("Database error: {}", source))
},
TradingServiceError::ConfigurationError { message } => {
tonic::Status::internal(format!("Configuration error: {}", message))
},
TradingServiceError::RateLimitExceeded { message } => {
tonic::Status::resource_exhausted(format!("Rate limit exceeded: {}", message))
},
TradingServiceError::SubscriptionTimeout { message } => {
tonic::Status::deadline_exceeded(format!("Subscription timeout: {}", message))
},
TradingServiceError::SubscriptionClosed { message } => {
tonic::Status::cancelled(format!("Subscription closed: {}", message))
},
TradingServiceError::Internal { message } => {
tonic::Status::internal(format!("Internal error: {}", message))
},
TradingServiceError::TimestampConversion { timestamp } => {
tonic::Status::invalid_argument(format!(
"Invalid timestamp conversion: {}",
timestamp
))
},
TradingServiceError::ValidationError { message } => {
tonic::Status::invalid_argument(format!("Validation error: {}", message))
},
}
}
}