diff --git a/docs/plans/2026-02-28-numeric-type-standardization-design.md b/docs/plans/2026-02-28-numeric-type-standardization-design.md new file mode 100644 index 000000000..d81d7e2d6 --- /dev/null +++ b/docs/plans/2026-02-28-numeric-type-standardization-design.md @@ -0,0 +1,228 @@ +# Numeric Type Standardization Design + +**Date**: 2026-02-28 +**Status**: Approved +**Goal**: Eliminate precision-lossy f64 casting by standardizing on domain-typed i64 fixed-point wrappers for all financial values + +## Problem + +- 23,730 `as` casts across 1,044 files — many are f32↔f64 conversions at ML↔trading boundaries +- 11,000+ raw f64 occurrences for prices, quantities, P&L — no type safety between different financial quantities +- Two competing implementations: `common::Price` (u64, 8dp) vs `trading_engine::IntegerPrice` (i64, 6dp) +- Neither widely adopted — raw `price: f64` still dominates (88+ field occurrences) +- Proactive hardening: prevent precision bugs before they affect P&L or risk calculations + +## Decision + +**Approach C: Domain-typed wrappers with i64 fixed-point internals (6 decimal places)** + +Rejected alternatives: +- **rust_decimal::Decimal everywhere**: 3-10x slower than f64, no sqrt/ln/exp, kills SIMD optimizations in trading_engine +- **Fixed-point tick integers**: Correct for HFT but requires per-instrument tick tables and is too large an architectural change +- **u64 Price (current common::Price)**: Cannot represent negative values (P&L, unrealized losses) + +## Canonical Type Set + +All types live in `crates/common/src/` and are re-exported from `common::`. + +### Price (i64, scale 1_000_000) + +Represents: bid, ask, mid, last, VWAP, limit price, stop price, fill price, OHLCV values. + +```rust +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)] +pub struct Price(i64); + +impl Price { + pub const ZERO: Self = Self(0); + pub const SCALE: i64 = 1_000_000; + + pub fn from_f64(value: f64) -> Result; + pub fn to_f64(self) -> f64; + pub fn to_f32(self) -> f32; // ML boundary only + pub fn from_f32(value: f32) -> Result; + pub fn to_decimal(self) -> Decimal; + pub fn from_decimal(decimal: Decimal) -> Self; + pub fn abs(self) -> Self; + pub fn sqrt(self) -> Self; // For volatility calculations on price diffs +} +``` + +6 decimal places sufficient for: ES (0.25), NQ (0.25), 6E (0.00005), ZN (1/64=0.015625), FX spot (0.00001), equities ($0.0001). + +### Quantity (i64, scale 1_000_000) + +Represents: contracts, shares, lots, position sizes. Signed to represent short positions. + +```rust +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)] +pub struct Quantity(i64); +``` + +### Money (i64, scale 1_000_000) + +Represents: P&L, margin, notional value, fees, commissions, account balance. Always signed. + +```rust +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)] +pub struct Money(i64); +``` + +### Ratio (f64 newtype) + +Represents: returns, percentages, volatility, Sharpe ratio, correlation, Kelly fraction — continuous mathematical quantities where IEEE 754 is correct. + +```rust +#[derive(Debug, Clone, Copy, PartialEq, PartialOrd, Serialize, Deserialize)] +pub struct Ratio(f64); + +impl Ratio { + pub const ZERO: Self = Self(0.0); + pub const ONE: Self = Self(1.0); + + pub fn new(value: f64) -> Result; // rejects NaN/Inf + pub fn to_f64(self) -> f64; + pub fn to_f32(self) -> f32; // ML boundary + pub fn abs(self) -> Self; + pub fn sqrt(self) -> Self; + pub fn ln(self) -> Self; + pub fn exp(self) -> Self; + pub fn powi(self, n: i32) -> Self; + pub fn is_positive(self) -> bool; + pub fn is_negative(self) -> bool; + pub fn clamp(self, min: f64, max: f64) -> Self; +} +``` + +Benefits of newtype over alias: +- **NaN/Inf rejection at construction** — invalid values caught at boundaries, not mid-calculation +- **Type safety** — can't accidentally assign a Ratio to a Price field +- **Audit trail** — all f64→Ratio conversions go through `Ratio::new()` which validates finiteness +- Implements standard math ops (Add, Sub, Mul, Div, Neg) delegating to f64 arithmetic + +## Arithmetic + +All types use saturating arithmetic (no panics on overflow): +- `Price + Price → Price` (saturating_add) +- `Price - Price → Price` (saturating_sub) +- `Price * Quantity → Money` (cross-type multiplication) +- `Money / Quantity → Price` (cross-type division) +- `Price * i64 → Price` (scaling) +- `Price / i64 → Price` (splitting) +- Division by zero returns `Self(0)` (consistent with existing IntegerPrice behavior) + +## Type Boundaries + +``` +┌─────────────────────┐ ┌──────────────────┐ ┌─────────────────┐ +│ FINANCIAL DOMAIN │ │ ML DOMAIN │ │ ANALYTICS │ +│ │ │ │ │ │ +│ Price(i64) │────▶│ f32 tensors │ │ Ratio (f64) │ +│ Quantity(i64) │ │ (Candle GPU) │────▶│ (returns, vol, │ +│ Money(i64) │◀────│ predictions f32 │ │ Sharpe, stats) │ +│ │ │ │ │ │ +└─────────────────────┘ └──────────────────┘ └─────────────────┘ +``` + +### Conversion rules + +- `Price → f64`: explicit `.to_f64()` only +- `f64 → Price`: explicit `Price::from_f64()` (fallible, rounds to scale) +- `Price → f32`: explicit `.to_f32()` (ML tensor input) +- `f32 → Price`: explicit `Price::from_f32()` (ML prediction output) +- **No `impl From for Price`** — all conversions are explicit and auditable +- **No implicit casting** — compiler enforces type boundaries + +### What stays raw f64/f32 + +- ML weights, gradients, losses, learning rates, activations (Candle tensor internals) +- Raw sensor/timing data (RDTSC timestamps, latency measurements) + +### What becomes Ratio + +- Returns, log-returns, percentage changes +- Volatility, implied vol, realized vol +- Sharpe ratio, Sortino, Calmar, information ratio +- Correlation coefficients, beta, R-squared +- Kelly fraction, position sizing multipliers +- Feature importance scores, confidence scores +- Any continuous mathematical quantity that is not a financial amount + +## Migration Strategy + +Bottom-up, dependency-driven, parallelized via git worktrees. + +### Phase 1: Foundation (1 worktree) + +**Crate**: `common/` + +- Consolidate `Price`/`Quantity` into i64/6dp (adopt IntegerPrice's API) +- Delete the old u64 `Price` type +- Add `Money` type (i64/6dp) +- Add `type Ratio = f64` +- Implement all arithmetic ops (Add, Sub, Mul, Div, Display, FromStr, Serialize) +- Implement cross-type ops (`Price * Quantity → Money`) +- Add Decimal interop (to_decimal, from_decimal) +- Update all `common/` internal usage +- All downstream crates will fail to compile — that's expected and intentional + +### Phase 2: Core crates (3 parallel worktrees) + +**Crates**: `trading_engine/`, `risk/`, `data/` + +- `trading_engine/`: delete `types/financial.rs` (IntegerPrice/IntegerQuantity/IntegerMoney), use `common::Price`/`Quantity`/`Money` everywhere +- `risk/`: replace f64 price/pnl/margin fields with Price/Money in risk_engine.rs, position_tracker.rs, risk_types.rs, VaR calculations +- `data/`: replace f64 in OHLCV structs, PricePoint, PriceBounds, validation, feature extraction + +### Phase 3: ML boundary (1 worktree) + +**Crate**: `ml/` + +- Replace f64 price fields in labeling (PricePoint, triple_barrier) with Price +- Replace f64 price fields in features (price_features, microstructure) with Price +- Add `.to_f32()` / `Price::from_f32()` at Candle tensor boundaries +- Keep f32/f64 for weights, gradients, losses, learning rates (these are NOT prices) +- Keep f64 for ensemble scores, confidence, feature importance (these are Ratios) + +### Phase 4: Services (4 parallel worktrees) + +**Crates**: `trading_service/`, `backtesting_service/`, `trading_agent_service/`, remaining services + +- Replace f64 price/qty/pnl fields in each service +- Update proto type conversions (f64 ↔ Price at gRPC boundary) +- Update sqlx queries (Decimal ↔ Price at DB boundary) + +### Phase 5: Peripheral (2 parallel worktrees) + +**Crates**: `bin/fxt/`, `web-gateway/`, `testing/` + +- Update proto-generated types +- Update API request/response types +- Update test fixtures and builders + +### Per-crate migration checklist + +1. Replace `price: f64` fields → `price: Price` +2. Replace `quantity: f64` fields → `quantity: Quantity` +3. Replace P&L/margin/notional f64 fields → `Money` +4. Replace `as f64` / `as f32` casts → `.to_f64()` / `.to_f32()` +5. Run `SQLX_OFFLINE=true cargo check -p ` +6. Fix compilation errors +7. Run `SQLX_OFFLINE=true cargo test -p --lib` +8. Run `cargo clippy -p ` + +## Scope Exclusions + +- No changes to Candle internals (f32 tensors stay f32) +- No changes to ML training loops (weights/gradients/losses stay f64/f32) +- Ratio is a proper f64 newtype with NaN/Inf rejection at construction +- No per-instrument tick size enforcement (future enhancement) +- No bigdecimal removal (deferred — only used in sqlx features) + +## Risk Mitigation + +- **Phased rollout**: each phase is independently compilable and testable +- **Worktree isolation**: parallel work doesn't conflict +- **4,500+ existing tests**: catch regressions immediately +- **Clippy deny rules**: `clippy::as_conversions` already denied — new types enforce this +- **Saturating arithmetic**: no panics on overflow (consistent with existing IntegerPrice)