Commit Graph

2782 Commits

Author SHA1 Message Date
jgrusewski
15c0fb5397 merge: pull liquid CfC v2 and codebase deduplication from main into production-hardening 2026-02-23 10:09:25 +01:00
jgrusewski
2e4b0c9455 fix(ml): per-model circuit breakers in ensemble — filter NaN/Inf, clamp confidence
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 10:06:53 +01:00
jgrusewski
5a2c92484c docs(risk): document Redis monitoring deferral for kill switch
Replace the TODO for spawning a background Redis health-check task
with an architectural decision comment. Local AtomicBool provides
immediate process-level protection; Redis monitoring is deferred to
multi-service deployment with a clear implementation roadmap.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 10:06:42 +01:00
jgrusewski
e36698ef14 fix(ml): GPU OOM detection with automatic CPU fallback in inference engine
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 10:06:40 +01:00
jgrusewski
3637b49218 fix(trading_service): wire real portfolio notional into VaR calculations
Move position fetching before VaR calculation in both get_va_r and
get_risk_metrics so the portfolio notional is computed from real
position data (sum of |quantity * avg_price|) instead of the fake
confidence_level * 1_000_000.0 placeholder. Falls back to 100_000.0
when the portfolio is empty.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 10:04:34 +01:00
jgrusewski
51378aa4cc fix(trading_agent): wire real position data into allocation calculations
Replaces hardcoded zeros in AssetAllocation with real position data
queried from agent_orders. Adds fetch_current_positions() helper that
derives net quantity per symbol (buy - sell) and reuses it in both
allocate_portfolio and rebalance_portfolio to eliminate SQL duplication.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 09:54:12 +01:00
jgrusewski
cc623af5fa fix(ml): replace std::process::exit with Result propagation in temporal_guard tests
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 09:46:05 +01:00
jgrusewski
7fd5ad4286 docs: production hardening phase 2 implementation plan — 19 tasks
4 layers: Safety Net (6 tasks), Correctness (6 tasks incl. CfC ensemble),
Verification (4 tasks), Training Infrastructure (3 tasks incl. Databento).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 09:39:13 +01:00
jgrusewski
fad71033ff docs: production hardening phase 2 design — $100K live trading readiness
4-layer risk-prioritized approach: safety net (crash prevention),
correctness (accurate calculations + liquid CfC ensemble integration),
verification (test coverage), training infrastructure (Databento + GPU).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 09:33:55 +01:00
jgrusewski
b88fd62af2 feat(ml): add Diffusion model (DDPM/DDIM) for price path generation
- NoiseScheduler: precomputed cosine/linear alpha_bar schedules
- Denoiser: FC network with sinusoidal time embedding + SiLU + residual
- DDIMSampler: deterministic fast sampling (10 steps from 1000 timesteps)
- DiffusionTrainableAdapter: UnifiedTrainable for unified training pipeline
- Hyperopt adapter with ParameterSpace (9 params, batch ≤64 for 4GB GPU)
- ModelType::Diffusion registered in common + coordinator
- 41 tests passing (config=3, noise=7, denoiser=4, sampler=5, trainable=12, hyperopt=7)
- OOM-safe: FC denoiser instead of U-Net, small hidden dims, conservative defaults

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 09:32:44 +01:00
jgrusewski
83054548b8 feat(ml): add xLSTM architecture (sLSTM + mLSTM blocks, network, trainable, hyperopt)
- sLSTM: exponential gating for long-range memory retention
- mLSTM: matrix memory with multi-head attention for higher capacity
- XLSTMBlock: pre-LayerNorm + residual connections
- XLSTMNetwork: stacked blocks with configurable sLSTM/mLSTM ratio
- UnifiedTrainable adapter for unified training pipeline
- Hyperopt adapter with ParameterSpace (9 params)
- ModelType::XLSTM registered in common + coordinator
- 45 tests passing (38 architecture + 7 hyperopt)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 09:13:35 +01:00
jgrusewski
a51fe0d30d Merge feat/production-hardening: resolve 53 TODOs across 5 phases
Phase 1: ML pipeline verification (checkpoint roundtrip tests, feature pipeline tests, DQN VarMap bug fix, deleted 585 lines dead code)
Phase 2: Service production logic (real portfolio metrics, VaR positions, proto population, safetensors loading, shutdown handling)
Phase 3: Backtesting & data (equity curve, DBN metadata, progress callbacks, cross-symbol validation, event filtering)
Phase 4: ML crate TODOs (statrs t-distribution, quantization savings, safetensors header, microstructure features, 45-action masking, confidence EMA, AttentionMask)
Phase 5: Infrastructure/cleanup (TLS/OCSP docs, compliance roadmap, metrics docs, regime features, execution roadmap, auth #[ignore], chaos docs)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 09:04:43 +01:00
jgrusewski
4eacd4e22f feat(ml): add KAN architecture + TLOB/KAN trainable/hyperopt adapters
Phase 2-3 of ensemble expansion:
- KAN (Kolmogorov-Arnold Network): B-spline basis, layer, network, trainable adapter
- TLOB UnifiedTrainable adapter with 3D input support (batch, seq, features)
- Hyperopt adapters for both KAN and TLOB (ParameterSpace + metrics)
- ModelType::KAN variant registered in common, coordinator, lib.rs
- 44 new tests, all passing, zero warnings

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 01:32:25 +01:00
jgrusewski
d777d714b2 feat(liquid): wire Liquid CfC into trading service and gRPC trainer
Closes the integration gap between Liquid CfC and DQN/PPO by adding:
- LiquidTrainer with gRPC progress callbacks, early stopping, and
  checkpoint management (ml/src/trainers/liquid.rs)
- LiquidModel wrapper in enhanced_ml.rs for hot-loading from safetensors
- Ensemble weight rebalance: DQN 0.25, PPO 0.25, TFT 0.20, Mamba2 0.15,
  Liquid-CfC 0.15

Verified: 81 liquid unit tests + 3 integration tests + 211 trading_service
tests pass, 0 compile errors across workspace.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 01:14:33 +01:00
jgrusewski
4db5b86d9b chore(tests): clean stale FIXME in test_runner, document lock contention tracking
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 01:14:24 +01:00
jgrusewski
3544e800f8 fix(services): regime feature extraction, correlation docs, execution roadmap, auth #[ignore], chaos docs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 01:14:14 +01:00
jgrusewski
2980e6c50a docs(services): TLS/OCSP delegation, compliance roadmap, metrics doc comments
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 01:14:05 +01:00
jgrusewski
0e3144813a feat(ml): add hyperopt adapter for TGGN (ParameterSpace + metrics)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 01:08:00 +01:00
jgrusewski
d4a0255595 feat(ml): AttentionMask causal masking, document multi-asset DQN and conversion layer
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 01:03:53 +01:00
jgrusewski
cfadd7d3dc feat(ml): factored 45-action masking, confidence EMA, RAII memory docs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 01:03:44 +01:00
jgrusewski
4e228f127f feat(ml): implement microstructure features, TFT resource monitoring, document disabled tests
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 01:03:35 +01:00
jgrusewski
9e7a4bf086 fix(ml): replace hardcoded t-distribution with statrs, compute quantization savings, parse safetensors header
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 01:03:26 +01:00
jgrusewski
de8b58ae06 Merge branch 'feat/tggn-fullstack' — TGGN UnifiedTrainable adapter
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 01:01:03 +01:00
jgrusewski
60593ba8bb fix(services): wire event filter, document job store, implement cross-symbol validation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 00:50:07 +01:00
jgrusewski
c45e4a039a feat(backtesting): DBN file metadata caching and progress callback for strategy engine
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 00:49:47 +01:00
jgrusewski
1896bb5a38 fix(backtesting): wire equity curve and drawdown periods to GetBacktestResults
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 00:49:17 +01:00
jgrusewski
b29817075e feat(ml): add UnifiedTrainable adapter for TGGN
Implements TGGNTrainableAdapter wrapping a candle-based projection
network (input→hidden→ReLU→output) with full UnifiedTrainable interface
including checkpoint save/load, validation, and gradient tracking.

11 tests covering forward, backward, optimizer step, checkpoint
roundtrip, validation, learning rate, and metrics collection.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 00:48:57 +01:00
jgrusewski
bb116bad89 fix(trading_service): safetensors DQN loading, prediction shutdown, emergency order query
- Add load_from_safetensors() to DQNAgent for weight loading via VarMap
- Update RealDQNModel::from_checkpoint to try safetensors first, fall back to JSON
- Replace std::mem::forget(prediction_shutdown_tx) with proper Vec-based storage
  that sends shutdown signal and drops senders during graceful shutdown
- Wire order_manager.get_open_orders() into emergency_stop response so callers
  see which orders were active when the kill switch engaged

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 00:26:13 +01:00
jgrusewski
b7c92d60f9 fix(trading_service): use real position sizes and marginal VaR contributions
Replace hardcoded position_size: 1000.0 with actual position quantities
from fetch_positions(). Compute contribution_pct as marginal VaR ratio
(symbol_var / portfolio_var * 100) instead of naive equal-weight split.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 00:20:25 +01:00
jgrusewski
164e96bf44 fix(trading_agent): wire real volatility, prices, and metrics into allocate_portfolio
Replace 7 hardcoded 0.0 values with real calculations:
- target_quantity from last close price
- portfolio_volatility from log return stddev * sqrt(252)
- portfolio_sharpe from weighted returns / portfolio vol
- var_95 parametric VaR
- max_drawdown_estimate from vol approximation
- rebalance_delta as target - current (0 until positions available)
- per-asset volatility from price bars (was hardcoded 0.15)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 00:20:05 +01:00
jgrusewski
952a09a89d Merge branch 'worktree-liquid-cfc-v2' 2026-02-23 00:18:55 +01:00
jgrusewski
33576dddb9 fix(services): populate event protos, compute max drawdown, document integration gaps
Tasks 8-14 production hardening batch:

- Populate Order/Position/Execution proto messages from JSON payload in event
  stream converters instead of returning None (Task 9)
- Compute max_drawdown from cumulative PnL samples in A/B testing pipeline
  instead of hardcoded 0.0 (Task 11)
- Document feature pipeline integration blockers with detailed roadmap
  comments in state.rs and trading.rs (Task 8)
- Document realized PnL gap: TradingPosition lacks the field, repository
  has async method incompatible with Iterator::map (Task 10)
- Document ML order quantity gap in api_gateway proxy: MlOrderResponse
  proto lacks quantity field (Task 12)
- Document per-symbol weight tracking roadmap in ensemble_coordinator (Task 13)
- Document OHLCV bar pipeline upgrade roadmap in state.rs (Task 14)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 00:18:39 +01:00
jgrusewski
5a90eaba8c Merge branch 'refactor/codebase-deduplication' 2026-02-23 00:11:34 +01:00
jgrusewski
888daa1ced feat(liquid): add CandleCfCTrainer, re-exports, integration tests, CUDA fix
- Add CandleCfCTrainer to training.rs with Candle-based gradient training
- Update mod.rs with full CfC v2 re-exports (CandleCfCNetwork, CfCCell, etc.)
- Fix CUDA variance bug (undefined variable) and kernel compilation stub
- Add 3 integration tests: full training loop, checkpoint roundtrip, validation
- 3 new unit tests for CandleCfCTrainer (creation, single epoch, loss decrease)

73 liquid tests pass, 0 errors, 0 clippy warnings in liquid module.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 00:08:56 +01:00
jgrusewski
62c2439fe5 test(ml): add feature extraction pipeline integration tests
4 tests validating the 51-dim feature extraction pipeline:
- DBN data loading (graceful skip if file absent)
- Synthetic bars: dimension check (51-dim), no NaN/Inf
- Value range bounds (-100 to 100)
- Streaming vs batch consistency (element-wise 1e-10 tolerance)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 00:08:17 +01:00
jgrusewski
20c09d8a54 refactor(ml): delete legacy SimpleNeuralNetwork and TrainingPipeline dead code
Remove 520 lines of ndarray-based placeholder code that never performed real
gradient descent. Production training uses Candle-based trainers in ml::trainers/.

Deleted: SimpleNeuralNetwork, TrainingPipeline, NetworkConfig, ActivationType,
TrainingMetrics, NetworkInterface, MockNetwork, and 9 associated tests.
Kept: DeviceCapabilities, TrainingConfig, sub-module re-exports.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 00:07:59 +01:00
jgrusewski
d9fa536511 docs: add ML ensemble expansion implementation plan (36 tasks, 6 phases)
Sequential build plan for 5 model integrations (TGGN, TLOB, KAN,
xLSTM, Diffusion) with TDD, worktree isolation, and full trait
implementations (UnifiedTrainable + ParameterSpace + hyperopt).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 00:05:28 +01:00
jgrusewski
200deaafe2 fix(ml): delete mock data loader from train_tft binary
Replace load_and_split_data() mock that generated 2000 synthetic samples
with an error-returning stub directing users to ml_training_service.

The binary retains its real infrastructure (TFTTrainer, CLI, checkpoint
storage, progress callbacks) — only the fake data generation is removed.

-120 lines of mock data, +16 lines error stub with documentation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 00:03:12 +01:00
jgrusewski
1408d33f4f docs: add ML ensemble expansion design (7→10 models, 5 integrations)
Approved design for expanding the ensemble with KAN, xLSTM, and
Diffusion architectures plus bringing TGGN/TLOB to full integration.
Sequential build order, no god classes, modular decomposition.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 23:56:38 +01:00
jgrusewski
4dd6e24a4e fix: resolve rebase conflict markers and align ModelType semantics
Fix leftover conflict markers from rebase onto main. Align as_str()
with main's semantics (general-purpose model names), add separate
s3_prefix() for S3 storage paths, and fix to_db_string() to preserve
per-variant database values.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 23:56:29 +01:00
jgrusewski
edd5a503da test(ml): add TFT + Mamba2 checkpoint roundtrip integration tests
Extend checkpoint_roundtrip.rs with 4 new tests for sequence-buffered models:

- TFT adapter deterministic inference: verifies same adapter produces
  identical direction/confidence on repeated calls with stable buffer
- TFT quantile metadata: confirms quantiles are absent during buffering
  phase and present (with correct count) after buffer fills
- Mamba2 adapter deterministic inference: same pattern as TFT, verifies
  direction/confidence stability and correct model name ("MAMBA-2")
- Mamba2 CheckpointManager roundtrip: saves/loads via Checkpointable
  trait on Mamba2SSM, verifies metadata tags and hyperparameters

All 10 tests (6 existing DQN/PPO + 4 new TFT/Mamba2) pass consistently.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 23:55:23 +01:00
jgrusewski
7c7f718272 feat(liquid): add LiquidTrainableAdapter implementing UnifiedTrainable
Full adapter bridging CandleCfCNetwork to the unified training pipeline
with VarMap-based checkpointing, AdamW optimizer, and gradient norm
tracking. Includes 10 unit tests covering creation, training steps,
validation, metrics, learning rate, checkpoint roundtrip, and error cases.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 23:55:04 +01:00
jgrusewski
834f097ffe feat(liquid): add LiquidInferenceAdapter for ensemble predictions
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 23:45:35 +01:00
jgrusewski
4ec8e58cf4 refactor(data): replace inline CircuitBreaker with common::resilience::CircuitBreaker
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 23:39:26 +01:00
jgrusewski
4a14e53eeb refactor(broker_gateway): replace inline CircuitBreaker with common::resilience::CircuitBreaker
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 23:39:26 +01:00
jgrusewski
bb4d4159d8 refactor(ml): implement CircuitBreakerTrait for ML circuit breaker
Bridge the ML crate's parking_lot::RwLock-based CircuitBreaker to the
shared CircuitBreakerTrait from common. Maps synchronous methods
(allow_request, record_success, etc.) to the async trait interface and
converts CircuitState variants to common::CircuitBreakerState.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 23:39:26 +01:00
jgrusewski
43f0fa90fc feat(common): add CircuitBreakerTrait for shared circuit breaker interface
Define an async trait that abstracts the circuit breaker state machine
(Closed -> Open -> HalfOpen) so different implementations can be used
polymorphically. Implement the trait for the existing tokio::Mutex-based
CircuitBreaker struct by delegating to its existing async methods.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 23:39:26 +01:00
jgrusewski
9693c7c590 refactor(ml): move Adam optimizer from lib.rs to optimizers/adam.rs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 23:39:26 +01:00
jgrusewski
1c940567ca refactor(adaptive-strategy): remove noisy hardcoded config eprintln warnings
Remove 8 warning prints (tracing::warn! and eprintln!) from Default impls
in config.rs. These fired on every test run and default construction,
creating noise. The migration notice in the module-level comment remains
as documentation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 23:39:26 +01:00
jgrusewski
64fa11b6d5 refactor(adaptive-strategy): remove dead code from execution, risk, and model modules
- execution/mod.rs: prefix 18 unused struct fields with _ (SlippageTracker,
  SmartOrderRouter, TWAP, VWAP, MarketImpactModel fields)
- risk/mod.rs: prefix 5 unused PnLTracker/RiskMetricsCalculator fields,
  remove #[allow(dead_code)] from DrawdownCalculator (fields are used),
  delete entirely unused CorrelationMatrix struct
- models/traditional.rs: prefix config/ready on 4 stub model structs
- models/deep_learning.rs: prefix config/ready on 3 stub model structs
- 80/80 tests pass, zero warnings

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 23:39:26 +01:00