Both route_order and cancel_order used let _ = to discard DB update
errors after successful broker operations. This means an order could be
live at the broker while the database still shows PENDING_SUBMIT or
CANCEL_PENDING, with no log entry to alert operators.
Replaced with if let Err(db_err) that logs at error level with CRITICAL
prefix, client_order_id, and broker_order_id for manual reconciliation.
The RPC still returns success since the broker operation completed.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The previous (quantity * 100_000.0) as i64 silently truncated
fractional lots and could overflow on extreme values. Added
convert_quantity_to_volume() that validates the result is finite,
non-negative, and within i64 range, using round() instead of
truncation. Includes 7 unit tests covering normal values, fractional
rounding, overflow, negative, NaN, infinity, and zero edge cases.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
from_checkpoint was creating fresh models with random weights and setting
is_trained=true, allowing trading on noise. Now validates checkpoint exists
and does NOT set is_trained=true when weights are random.
Mamba2 predict errors are now wrapped as MLError::InferenceError so the
fallback manager can degrade model health. Both TFT and Mamba2 predict
refuse to run on untrained models, and is_ready() reflects trained state.
Also fixes TFT input_dim mismatch (16 vs 5+10+16=31).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
try_write() is non-blocking and silently drops the CTraderClient when
the lock is contended. This means a successfully connected broker
client could be lost without any error. Changed to write().await which
guarantees the client is stored. Made function async and updated the
call site in main.rs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Low-confidence ensemble votes now forced to Hold (was letting weak
signals through to order generation in both vote paths)
- Inference latency tracking uses EMA (alpha=0.1) instead of
overwriting with the latest sample
- retrain_model returns Status::Unimplemented instead of faking
success with a random job_id
- get_model_performance returns honest zeroes instead of hardcoded
fake metrics (accuracy=0.85 etc.)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Use chrono-tz America/New_York for correct EST/EDT trading session
boundaries (was hardcoded UTC-5, off by 1h during daylight saving)
- Normalize effective weights to sum to 1.0 before signal aggregation
(raw weights could sum to anything, biasing the ensemble)
- Skip adapter predictions that return NaN/Inf direction or confidence
instead of letting them poison the weighted average
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
NaN initial features propagate through the PPO policy network and
produce garbage position sizing on the very first inference call.
Replace with 0.0 so the first prediction is safe (neutral).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Was only checking quantity limit and VaR. Now runs all 5 risk checks:
1. Kill switch / circuit breaker (via TradingServiceKillSwitch)
2. Max order size + position limits (via RiskRepository.get_risk_limits)
3. Daily loss / drawdown limit (via RiskRepository.get_risk_metrics)
4. Leverage limit (via config + RiskRepository.get_risk_metrics)
5. VaR limit (via RiskEngine.check_var_limit, existing)
Violations accumulate rather than short-circuit so callers see all
failures at once. Added 6 tests verifying violation type coverage.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
VaR, drawdown, and leverage checks in check_risk_limits() only logged
warnings and returned Ok(()). Now they return errors to block order
generation when limits are breached. Also removes unused warn import.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Linear extrapolation above 0.99 gave z=2.48 for 99.9% (correct: 3.09).
VaR was understated by ~25% at high confidence levels.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Unlimited concurrent probes could overwhelm recovering services.
Added probe_in_flight guard to ensure only one request probes the
recovering service at a time in HalfOpen state.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Per-account circuit breaker and daily loss tracking was never applied to
real accounts because check_order() hardcoded "default". Added account_id
field to OrderInfo with backward-compatible None fallback. Updated all
construction sites across risk crate tests and integration tests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Position and leverage checks returned Approved by default when
broker_account_service was None. Now returns ServiceUnavailable
error to prevent unvalidated orders from passing through.
Added 3 tests verifying the fail-safe behavior for both
check_position_limits and check_leverage_limits, plus the
public check_order integration path.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Duplicate execution reports could double fill_quantity and corrupt
average price. Added guards in process_execution to reject fills on
already-Filled orders and to reject fills that would exceed order qty.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fix zstd decoder in train_baseline.rs and evaluate_baseline.rs (same
pattern as hyperopt adapters — branch on .dbn.zst extension). Add CLI
flags for walk-forward config (train/val/test/step months), learning
rate, and max-steps-per-epoch to make pipeline validation feasible.
Pipeline validated end-to-end: hyperopt (5 trials, best Sharpe 2.37) →
walk-forward training (4 folds, 6/1/1 month windows on ES.FUT) →
evaluation (4 fold test sets, checkpoints + norm stats saved).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PPO hyperopt: Split DATA 80/20 for train/val (not trajectory count).
Previously indexed training_data[..num_trajectories] which was only
16 samples from 56K — causing num_batches=0 and NaN from 0/0 division.
Now properly splits data array and uses min(64, num_train) for batch
episodes with ceil division for batch count.
DQN: Fix hardcoded "45 actions" in diagnostic log (now uses actual
q_vec.len()). Replace indexed[..top_n] slice with .take(top_n)
iterator to prevent potential slice panic.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
validate_order and add_order were separate operations. The gap between
them allowed duplicate order IDs to both pass validation because
validate_order only took a read lock. New validate_and_add_order method
holds a single write lock for the entire check-and-insert operation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The update_position method previously acquired a read lock to get the
Arc<AtomicPosition>, dropped it, then called update_with_execution
outside any lock. Two concurrent fills for the same position could both
load the same old_quantity via Acquire, compute independent new quantities,
and the last Release store would silently discard the other fill — causing
the position to show e.g. 10 shares when it should show 20.
Now holds the positions write lock across the entire get-or-create +
update_with_execution sequence, serializing concurrent fills per position.
The validate_position_update call remains outside the lock to minimize
hold time (it does its own async reads and risk checks).
Also removes excessive per-step RDTSC latency tracking from the critical
path (total latency tracking is preserved) and adds a multi-threaded
regression test that spawns 10 concurrent +1 fills and asserts the final
quantity equals 10.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Databento data pipeline stored files as .dbn.zst in symbol subdirectories
(e.g., ES.FUT/ES.FUT_2024-Q1.dbn.zst), but the hyperopt adapters and DQN
data loader only matched .dbn extension and used flat directory listing.
Three fixes:
- Match both .dbn and .dbn.zst file extensions in collect_dbn_files_recursive
- Use recursive directory traversal instead of flat read_dir
- Branch on extension to use Decoder::from_zstd_file() for .dbn.zst files
(from_file() doesn't handle zstd decompression)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The old code computed `_execution_value = quantity * price` but
discarded it (underscore prefix). Only commission was subtracted,
so the account cash balance never reflected the cost of buying or
the proceeds from selling.
Now `update_from_execution` matches on the new `execution.side` field:
- Buy → cash -= execution_value + commission
- Sell → cash += execution_value − commission
Test assertions updated to expect the corrected balances.
Addresses audit item H2 (account never deducting trade value).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ExecutionResult.executed_quantity is always a positive magnitude, so the
old `is_buy = executed_quantity > ZERO` check was always true — every
fill was treated as a buy regardless of order side.
Add an explicit `pub side: OrderSide` field to ExecutionResult and use
`execution.side == OrderSide::Buy` in PositionManager. All construction
sites (source, tests, benchmarks) updated; sell-side tests now use
positive quantities with `side: OrderSide::Sell` instead of the former
negative-quantity hack.
Addresses audit item H4 (position direction always buy).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
clone_for_async() was creating independent AtomicU64 instances for
message_count, drop_count, last_heartbeat, and reconnect_attempts
instead of sharing the originals. This meant spawned tasks incremented
their own counters while get_stats() read the original's (always 0),
and the heartbeat monitor watched a counter never updated by the
connection task.
Also fixes last_heartbeat initializing to 0, which caused the first
heartbeat check to compute a huge elapsed time and immediately trigger
a false "connection appears dead" alert.
Changes:
- Change 4 struct fields from AtomicU64 to Arc<AtomicU64>
- Initialize last_heartbeat to HardwareTimestamp::now() instead of 0
- clone_for_async() now clones the Arcs (shared counters)
- Add test verifying counters are shared between original and clone
- Add test verifying heartbeat initialized to current time
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
4 symbols (ES, NQ, ZN, 6E) x 9 quarters (2024-Q1 through 2026-Q1),
723 days of 1-minute OHLCV bars from GLBX.MDP3 dataset.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The download binary was using the default SType::RawSymbol which
returned empty DBN files (95 bytes) for parent symbols like ES.FUT.
Adding SType::Parent resolves the symbol mapping. Also adjusted end
date to 2026-02-22 (latest available data).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Casting negative f64 to u64 saturates to 0, corrupting avg_price and all
downstream PnL calculations. Add price_to_fixed_checked() that rejects
non-finite, zero, and negative prices with PositionError::InvalidPrice.
The critical update_with_execution path now uses the checked variant.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The get_orders() status filter used `matches!(order.status, _status)` which
creates a new wildcard binding instead of comparing against the captured
`status` variable. This caused every order to match regardless of filter,
meaning get_orders(Some(Filled)) returned ALL orders — inflating exposure
calculations. Replaced with direct equality comparison `order.status == *status`.
Added test_get_orders_status_filter_only_returns_matching to prevent regression.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add #![allow(unused_crate_dependencies)] for consistency with other ml tests
- Extract EXPECTED_FEATURE_DIM constant (replaces magic number 51)
- Add validated_folds counter to prevent vacuous pass in walk-forward test
- Replace unwrap_or_else(panic) with match pattern
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three integration tests validating the full pipeline end-to-end:
1. Feature extraction: 90-day synthetic bars -> 51-dim features, no NaN/Inf
2. Walk-forward with normalization: 24-month bars -> windows -> NormStats from train only -> normalized mean ~0
3. No lookahead bias: train timestamps < val start, val timestamps < test start
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Loads trained DQN/PPO checkpoints, runs greedy inference on walk-forward
test windows, computes Sharpe ratio, max drawdown, win rate, profit factor,
and total return per fold. Outputs a JSON evaluation report with aggregate
metrics and sanity checks (beats-random, action diversity, fold consistency).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add ml/examples/train_baseline.rs that trains DQN and PPO models using
expanding walk-forward windows on real Databento OHLCV data.
Features:
- CLI args via clap (--model, --epochs, --batch-size, --data-dir, etc.)
- Recursive .dbn.zst file discovery and OHLCV bar loading
- 51-dim feature extraction via extract_ml_features()
- Walk-forward window generation with NormStats per fold
- DQN training loop with epsilon-greedy, experience replay, early stopping
- PPO training loop with GAE, trajectory collection, early stopping
- PnL-based reward (BUY/SELL/HOLD)
- Safetensors checkpoint saving per fold
- NormStats JSON export for evaluation reproducibility
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements expanding-window walk-forward cross-validation for time-series
ML models, preventing lookahead bias by always evaluating on unseen future
data. Includes z-score NormStats computed from training splits only.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds `ml/examples/download_baseline.rs` that downloads 730 days of
Databento OHLCV-1m data in quarterly chunks for 4 CME futures symbols.
Features:
- Reads universe config from TOML (symbols, date range, dataset)
- Splits date range into calendar-quarter chunks (~90 days each)
- Resume support: skips existing non-empty files
- Uses `get_range_to_file` for streaming writes to .dbn.zst
- Dry-run mode with cost estimate ($0.12/symbol/day)
- Confirmation prompt (skippable with --yes)
- Per-file progress with timing and byte counts
- Failure-tolerant: logs errors and continues
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Ubuntu 22.04's protobuf-compiler (v3.12) lacks proto3 optional field
support. Install protoc v25.1 from GitHub releases instead, which
matches the protoc version available in Debian bookworm-based images
used by the other 5 services.
Validated: docker build completes successfully, binary starts with
ml_training_service --help.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All 6 service Dockerfiles were missing newly-added workspace crates
(web-gateway, ctrader-openapi, foxhunt-deploy, broker_gateway_service),
causing cargo workspace resolution failures during Docker builds.
Changes across all Dockerfiles:
- Add COPY directives for web-gateway, ctrader-openapi, foxhunt-deploy,
broker_gateway_service (new workspace members since Dockerfiles written)
- Add SQLX_OFFLINE=true env and .sqlx cache copy where missing
- Add perl and make system deps (needed for OpenSSL build from source)
- Remove COPY migrations (dir excluded by .dockerignore, not needed)
- Expand broker_gateway_service from 3-crate to full workspace copy
Validated: docker build --check passes all 6, cargo check -p passes all 6.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Essential workflow fixes:
- ci.yml: add SQLX_OFFLINE=true, replace fictitious cargo subcommands
(test-unit, ci-lint, audit-deps, etc.) with real cargo commands,
remove broken 9-way test matrix, remove instrument-coverage RUSTFLAGS
- test.yml: add SQLX_OFFLINE=true to build-check job, fix conflicting
clippy flags (-D and -W on clippy::all), update JWT secret to 32+ chars
- compilation-guard.yml: add SQLX_OFFLINE=true, remove references to
non-existent paths (services/trading-engine, crates/common/types),
remove dangerous auto-commit-to-main, remove MIRI on missing packages
Deleted duplicates (justification):
- comprehensive_testing.yml: duplicate of comprehensive-testing.yml
(same purpose, underscore vs hyphen naming)
- production-deploy.yml: duplicate of production-deployment.yml
(both named "Production Deployment Pipeline", this one has stale paths)
- coverage-fixed.yml: duplicate of coverage.yml
(uses deprecated actions-rs/toolchain@v1 and actions/cache@v3)
- performance.yml: duplicate of benchmark_regression.yml
(both "Performance Regression Detection", this one less mature)
- quality-baseline.json: not a workflow, stale fake data (999 warnings)
- quality-metrics.json: not a workflow, stale fake data
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Proves the critical trading pipeline path works end-to-end:
OHLCV -> features -> ensemble -> prediction. Uses all 10 real candle
inference adapters (DQN, PPO, TFT, Mamba2, Liquid-CfC, TGGN, TLOB,
KAN, xLSTM, Diffusion) with random weights, 60 synthetic OHLCV bars,
and asserts confidence/action/model-count invariants. No external
dependencies (no DB, no Docker).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Spawn a background tokio task that feeds synthetic OHLCV bars into the
ensemble coordinator's per-symbol feature extractors. The extractors
require ~51 bars of warmup before they can produce real 51-dim feature
vectors; without this feed they remain cold and return zeros.
The task uses a random-walk price generator with realistic base prices
for the four baseline futures symbols (ES, NQ, ZN, 6E) and configurable
interval (MARKET_FEED_INTERVAL_MS, default 1s) and symbol list
(MARKET_FEED_SYMBOLS). In production this will be replaced by a
Databento or exchange data feed.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Register TGGN, TLOB, KAN, xLSTM, and Diffusion inference adapters
alongside the existing DQN, PPO, TFT, Mamba2, and Liquid-CfC.
Rebalance weights to 0.10 each (equal weighting across 10 models).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sequence-buffered adapter with 3-layer MLP projection
(flat_dim -> hidden -> hidden/2 -> 1). Ring buffer collects
seq_len feature vectors, returns neutral prediction until full,
then flattens and projects through the MLP with sigmoid output
mapping to direction [-1,1] and confidence [0,1].
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove panic!() calls from test_futures_baseline_micro_mapping,
use map()+Some() pattern consistent with rest of test suite
- Make DatasetSpec::from_universe() accept a name parameter instead
of hardcoding "futures-baseline"
- Add doc comment to UniverseConfigMeta explaining dead_code fields
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
