Commit Graph

4015 Commits

Author SHA1 Message Date
jgrusewski
ff15341c3e Merge branch 'worktree-ml-production-hardening' 2026-02-24 09:56:03 +01:00
jgrusewski
57bae2cb68 fix(ml): OOM hardening + battle-test KAN/xLSTM/Diffusion models
Replace 8 unbounded Vec accumulation patterns with bounded VecDeque
across ensemble, PPO, DQN, Mamba2, and data pipeline code to prevent
OOM on RTX 3050 Ti (4GB VRAM) during live trading and extended training.

Key OOM fixes:
- Ensemble price/volatility history: Vec → VecDeque with O(1) eviction
- Data pipeline: MAX_FEATURES=500K cap (~512MB) prevents unbounded loading
- DQN replay buffer: full-array shuffle → HashSet random sampling (8MB → 256B)
- PPO loss histories: bounded VecDeque (cap 1K), eliminated batch.clone()
- Mamba2 scan: pre-allocated Vecs, explicit drop() after Tensor::cat
- Mamba2 training history: capped at 100, Tensor::randn replaces Vec→Tensor
- Mamba2 SSM reset: 2 unwrap() violations replaced with proper error handling

Battle-testing (19 new integration tests):
- KAN: 5 tests (forward, 50-epoch training 89.9% loss reduction, checkpoint)
- xLSTM: 7 tests (2D+3D forward, 30-epoch training 82% reduction, checkpoint)
- Diffusion: 7 tests (2D+3D forward, 20-epoch pipeline, checkpoint, validation)

Bonus: fix pre-existing cache test failure (match .dbn.zst files, graceful skip)

All 2390 lib tests pass, 0 new clippy errors.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 09:55:39 +01:00
jgrusewski
f2d9fb9328 docs: update stub audit results with actual fix status
All 18 findings now have resolution status:
- 8 fixed with real implementations
- 3 fixed with honest errors (instead of fake data)
- 4 deferred (need infrastructure)
- 3 acceptable as-is

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 09:46:17 +01:00
jgrusewski
89692ac4c1 fix: replace stub code with real logic and honest errors
Coordinator (ml/src/integration/coordinator.rs):
- Delete 9 fake heuristic methods (500+ lines) that pretended to be
  real DQN/TFT/TGGN/LNN/Mamba predictions using sin()/tanh() math
- Make generate_model_specific_prediction() return Err instead of
  fake predictions — prevents trading on fabricated signals
- Make ensemble fault-tolerant: skip failed models instead of
  failing entire ensemble (execute_parallel/execute_sequential)
- Remove double-fallback in execute_single_model error path

Enhanced ML (trading_service):
- get_model_performance(): compute accuracy from real
  inference_count/error_count instead of returning all zeros
- get_feature_importance(): return Status::unavailable instead of
  hardcoded fake values — honest about missing SHAP implementation

Autonomous scaling (trading_agent_service):
- diversification_score: compute real HHI from instrument volume
  distribution instead of hardcoded 0.8
- ml_confidence: keep liquidity heuristic but remove warn!() spam

Position limiter (risk):
- Remove redundant portfolio_id field from CachedPosition — the
  DashMap key already provides account-based isolation
- Remove misleading "stub" comment — was not a stub

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 09:21:21 +01:00
jgrusewski
8e20e509df chore: track pre-commit hook with stub detection patterns
Backs up the .git/hooks/pre-commit hook to a tracked file.
Includes stub detection (hardcoded returns, marker strings).
Cargo check removed — agents validate before commit.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 02:15:13 +01:00
jgrusewski
372ce028bc Merge branch 'fix/stub-audit-fixes' 2026-02-24 02:14:46 +01:00
jgrusewski
548737a936 fix: resolve stub audit findings — VPIN, correlation, dead code, warnings
- VPIN Calculator: implement real tick-rule classification (was entirely stubbed)
- Correlation matrix: replace hardcoded 0.5 with Pearson from log-returns
- Stress test: per-factor accumulation instead of single-max shortcut
- EnsembleModel: delete dead code, redirect to MockModel with warning
- Coordinator fallbacks: relabel fake "REAL" predictions as FALLBACK SIMULATION
- Enhanced ML: add warn!() to 5 stub endpoints, fix retrain status code
- Autonomous scaling: add warn!() to mock ml_confidence and diversification
- Position limiter: add warn!() for unused portfolio_id

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 02:14:30 +01:00
jgrusewski
9dd967a016 Merge cleanup: warnings, legacy vendor, dependency trimming
- fix(ml): resolve all 12 warnings (unused imports, Debug impls, lifetimes)
- chore: remove broken vendor/candle-optimisers gitlink
- chore: replace tokio features=["full"] with workspace defaults in 5 crates
- chore: remove unused plotters dep from services/load_tests
- feat(api_gateway): feature-gate MFA deps behind optional 'mfa' feature

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 02:08:34 +01:00
jgrusewski
759795854b feat(api_gateway): feature-gate MFA deps behind optional 'mfa' feature
MFA deps (qrcode, image, totp-rs, base32, hmac, urlencoding) now behind
optional 'mfa' feature, enabled by default. sha1 and secrecy remain
non-optional (used by mtls/revocation.rs and jwt/service.rs respectively).
Builds with --no-default-features --features minimal skip ~50 transitive
deps (rav1e, ravif, image codecs, etc.).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 02:01:15 +01:00
jgrusewski
44bef4bce4 fix(ml): resolve all 12 warnings in diffusion, liquid, ensemble modules
- Remove unused imports: DType (noise, sampler), Device (candle_cfc), TimeZone (coordinator)
- Add Debug impls for diffusion structs (manual for candle types, derive for DDIMSampler)
- Fix hidden lifetime params: VarBuilder → VarBuilder<'_> in denoiser.rs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 01:54:09 +01:00
jgrusewski
fb53efd9e2 chore: replace tokio features=["full"] with workspace defaults in 5 crates
Workspace tokio already specifies the needed features (rt-multi-thread,
macros, net, sync, time, fs, signal, io-util, test-util). Three crates
(test_common, test_harness, vault_integration) were skipped because
they are not workspace members and cannot use workspace = true.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 01:53:29 +01:00
jgrusewski
b4f7d4d60e chore: remove unused plotters dep from services/load_tests
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 01:51:13 +01:00
jgrusewski
5be297501b docs: add stub audit results — 18 findings across 37 crates
8 critical stubs found in risk-data, adaptive-strategy, trading_service,
and ml/integration. 3 crate groups clean (infra, frontend, CLI).
Priority: VPIN calculator, portfolio positions, enhanced_ml endpoints.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 01:48:12 +01:00
jgrusewski
5f7a913a7b docs: add stub detection implementation plan
9-task plan: 6 parallel audit agents, aggregate report,
pre-commit lint rules, and TLI→CLI rename.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 01:42:48 +01:00
jgrusewski
decf9ef957 docs: add stub detection & prevention design
Approved design for full-codebase audit using 6 parallel agents
scanning 14 stub patterns, plus lint prevention in pre-commit hook.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 01:39:42 +01:00
jgrusewski
1d9e7fe64c Merge branch 'fix/review-remaining-items' 2026-02-24 01:32:26 +01:00
jgrusewski
72addbbbc4 docs: add cleanup implementation plan
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 01:29:24 +01:00
jgrusewski
199287f78e fix(ml): warmup assertions, hyperopt architecture alignment, greedy eval
- Add debug_assert_eq! guards in 4 train_baseline functions to catch
  bar/feature length misalignment at debug time (#4)
- Remove "last sample targets itself" block in hyperopt PPO adapter
  that created ~0 return sample biasing toward HOLD (#5)
- Align hyperopt state_dim 54→51 and num_actions 45→3 to match
  train_baseline architecture, making tuned hyperparams transferable (#6)
- Use greedy_action() in evaluate_baseline PPO eval for deterministic results

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 01:28:50 +01:00
jgrusewski
b1a12ce7d9 docs: add cleanup design plan for warnings, deps, and legacy removal
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 01:26:23 +01:00
jgrusewski
1330807b54 chore: untrack .env.runpod credentials and local state files
- Remove !.env.runpod exclusion from .gitignore (file says "DO NOT COMMIT")
- Untrack .claude/ralph-loop.local.md (session-local state)
- .env.runpod.template remains tracked as the safe reference

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 01:09:22 +01:00
jgrusewski
1eb1b2a032 chore: gitignore .serena/ and untrack local config files
.claude/settings.local.json was already in .gitignore but still tracked.
.serena/project.yml is IDE-specific config that shouldn't be versioned.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 01:07:33 +01:00
jgrusewski
a415c06e72 refactor(ml): extract shared DBN utilities into baseline_common module
Shared DBN loading, timestamp dedup, and spread cost helpers used by
train_baseline, evaluate_baseline, and hyperopt_baseline extracted to
ml/examples/baseline_common/mod.rs to eliminate ~190 lines of duplication.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 01:05:38 +01:00
jgrusewski
aaaaef7f48 fix(ml): address code review — greedy PPO eval, gamma alignment, Sharpe fix
Fixes from code review of DQN/PPO validation pipeline:

1. PPO validation: use greedy_action() (argmax) instead of stochastic
   act() — deterministic early-stopping signal, matching DQN's eps=0.

2. DQN eval gamma: align to 0.95 (was 0.99) matching train_baseline.
   Gamma doesn't affect greedy inference but configs should match.

3. Sharpe annualization: use 1380 bars/day (23h futures session) not
   390 (6.5h equities). Fixes ~1.8x underestimate for ES futures.

4. compute_reward: accept f64 total_cost_bps (was f32) to match
   shared spread_cost_bps() from baseline_common.rs.

5. Default max_steps_per_epoch: 2000 (was 0/unlimited) for OOM safety.

6. Hyperopt PPO: add timestamp dedup to decode_ohlcv_bars for .FUT
   parent symbols with overlapping contracts.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 01:02:51 +01:00
jgrusewski
24e72b370e fix(ml/ppo): use real policy actions and cap trajectory length to prevent OOM
Three critical PPO fixes:

1. Add PPO::act_with_log_prob() returning (action, log_prob, value).
   The existing act() discarded the policy log-probability, making
   PPO importance sampling use wrong ratios during training.

2. Cap trajectory length with --max-steps-per-epoch in train_baseline
   PPO path. DQN already had this limit; PPO iterated all features
   (~500K per fold), causing OOM on 4GB GPU.

3. Replace random actions and fake log_prob/value in hyperopt PPO
   adapter with real agent.act_with_log_prob() calls. Trajectories
   now reflect actual policy behavior for meaningful hyperopt.

Also adds warmup offset alignment to PPO trajectory collection
(matching the DQN fix) and fixes .unwrap() in test.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 00:52:02 +01:00
jgrusewski
26b51a4f99 feat(ml): real validation, transaction costs, and data fixes for DQN/PPO pipeline
Replace stub validation functions with real model inference (DQN greedy,
PPO act()) so early stopping optimizes actual trading performance instead
of market volatility. Add transaction costs (commission + bid-ask spread)
to reward computation across train/hyperopt/evaluate examples.

Key changes:
- Symbol filtering (--symbol ES.FUT) prevents mixing futures contracts
- BTreeMap timestamp dedup handles overlapping .FUT contract bars
- Return clamping (--max-bar-return) filters contract roll boundaries
- Warmup offset alignment fixes feature-to-bar index mismatch
- Kelly sizing: 3 stubs replaced with real data-driven implementations
- Adam optimizer: BUG #14 diagnostic logging demoted to trace
- TFT: varmap_mut() accessor for checkpoint loading
- PPO hyperopt: with_costs() builder for tx cost configuration

DQN eval (ES.FUT, 2 folds): Sharpe=11.36, MaxDD=7.42%, WinRate=33.2%

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 00:43:11 +01:00
jgrusewski
c3b5e124f0 chore: update .gitignore and add design plan docs
Ignore ML checkpoints, trained model safetensors, stray ml/ml/ dir,
and .claude/worktrees/. Clean up duplicate hive-mind-prompt entries.
Add 17 design/implementation plan docs from 2026-02-20 to 2026-02-22.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 00:30:15 +01:00
jgrusewski
55df034880 fix(trading_service): resolve merge conflict — use VarMap::load for TFT/Mamba2 checkpoints
Stashed changes had real safetensors weight loading via VarMap::load.
Audit branch was conservative (validate-only). Kept the real loading
and simplified Mamba2 to use the same VarMap::load pattern as TFT.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 00:04:41 +01:00
jgrusewski
0ade1c1b19 Merge branch 'feat/production-safety-audit' 2026-02-23 23:56:26 +01:00
jgrusewski
9495cd39ce fix(web-gateway): use TCP peer address for rate limiting instead of X-Forwarded-For
The rate limiter trusted the client-controlled X-Forwarded-For header
to identify clients. An attacker could rotate this header value on every
request to bypass rate limits entirely.

Now uses ConnectInfo<SocketAddr> (the actual TCP connection address) as
the rate limiting key. Server is configured with
into_make_service_with_connect_info to populate this.

X-Forwarded-For and X-Real-Ip headers are no longer consulted.
Falls back to "unknown" if ConnectInfo is unavailable (tests).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 23:25:05 +01:00
jgrusewski
434f9fbf7a fix(adaptive-strategy): replace fake Kelly return history with error + fallback
get_historical_returns was returning a hardcoded 20-value vector of fake
returns for ALL symbols, causing Kelly criterion to compute position sizes
based on fabricated data. Now returns an error explaining that no market
data feed is connected.

calculate_position_size catches the Kelly/PPO sizing errors and falls
back to standard fixed-fraction sizing with a warning log, instead of
propagating the error to callers.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 23:22:50 +01:00
jgrusewski
477dd47d3f fix(risk): warn when VaR uses static config volatility instead of market data
The VaR engine's get_symbol_volatility used hardcoded per-asset-class
annual volatility values (e.g. 25% for equities, 80% for crypto) without
any indication to operators that real market data was not being used.

Now emits a tracing::warn on every static volatility lookup so operators
see it in logs. Also adds a volatility_overrides HashMap<String, f64>
field on VarEngine for manual per-symbol overrides until a real market
data feed is integrated.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 23:20:26 +01:00
jgrusewski
8641499a6b fix(broker_gateway): gate reconnect Active state on health check
reconnect() previously slept 500ms then unconditionally set
SessionState::Active, faking a successful reconnection regardless of
whether any broker or infrastructure was actually reachable.

Now performs a DB health check (SELECT 1) after the sleep. If the
check fails, state is set to Disconnected and an error is returned.
Active is only set when the health check passes. The DB ping is the
best available liveness probe until a real FIX/cTrader session is
wired into SessionRecovery.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 23:18:06 +01:00
jgrusewski
960e8f9dcc fix(broker_gateway): require confirmation for cTrader live trading mode
A single CTRADER_LIVE=true env var was enough to route real money orders
through the broker. Now requires CTRADER_LIVE_CONFIRMED=I_UNDERSTAND_REAL_MONEY
alongside it, preventing accidental live trading from copy-pasted configs.

Also adds prominent warning log when live mode activates and info log
for demo mode.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 23:17:43 +01:00
jgrusewski
f1b5f84c6c fix(broker_gateway): log CRITICAL on DB update failure after broker submission
Both route_order and cancel_order used let _ = to discard DB update
errors after successful broker operations. This means an order could be
live at the broker while the database still shows PENDING_SUBMIT or
CANCEL_PENDING, with no log entry to alert operators.

Replaced with if let Err(db_err) that logs at error level with CRITICAL
prefix, client_order_id, and broker_order_id for manual reconciliation.
The RPC still returns success since the broker operation completed.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 23:16:33 +01:00
jgrusewski
a45aec5605 fix(broker_gateway): safe cTrader volume conversion with overflow checks
The previous (quantity * 100_000.0) as i64 silently truncated
fractional lots and could overflow on extreme values. Added
convert_quantity_to_volume() that validates the result is finite,
non-negative, and within i64 range, using round() instead of
truncation. Includes 7 unit tests covering normal values, fractional
rounding, overflow, negative, NaN, infinity, and zero edge cases.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 23:14:37 +01:00
jgrusewski
c71c32fff5 fix(trading_service): TFT/Mamba2 validate checkpoint exists and propagate inference errors
from_checkpoint was creating fresh models with random weights and setting
is_trained=true, allowing trading on noise. Now validates checkpoint exists
and does NOT set is_trained=true when weights are random.

Mamba2 predict errors are now wrapped as MLError::InferenceError so the
fallback manager can degrade model health. Both TFT and Mamba2 predict
refuse to run on untrained models, and is_ready() reflects trained state.

Also fixes TFT input_dim mismatch (16 vs 5+10+16=31).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 23:14:16 +01:00
jgrusewski
4e5a269353 fix(broker_gateway): use blocking write for set_broker_client
try_write() is non-blocking and silently drops the CTraderClient when
the lock is contended. This means a successfully connected broker
client could be lost without any error. Changed to write().await which
guarantees the client is stored. Made function async and updated the
call site in main.rs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 23:08:43 +01:00
jgrusewski
2f6b76b8fe fix(trading_service): enforce confidence threshold, EMA latency, honest retrain stub
- Low-confidence ensemble votes now forced to Hold (was letting weak
  signals through to order generation in both vote paths)
- Inference latency tracking uses EMA (alpha=0.1) instead of
  overwriting with the latest sample
- retrain_model returns Status::Unimplemented instead of faking
  success with a random job_id
- get_model_performance returns honest zeroes instead of hardcoded
  fake metrics (accuracy=0.85 etc.)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 23:03:39 +01:00
jgrusewski
5deb618864 fix(ml): DST-aware sessions, normalize ensemble weights, filter non-finite predictions
- Use chrono-tz America/New_York for correct EST/EDT trading session
  boundaries (was hardcoded UTC-5, off by 1h during daylight saving)
- Normalize effective weights to sum to 1.0 before signal aggregation
  (raw weights could sum to anything, biasing the ensemble)
- Skip adapter predictions that return NaN/Inf direction or confidence
  instead of letting them poison the weighted average

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 23:03:20 +01:00
jgrusewski
17e60a48de fix(adaptive-strategy): init MarketStateTracker with 0.0 instead of NaN
NaN initial features propagate through the PPO policy network and
produce garbage position sizing on the very first inference call.
Replace with 0.0 so the first prediction is safe (neutral).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 23:02:11 +01:00
jgrusewski
1da0d3bc31 fix(trading_service): validate_order gRPC uses full pre-trade risk checks
Was only checking quantity limit and VaR. Now runs all 5 risk checks:
1. Kill switch / circuit breaker (via TradingServiceKillSwitch)
2. Max order size + position limits (via RiskRepository.get_risk_limits)
3. Daily loss / drawdown limit (via RiskRepository.get_risk_metrics)
4. Leverage limit (via config + RiskRepository.get_risk_metrics)
5. VaR limit (via RiskEngine.check_var_limit, existing)

Violations accumulate rather than short-circuit so callers see all
failures at once. Added 6 tests verifying violation type coverage.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 22:25:41 +01:00
jgrusewski
774bbe6506 fix(adaptive-strategy): risk limit breaches return errors instead of warnings
VaR, drawdown, and leverage checks in check_risk_limits() only logged
warnings and returned Ok(()). Now they return errors to block order
generation when limits are breached. Also removes unused warn import.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 22:25:22 +01:00
jgrusewski
fc95dca6ed fix(risk): replace linear z-score interpolation with Abramowitz-Stegun
Linear extrapolation above 0.99 gave z=2.48 for 99.9% (correct: 3.09).
VaR was understated by ~25% at high confidence levels.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 22:17:51 +01:00
jgrusewski
77f7b1e0dc fix(common): limit HalfOpen circuit breaker to single probe
Unlimited concurrent probes could overwhelm recovering services.
Added probe_in_flight guard to ensure only one request probes the
recovering service at a time in HalfOpen state.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 22:16:17 +01:00
jgrusewski
bfe1b2ce2c fix(risk): add account_id field to OrderInfo for per-account risk tracking
Per-account circuit breaker and daily loss tracking was never applied to
real accounts because check_order() hardcoded "default". Added account_id
field to OrderInfo with backward-compatible None fallback. Updated all
construction sites across risk crate tests and integration tests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 22:07:50 +01:00
jgrusewski
5fe0cf953c fix(risk): fail-safe when broker service not configured
Position and leverage checks returned Approved by default when
broker_account_service was None. Now returns ServiceUnavailable
error to prevent unvalidated orders from passing through.

Added 3 tests verifying the fail-safe behavior for both
check_position_limits and check_leverage_limits, plus the
public check_order integration path.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 22:06:11 +01:00
jgrusewski
041e6f3d9a fix(trading_engine): reject overfills and fills on completed orders
Duplicate execution reports could double fill_quantity and corrupt
average price. Added guards in process_execution to reject fills on
already-Filled orders and to reject fills that would exceed order qty.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 21:52:34 +01:00
jgrusewski
e3f32742fa feat(ml): walk-forward training pipeline with real Databento data
Fix zstd decoder in train_baseline.rs and evaluate_baseline.rs (same
pattern as hyperopt adapters — branch on .dbn.zst extension). Add CLI
flags for walk-forward config (train/val/test/step months), learning
rate, and max-steps-per-epoch to make pipeline validation feasible.

Pipeline validated end-to-end: hyperopt (5 trials, best Sharpe 2.37) →
walk-forward training (4 folds, 6/1/1 month windows on ES.FUT) →
evaluation (4 fold test sets, checkpoints + norm stats saved).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 21:48:10 +01:00
jgrusewski
d65ea067f0 fix(ml): fix PPO hyperopt data split and DQN diagnostic panic
PPO hyperopt: Split DATA 80/20 for train/val (not trajectory count).
Previously indexed training_data[..num_trajectories] which was only
16 samples from 56K — causing num_batches=0 and NaN from 0/0 division.
Now properly splits data array and uses min(64, num_train) for batch
episodes with ceil division for batch count.

DQN: Fix hardcoded "45 actions" in diagnostic log (now uses actual
q_vec.len()). Replace indexed[..top_n] slice with .take(top_n)
iterator to prevent potential slice panic.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 21:37:20 +01:00
jgrusewski
5f47df464b fix(trading_engine): atomic validate-and-add prevents duplicate order race
validate_order and add_order were separate operations. The gap between
them allowed duplicate order IDs to both pass validation because
validate_order only took a read lock. New validate_and_add_order method
holds a single write lock for the entire check-and-insert operation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 21:25:22 +01:00