- Trade detection now counts reversals (S100→L50) as completed segments
- old_pos_pnl saved before position update for correct reversal P&L
- realized_pnl writeback uses old position PnL on reversal bars
- 0% win rate persists — needs deeper investigation (likely tx cost interaction)
WIP: The trade_return formula produces correct sign for raw market moves,
but every trade still shows as a loss. Suspect tx costs on both entry AND
exit of each reversal segment exceed the 1-bar price movement.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ROOT CAUSE: 5 interlocking bugs made learning impossible:
1. DSR denominator floor 1e-12 produced values in millions → drowned all signal
2. Global [-1,+1] clamp destroyed Bellman equation signal (can't distinguish
catastrophic loss from mild loss)
3. v_range=20 exactly equals V_max for gamma=0.95 → Bellman target pins at
ceiling → Q-values saturate → Q-gap collapses to 0.0000
4. num_atoms=11 over 40-unit range = 4.0 per atom (C51 paper min is 51)
5. 6/7 reward components were penalties → mean_reward=-0.311 regardless of action
FIXES:
- DSR denominator floor: 1e-12 → 0.01 (prevents million-scale spikes)
- Each component individually clamped BEFORE weighting (DSR to [-1,+1],
z-score to [-3,+3], drawdown to [0,1], time decay to [0,0.3])
- Removed global [-1,+1] clamp (no longer needed with bounded components)
- profit_take_bonus: 0.1 → 0.01 (was 100x too large, caused reward hacking)
- Removed confidence scaling (positive feedback loop destabilized learning)
- Removed regime scaling (non-stationary reward confused the model)
- Dynamic v_range from gamma: v_range = 2.5/(1-gamma)*1.2 (always covers Q range)
- num_atoms minimum: 11 → 51 (C51 paper standard)
- gamma default: 0.99 → 0.95
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move 17 library crates into crates/, CLI binary into bin/fxt,
consolidate 10 test crates into testing/, split config crate
from deployment config files.
Root directory reduced from 38+ to ~17 directories.
All Cargo.toml paths and build.rs proto refs updated.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
filter-repo stripped the blobs but left tree entries. Remove from
index so .gitignore rules take effect. Adds checkpoints/ to gitignore.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add #![deny(clippy::unwrap_used, clippy::expect_used)] to 11 crates that
were missing it, and standardize 3 existing crates to deny both lints.
Test code is exempted via #![cfg_attr(test, allow(...))].
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace _param suppression pattern with actual usage across 21 files:
- adaptive-strategy: wire EpistemicConfig/AleatoricConfig into
UncertaintyQuantifier, KellyConfig into DrawdownTracker,
TLOBConfig into TLOBTransformer
- trading_engine/compliance: store config in 26 compliance structs
(audit_trails, best_execution, sox, iso27001, transaction_reporting,
compliance_reporting, automated_reporting) with public accessors
- fxt: store Channel in LoginClient, ConnectionConfig in ConnectionManager
- ml: remove unused path param from ReplayBuffer::new(), wire
Mamba2Config.target_latency_us into HardwareOptimizer
- services: store TrainingConfig in GpuConfigManager, symbol in
TechnicalIndicatorCalculator
- database: change let _result to let _ (intentional discard)
- trading_engine/brokers: store BrokerConnectorConfig in BrokerConnector
Result: 0 warnings across all 37+ workspace crates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace 44 incorrect drop(write!()) patterns with let _ = write!()
(drop() on fmt::Result triggers clippy warning; let _ = is idiomatic)
- Fix syntax errors from botched drop→let_ replacement (extra closing paren)
- Remove unused imports in ml/src/dqn/agent.rs (std::fs::File, std::io::Read)
- Remove unused #[allow(clippy::expect_used)] in ml/src/inference.rs
- Fix backtesting_service binary re-declaring library modules (mod x instead
of use backtesting_service::x), which caused false dead_code warnings
Result: 0 warnings across all 37+ workspace crates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Coordinator (ml/src/integration/coordinator.rs):
- Delete 9 fake heuristic methods (500+ lines) that pretended to be
real DQN/TFT/TGGN/LNN/Mamba predictions using sin()/tanh() math
- Make generate_model_specific_prediction() return Err instead of
fake predictions — prevents trading on fabricated signals
- Make ensemble fault-tolerant: skip failed models instead of
failing entire ensemble (execute_parallel/execute_sequential)
- Remove double-fallback in execute_single_model error path
Enhanced ML (trading_service):
- get_model_performance(): compute accuracy from real
inference_count/error_count instead of returning all zeros
- get_feature_importance(): return Status::unavailable instead of
hardcoded fake values — honest about missing SHAP implementation
Autonomous scaling (trading_agent_service):
- diversification_score: compute real HHI from instrument volume
distribution instead of hardcoded 0.8
- ml_confidence: keep liquidity heuristic but remove warn!() spam
Position limiter (risk):
- Remove redundant portfolio_id field from CachedPosition — the
DashMap key already provides account-based isolation
- Remove misleading "stub" comment — was not a stub
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add debug_assert_eq! guards in 4 train_baseline functions to catch
bar/feature length misalignment at debug time (#4)
- Remove "last sample targets itself" block in hyperopt PPO adapter
that created ~0 return sample biasing toward HOLD (#5)
- Align hyperopt state_dim 54→51 and num_actions 45→3 to match
train_baseline architecture, making tuned hyperparams transferable (#6)
- Use greedy_action() in evaluate_baseline PPO eval for deterministic results
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Shared DBN loading, timestamp dedup, and spread cost helpers used by
train_baseline, evaluate_baseline, and hyperopt_baseline extracted to
ml/examples/baseline_common/mod.rs to eliminate ~190 lines of duplication.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three critical PPO fixes:
1. Add PPO::act_with_log_prob() returning (action, log_prob, value).
The existing act() discarded the policy log-probability, making
PPO importance sampling use wrong ratios during training.
2. Cap trajectory length with --max-steps-per-epoch in train_baseline
PPO path. DQN already had this limit; PPO iterated all features
(~500K per fold), causing OOM on 4GB GPU.
3. Replace random actions and fake log_prob/value in hyperopt PPO
adapter with real agent.act_with_log_prob() calls. Trajectories
now reflect actual policy behavior for meaningful hyperopt.
Also adds warmup offset alignment to PPO trajectory collection
(matching the DQN fix) and fixes .unwrap() in test.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Use chrono-tz America/New_York for correct EST/EDT trading session
boundaries (was hardcoded UTC-5, off by 1h during daylight saving)
- Normalize effective weights to sum to 1.0 before signal aggregation
(raw weights could sum to anything, biasing the ensemble)
- Skip adapter predictions that return NaN/Inf direction or confidence
instead of letting them poison the weighted average
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fix zstd decoder in train_baseline.rs and evaluate_baseline.rs (same
pattern as hyperopt adapters — branch on .dbn.zst extension). Add CLI
flags for walk-forward config (train/val/test/step months), learning
rate, and max-steps-per-epoch to make pipeline validation feasible.
Pipeline validated end-to-end: hyperopt (5 trials, best Sharpe 2.37) →
walk-forward training (4 folds, 6/1/1 month windows on ES.FUT) →
evaluation (4 fold test sets, checkpoints + norm stats saved).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PPO hyperopt: Split DATA 80/20 for train/val (not trajectory count).
Previously indexed training_data[..num_trajectories] which was only
16 samples from 56K — causing num_batches=0 and NaN from 0/0 division.
Now properly splits data array and uses min(64, num_train) for batch
episodes with ceil division for batch count.
DQN: Fix hardcoded "45 actions" in diagnostic log (now uses actual
q_vec.len()). Replace indexed[..top_n] slice with .take(top_n)
iterator to prevent potential slice panic.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Databento data pipeline stored files as .dbn.zst in symbol subdirectories
(e.g., ES.FUT/ES.FUT_2024-Q1.dbn.zst), but the hyperopt adapters and DQN
data loader only matched .dbn extension and used flat directory listing.
Three fixes:
- Match both .dbn and .dbn.zst file extensions in collect_dbn_files_recursive
- Use recursive directory traversal instead of flat read_dir
- Branch on extension to use Decoder::from_zstd_file() for .dbn.zst files
(from_file() doesn't handle zstd decompression)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The download binary was using the default SType::RawSymbol which
returned empty DBN files (95 bytes) for parent symbols like ES.FUT.
Adding SType::Parent resolves the symbol mapping. Also adjusted end
date to 2026-02-22 (latest available data).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add #![allow(unused_crate_dependencies)] for consistency with other ml tests
- Extract EXPECTED_FEATURE_DIM constant (replaces magic number 51)
- Add validated_folds counter to prevent vacuous pass in walk-forward test
- Replace unwrap_or_else(panic) with match pattern
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three integration tests validating the full pipeline end-to-end:
1. Feature extraction: 90-day synthetic bars -> 51-dim features, no NaN/Inf
2. Walk-forward with normalization: 24-month bars -> windows -> NormStats from train only -> normalized mean ~0
3. No lookahead bias: train timestamps < val start, val timestamps < test start
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Loads trained DQN/PPO checkpoints, runs greedy inference on walk-forward
test windows, computes Sharpe ratio, max drawdown, win rate, profit factor,
and total return per fold. Outputs a JSON evaluation report with aggregate
metrics and sanity checks (beats-random, action diversity, fold consistency).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add ml/examples/train_baseline.rs that trains DQN and PPO models using
expanding walk-forward windows on real Databento OHLCV data.
Features:
- CLI args via clap (--model, --epochs, --batch-size, --data-dir, etc.)
- Recursive .dbn.zst file discovery and OHLCV bar loading
- 51-dim feature extraction via extract_ml_features()
- Walk-forward window generation with NormStats per fold
- DQN training loop with epsilon-greedy, experience replay, early stopping
- PPO training loop with GAE, trajectory collection, early stopping
- PnL-based reward (BUY/SELL/HOLD)
- Safetensors checkpoint saving per fold
- NormStats JSON export for evaluation reproducibility
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements expanding-window walk-forward cross-validation for time-series
ML models, preventing lookahead bias by always evaluating on unseen future
data. Includes z-score NormStats computed from training splits only.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds `ml/examples/download_baseline.rs` that downloads 730 days of
Databento OHLCV-1m data in quarterly chunks for 4 CME futures symbols.
Features:
- Reads universe config from TOML (symbols, date range, dataset)
- Splits date range into calendar-quarter chunks (~90 days each)
- Resume support: skips existing non-empty files
- Uses `get_range_to_file` for streaming writes to .dbn.zst
- Dry-run mode with cost estimate ($0.12/symbol/day)
- Confirmation prompt (skippable with --yes)
- Per-file progress with timing and byte counts
- Failure-tolerant: logs errors and continues
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sequence-buffered adapter with 3-layer MLP projection
(flat_dim -> hidden -> hidden/2 -> 1). Ring buffer collects
seq_len feature vectors, returns neutral prediction until full,
then flattens and projects through the MLP with sigmoid output
mapping to direction [-1,1] and confidence [0,1].
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove panic!() calls from test_futures_baseline_micro_mapping,
use map()+Some() pattern consistent with rest of test suite
- Make DatasetSpec::from_universe() accept a name parameter instead
of hardcoding "futures-baseline"
- Add doc comment to UniverseConfigMeta explaining dead_code fields
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
