Three things landing atomically because they're load-bearing for each other:
1. **Trend-scanning leakage fix** — trend_scanning.rs was emitting OLS slope+t-stat
over a *forward* window [t, t+L]. With the Phase 1a label = sign(price[t+60]
− price[t]), the forward feature window overlaps the label window, contaminating
it. Purged walk-forward only sterilizes forward-looking *labels* that cross
the train/val split, not forward-looking *features* that peek inside the same
horizon the label measures. The leak inflated MLP accuracy from 0.49
(legacy 74-dim baseline) to 0.75 — vanished to 0.50 after switching to a
trailing window. Bounded the perfect-fit t-stat sentinel from ±1e6 → ±20
(p<1e-30 is already meaningless); eliminated the 16k corruption-cap drops.
2. **Variable-dim alpha column** — fxcache schema now carries the alpha-feature
width via metadata (`alpha_feature_dim`), not a compile-time constant. Same
on-disk format hosts the 134-dim bar-level stack OR the 81-dim snapshot stack.
Reader + auto-detect honor the metadata-declared dim; downstream MLP auto-sizes
`in_dim`. Single schema, no forks.
3. **Snapshot pipeline (Phase 1c falsification)** — `snapshot_pipeline.rs`: 81-dim
per-MBP10-snapshot extractor reusing 10 snapshot-native alpha blocks + 6 new
snapshot-specific features (time-since-trade, time-since-snap, event-rate,
spread-bps, L1-imbalance, microprice-mid drift). `precompute_features` gets
`--row-unit snapshot` flag; emits one fxcache row per LOB update (1.97M rows
from MBP-10 data vs 206K for bar mode).
**Smoke verdict on real data** (ES.FUT, 1.97M snapshots, 384K val):
- Bar-level honest alpha: accuracy=0.5005, AUC=0.5043 (no signal)
- **Snapshot-level alpha**: accuracy=0.5241, AUC=0.6849 (real signal, 384K val)
- GBM corroboration: accuracy=0.5401 (non-linear partitioning sees more)
- Horizon decay: alpha peaks at K=20-50 snapshots (~5-25ms), gone by K=500
- Regime-conditional: spread-Q4 quintile hits 0.752 accuracy on 76k samples
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Phase 2A scaffolding lands FIRST per spec §4.4 ABI contract. Phase 1.2
cost kernel reads LobBar; both dev synthetic and prod fxcache produce
LobBar — dev/prod parity per Q3.
Generators: flat_market, drift_market, ou_market, regime_switch_market
(seeded RNG for reproducible tests). Regime-switch test uses sticky
0.99/0.01 transitions (true regime persistence; spec's 50/50 was a
random walk, not a regime switch — corrected with code comment).
behavioral_suite test target wired into Cargo.toml; will run all
22 Phase 2 tests once they land in Phase 2B/2C.
Audit doc: SP15 Phase 2A.1 entry appended to docs/dqn-wire-up-audit.md
per Invariant 7 (component changes require audit-doc update).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Direct inspection of ES.FUT_2024-Q1.dbn.zst (databento python client, offline
kubectl-cp from PVC) pinpoints the source of the 8,799 corrupt bars that
sanitize_bars (Fix 25) caught at the bar-level gate.
Q1 file content:
Outrights (correct): ESH4/ESM4/ESU4/ESZ4/ESH5 — 43,353 records (76.87%)
price range $5,063–$5,478
Spreads (poison): ESH4-ESM4/ESM4-ESU4/... — 13,048 records (23.13%)
price range $47.30–$219.70
Calendar spreads trade at the price DIFFERENCE between adjacent contracts
(~$60-150 cost-of-carry roll basis). Databento's stype_in=parent resolution
for ES.FUT returns BOTH outrights AND every spread combination in the same
DBN stream. The legacy decoder keyed only by ts_event + dedup-by-volume;
during low-volume overnight windows + active rollover periods, spread bars
beat the outright on volume and survived the dedup. 780 spread bars
survived in Q1 alone; ~8,800 across 2024-2026.
Fix: build dbn::TsSymbolMap from metadata once, resolve each record's
instrument_id → symbol, skip any symbol containing `-` (spread separator).
Same-ts dedup-by-volume continues to handle legitimate front/back-month
overlap among outrights.
Adds `time = "0.3"` to ml/Cargo.toml (dbn::TsSymbolMap uses time::Date).
Why complementary to the Fix 25 sanitize_bars gate:
- Spread filter (this commit) catches the cause precisely by symbol pattern,
but only for instruments where parent-symbol resolution is the source.
- sanitize_bars catches the symptom universally by close-ratio bound, but
can't distinguish a low-basis spread from a fast-moving outright in
extreme cases.
Both layers active: spread filter dispatches at parser level, sanitize as
defense-in-depth backstop for unknown future contamination shapes.
Validation: `cargo check -p ml --example train_baseline_rl` clean.
Refs: Bug 2 chain (#191, #194, label_scale=5443 leaks), today's
sanitize_bars find (Fix 25). Closes the contamination-source investigation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two-part fix that finally removes ~35% of ml-* sub-crates from
service-binary build closures:
1. crates/ml/Cargo.toml: 8 leaf-level ml-* sub-crates become optional
behind feature flags (one feature per `pub mod` in lib.rs):
ml-backtesting ↔ feature `backtest-mod` (ml::backtesting)
ml-paper-trading ↔ feature `paper-trading-mod` (ml::paper_trading)
ml-stress-testing ↔ feature `stress-testing-mod` (ml::stress_testing)
ml-explainability ↔ feature `explainability-mod` (ml::explainability)
ml-universe ↔ feature `universe-mod` (ml::universe)
ml-regime-detection ↔ feature `regime-detection-mod`(ml::regime_detection)
ml-validation ↔ feature `validation-mod` (ml::validation)
ml-data-validation ↔ feature `data-validation-mod` (ml::data_validation)
Aggregated into the umbrella `full-stack` feature, which is in
`default = [...]` so existing `ml.workspace = true` callers see
identical behavior (testing/integration, crates/backtesting).
2. services/trading_service + services/backtesting_service: switched
from `ml.workspace = true` to explicit `path = "../../crates/ml"`
form with `default-features = false`. This is necessary because
cargo 1.89 silently ignores `default-features = false` when
combined with `workspace = true` (a known cargo limitation).
Path form bypasses the workspace dep and applies the opt-out.
Verified by `cargo tree -p X | grep ml-* | wc -l`:
trading-service 23 → 16 (-30%)
backtesting-service 23 → 16 (-30%)
Source-grep verified neither service references any of the 8 gated
modules (`use ml::backtesting`, `use ml::explainability`, etc. all
zero hits in src/). The only `ml::explainability` mention in
trading-service is a string in an error log — not a code path.
ml-training-service and trading-agent-service were already on path
form with default-features = false; they're unaffected here.
cargo check --workspace passes.
- ml/Cargo.toml: describe why ndarray's blas feature stays disabled
(CI compile pool has no libopenblas-dev; GPU cuBLAS handles the
hot path) rather than labelling it a TODO.
- dbn_sequence_loader.rs: Wave C regime-detection branch emits zeros
when only that flag is enabled; the live feed lives under WaveD.
Reword from "TODO Wave C" to a description of that superseding.
- ensemble/adapters/{liquid,tggn,tlob,xlstm}.rs: checkpoint loading
currently constructs fresh GpuLinear weights and logs a runtime
warning so the ignored checkpoint path is visible. No new
functionality, just reword the repeated TODO.
- ensemble/model_adapter.rs: the neutral-prediction adapter is
guarded by the ensemble's confidence threshold (0.0 = filtered),
making it a no-op stub used for end-to-end wiring. Describe that
contract explicitly.
- hyperopt/adapters/tft.rs: the input_dim=51 line is load-bearing
(5 static + 10 known + 36 unknown matches the CUDA layout). Drop
the "should be 42" aside.
- trainers/tft/trainer.rs: initialize_optimizer returns Err until
GpuAdamW is wired; the `let _ = &self.optimizer;` anchor in the
training loop keeps the migration target visible.
- trainers/tlob.rs: save_checkpoint / serialize_model both surface
errors until GpuVarStore safetensors serialisation lands. Mark
the gap declaratively rather than as a TODO.
- transformers/mod.rs: only AttentionMask is implemented in the
attention submodule; drop the aspirational re-export list.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
cublasLtMatmul passes workspace+stream per-call (no handle state),
eliminating the cublasSetStream workspace conflict that hung CUDA
Graph mega-capture on H100 with cublasGemmEx.
Forward (22 call sites via sgemm_f32/sgemm_f32_ldb):
- cublasLtMatmul with CUBLAS_COMPUTE_32F + per-call stream
- No more cublasSetStream for branch dispatch (stream passed directly)
- 32MB workspace unconditionally (TF32 HMMA needs it even on Ampere)
Backward (7 call sites via sgemm_f32):
- Same cublasLtMatmul pattern with flexible transa/transb
- Stream parameter threaded through backward_fc_layer methods
TF32 tensor cores enabled via CUBLAS_TF32_TENSOR_OP_MATH math mode
on the cuBLAS handle. CUBLAS_COMPUTE_32F (not FAST_TF32) used in
matmul descriptors for SM86 compatibility.
19/19 smoke tests pass (sequential). Ready for H100 graph_mega test.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- cudarc features: removed "nvrtc" (no longer used, all kernels precompiled)
- cudarc features: added "f16" (enables DeviceRepr + ValidAsZeroBits for half::bf16)
- CudaSlice<half::bf16> now fully functional with all cudarc operations
- Foundation for full BF16 tensor core conversion (next session)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Final cleanup:
- 61 test files + 5 example files: candle imports replaced
- 8 testing/integration files: migrated to cudarc/ml-core types
- 3 services/trading_service test files: migrated
- Root Cargo.toml: candle-core, candle-nn removed from [workspace.dependencies]
- crates/ml/Cargo.toml: candle-nn dependency removed
- testing/e2e/Cargo.toml: candle-core dependency removed
Zero active candle_core/candle_nn/candle_optimisers code references remain.
Zero candle dependency declarations in any Cargo.toml.
Remaining "candle" strings are exclusively in doc comments.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove ALL #[cfg(feature = "cuda")] guards (~400+ occurrences)
- Remove ALL #[cfg_attr(not(feature = "cuda"), ignore)] test annotations (~250)
- Make cuda default feature in 9 ML crates (ml, ml-core, ml-dqn, ml-ppo, etc.)
- Convert nvrtc JIT compilation to precompiled nvcc (searchsorted, prefix_sum)
- Move compile_ptx_for_device() to ml-core for shared access
- Delete dead CPU code: multi_step.rs, self_supervised_pretraining.rs,
training_guard_gpu_tests.rs, CPU PER buffer paths, CPU Q-diagnostics
- Replace unwrap_or(Device::Cpu) with hard errors everywhere
- Remove dead is_cuda() else branches in DQN/PPO/hyperopt trainers
- Change config defaults from "cpu" to "cuda" (rainbow, tlob, pipeline)
- Port IQL value network to GPU kernel (5 CUDA entry points)
- Port HER goal relabeling to GPU kernel (warp-per-sample)
- Wire DSR GPU-to-CPU sync in training loop
- cfg!(feature = "cuda") → true in inference_validator
Zero warnings, zero errors across entire workspace.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The ml crate had `default = ["minimal-inference", "cuda"]` which pulled
in cudarc/candle-kernels requiring nvcc. CI test-gate runs on
ci-builder-cpu (no CUDA) so `cargo clippy --workspace` always panicked
with "Failed to execute nvcc: No such file or directory".
CUDA is now opt-in only — compile-and-train template already passes
`--features ml/cuda` explicitly for GPU builds.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three fixes for GPU PER hot path:
1. IQN quantile loss used empty CPU weights Vec instead of GPU-resident
weights_tensor_cached — caused CUDA_ERROR_ILLEGAL_ADDRESS from
uninitialized GPU memory. Now uses cached GPU tensor matching C51
and standard DQN paths.
2. GpuPrioritized add()/add_batch() replaced with StagedGpuBuffer:
add() stages on CPU (Vec::push, zero GPU ops), sample() batch-flushes
staging→GPU in one DMA before sampling. Production path (insert_batch_tensors)
bypasses staging entirely — GPU→GPU with zero CPU.
3. All 9 ML sub-crates default to cuda feature so `cargo test -p ml-dqn`
exercises GPU code paths on CUDA workstations. CI service crates use
default-features=false, unaffected.
Test results: 350 passed (was 343+7 failed), 0 failed, 1 ignored.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ml-supervised, ml-ensemble, ml-labeling, ml-explainability, ml-hyperopt
all had default = ["cuda"] which pulled cudarc into the CPU services
build, causing compile-services to fail with "nvcc not found".
Changed all to default = [] and wired ml/Cargo.toml cuda feature to
propagate to all 8 sub-crates (was only 3: ml-core, ml-dqn, ml-ppo).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove cuda from default features in ml-core, ml-dqn, ml-ppo, ml
- Propagate cuda feature from ml → ml-core/ml-dqn/ml-ppo
- CI compile-training already uses --features ml/cuda explicitly
- Fix MaxDD log format: {:.1}% → {:.3}% (was rounding 0.033% to 0.0%)
- Suppress unused_labels/unused_variables warnings for cfg(cuda) code
- Add CALLBACK_ENDPOINT env to ml-training-service deployment
- Fix Grafana active_workers query to use sum() with fallback
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move TFT, Mamba-2, Liquid, TGGN, TLOB, KAN, xLSTM, and Diffusion model
implementations to ml-supervised. Bridge files (UnifiedTrainable adapters,
Checkpointable impls) stay in ml. Delete AsyncDataLoader (replaced by
StreamingDbnLoader + simple .chunks() batching). Remove empty ml-infra
scaffold — the remaining ml modules are too tightly coupled for clean
extraction, so ml stays as the orchestration facade.
- ml-supervised: 234 tests, 0 failures
- ml: 1687 tests, 0 failures
- Workspace: 0 compilation errors
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move 24 PPO source files + flow_policy/ from ml into standalone
ml-ppo crate. The ml crate's ppo module is now a thin re-export layer
(`pub use ml_ppo::*`) plus two bridge files (trainable_adapter.rs,
stress_testing.rs) that depend on ml-internal types.
Key changes:
- ml-ppo: 25 modules (incl flow_policy subdir), 198 tests, standalone
- ml: depends on ml-ppo, re-exports via ppo/mod.rs
- Import rewrites: crate::common::action → ml_core::action_space,
crate::dqn::{mixed_precision,xavier_init} → ml_core::*,
crate::gradient_accumulation → ml_core::gradient_accumulation
- ml tests: 1929 pass (down from 2127 — 198 moved to ml-ppo)
- Workspace: 0 errors
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move 53 DQN source files + gpu_replay_buffer from ml into standalone
ml-dqn crate. The ml crate's dqn module is now a thin re-export layer
(`pub use ml_dqn::*`) plus two bridge files (trainable_adapter.rs,
stress_testing.rs) that depend on ml-internal types.
Key changes:
- ml-dqn: 45 modules, 334 tests, compiles standalone
- ml: depends on ml-dqn, re-exports via dqn/mod.rs
- DQN.config: pub(crate) → pub for cross-crate access
- DQNAgent checkpoint methods moved into ml-dqn
- gpu_replay_buffer moved from cuda_pipeline/ to ml-dqn
- 8 dead-code files removed (never declared in mod.rs)
Net: -30,648 lines from ml crate. Workspace: 0 errors.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move all inline type definitions from ml/src/lib.rs to ml-core:
MLError, MLResult, Trade, MarketRegime, HealthStatus, Features,
MLModel trait, ModelRegistry, ParallelExecutor, LatencyOptimizer,
TrainingMetrics, ValidationMetrics, InferenceResult, ModelMetadata.
Dedup: consolidate ConfigError{reason}/ConfigurationError(msg) into
single ConfigError(String) tuple variant (was 2 variants, 174 refs).
Cleanup: convert create_hft_* free functions to associated methods
(HFTPerformanceProfile::ultra_low_latency(), ParallelExecutor::hft()).
ml facade re-exports via `pub use ml_core::*`.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add QuestDB ILP sink for training metrics, update Prometheus scrape
configs, and fix network policies for monitoring stack connectivity.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Re-export HardwareTimestamp through data crate instead of ml/risk
depending directly on trading_engine. Reduces coupling between
the ML pipeline and the trading engine.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Embed DQNConfig architecture hash (SHA-256 of state_dim, num_actions,
hidden_dims, dueling/distributional/noisy/IQN flags) in safetensors
file header on save. Validate hash on load to catch shape mismatches
before touching the VarMap - prevents silent corruption from loading
checkpoints trained with different network architectures.
All 5 save paths (trainable_adapter, trainer serialize_model,
DQNAgentType::save_checkpoint, RegimeConditionalDQN per-head) now
embed metadata. All 3 load paths (DQN::load_from_safetensors,
trainable_adapter::load_checkpoint, ensemble adapter) validate.
No backward compatibility: checkpoints without metadata are rejected
with a clear error message to re-train.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- MultiGpuConfig::detect() probes CUDA ordinals 0..8, returns None
on single-GPU/CPU setups
- shard_indices() splits dataset across devices with remainder handling
- NcclGradientSync (behind `nccl` feature flag) for all-reduce averaging
- Feature: `nccl = ["cuda"]` — requires NCCL library on system
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Split the build pipeline: one compile-services job builds all 8 service
binaries with PVC-backed sccache, saves as artifacts. Then 9 Kaniko jobs
just package pre-built binaries into slim runtime images (~30s each).
Before: 9 parallel Kaniko jobs each doing full cargo build --release
(~20min each, no sccache, 9x duplicated dep compilation)
After: 1 compile job with sccache (~5min cached) + 9 package jobs (~30s)
- Add compile stage between test and build
- Add Dockerfile.runtime (minimal debian + pre-built binary)
- Add Dockerfile.web-gateway-runtime (Node dashboard + pre-built binary)
- Keep Dockerfile.training via Kaniko (needs CUDA dev image for H100)
- Remove all SCCACHE_BUCKET build-args from service builds
- Use dir:// context for Kaniko (only sends build-out/ dir, not full repo)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Mass cleanup of crates/ml/:
- Delete 121 dead/broken/superseded example files (-44,098 lines)
- Delete 6 QAT test files (-4,201 lines) — QAT is disabled at runtime
(trainer.rs falls back to FP32 with warning)
- Delete 2 stale markdown files in examples/
- Delete orphaned src/bin/train_tft.rs (unimplemented stub)
- Clean up Cargo.toml: remove stale [[example]] entries, add missing ones
- Fix stale binary references in log_size_test.rs
- Add infra consistency test (35 checks across Dockerfile/train.sh/Cargo.toml)
Remaining examples (7): train_baseline_rl, train_baseline_supervised,
evaluate_baseline, hyperopt_baseline_rl, hyperopt_baseline_supervised,
download_baseline, cuda_test
All 2390 lib tests pass. All examples compile. 35/35 infra checks pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move 17 library crates into crates/, CLI binary into bin/fxt,
consolidate 10 test crates into testing/, split config crate
from deployment config files.
Root directory reduced from 38+ to ~17 directories.
All Cargo.toml paths and build.rs proto refs updated.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
