Direct inspection of ES.FUT_2024-Q1.dbn.zst (databento python client, offline
kubectl-cp from PVC) pinpoints the source of the 8,799 corrupt bars that
sanitize_bars (Fix 25) caught at the bar-level gate.
Q1 file content:
Outrights (correct): ESH4/ESM4/ESU4/ESZ4/ESH5 — 43,353 records (76.87%)
price range $5,063–$5,478
Spreads (poison): ESH4-ESM4/ESM4-ESU4/... — 13,048 records (23.13%)
price range $47.30–$219.70
Calendar spreads trade at the price DIFFERENCE between adjacent contracts
(~$60-150 cost-of-carry roll basis). Databento's stype_in=parent resolution
for ES.FUT returns BOTH outrights AND every spread combination in the same
DBN stream. The legacy decoder keyed only by ts_event + dedup-by-volume;
during low-volume overnight windows + active rollover periods, spread bars
beat the outright on volume and survived the dedup. 780 spread bars
survived in Q1 alone; ~8,800 across 2024-2026.
Fix: build dbn::TsSymbolMap from metadata once, resolve each record's
instrument_id → symbol, skip any symbol containing `-` (spread separator).
Same-ts dedup-by-volume continues to handle legitimate front/back-month
overlap among outrights.
Adds `time = "0.3"` to ml/Cargo.toml (dbn::TsSymbolMap uses time::Date).
Why complementary to the Fix 25 sanitize_bars gate:
- Spread filter (this commit) catches the cause precisely by symbol pattern,
but only for instruments where parent-symbol resolution is the source.
- sanitize_bars catches the symptom universally by close-ratio bound, but
can't distinguish a low-basis spread from a fast-moving outright in
extreme cases.
Both layers active: spread filter dispatches at parser level, sanitize as
defense-in-depth backstop for unknown future contamination shapes.
Validation: `cargo check -p ml --example train_baseline_rl` clean.
Refs: Bug 2 chain (#191, #194, label_scale=5443 leaks), today's
sanitize_bars find (Fix 25). Closes the contamination-source investigation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `ig_diag` binary under ml-explainability/src/bin/ that runs
Integrated Gradients on a trained DQN safetensors checkpoint and
writes a per-feature attribution report as JSON. Designed for offline
model inspection, not the inference hot path.
Forward target: mean(Q[direction, 0..4]), which simplifies to V(s)
under the dueling identity (mean of centered advantages is zero by
the identifiability constraint). Smooth, no argmax discontinuity,
well-posed for IG.
Regime-head handling: loads only the `trending__`-prefixed weights
(matches RegimeConditionalDQN::load_from_merged_safetensors). Full
multi-head attribution is a future extension.
CLI args:
--checkpoint PATH safetensors file
--states auto|PATH `auto` samples from test_data/feature-cache/
*.fxcache; otherwise a JSON file containing
{ "states": [[f32; STATE_DIM], ...] }
--feature-names PATH JSON array of STATE_DIM names (optional)
--num-steps N IG Riemann steps (default 50)
--output PATH output JSON (default ig_report.json)
--auto-samples N fxcache sample count (default 16)
--seed N LCG seed for reproducibility (default 42)
Output JSON schema:
{
"schema_version": 1,
"checkpoint", "num_steps", "state_dim", "num_states",
"forward_target", "regime_head",
"features": [ { "name", "mean_abs", "stddev", "mean_signed" } ],
"top_10_by_mean_abs": [...],
"completeness": {
"worst_relative_error": f64,
"per_state": [ { "state_idx", "sum_attributions",
"f_input_minus_f_baseline", "relative_error" } ]
}
}
NoisyNet is disabled on both the Q-network and target network before
IG runs so attributions are deterministic (same checkpoint + states +
num_steps = bit-identical attributions).
Gated behind the `ig-diag-cli` Cargo feature (optional Cargo binary
feature, not a runtime flag) because the binary depends on ml-dqn.
Cannot depend on `ml` due to a cyclic dependency (ml already depends
on ml-explainability). To avoid pulling in the heavy `ml` crate, the
fxcache header parser is reimplemented inline in the binary (~80 LOC,
matches ml/src/fxcache.rs byte-for-byte for v4 files).
Tests:
- tests/ig_diag_cli_integration.rs: end-to-end test that builds a
DQN, saves a checkpoint, writes a states JSON, invokes the binary
via std::process::Command, parses the output, asserts schema +
completeness axiom (worst relative error < 5%). Gated with
#[cfg(all(feature = "cuda", feature = "ig-diag-cli"))] + #[ignore]
because it needs CUDA + the binary pre-built.
Build/run:
cargo build --release -p ml-explainability \
--features ig-diag-cli --bin ig_diag
cargo test -p ml-explainability --features ig-diag-cli \
--test ig_diag_cli_integration -- --ignored --nocapture
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
OFICalculator + MicrostructureState fed per-tick. Bar close snapshots
and broadcasts 20 features via tokio::watch channel. Intermediate
snapshots available for speculative compute between bars.
Databento live client integration deferred (requires crate dep).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Integration test now loads imbalance_bar_threshold and ewma_alpha from
config/training/dqn-smoketest.toml. Single source of truth for all
config values. Production threshold lowered to 0.5 for maximum bar yield.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
First load: full MBP-10 decompression (~450s for 19G).
Subsequent loads: bincode deserialize (<1s).
Cache key: hash(dir + symbol + threshold + alpha).
Auto-invalidates when any source .dbn file is newer than cache.
Cache failures are non-fatal — falls through to recomputation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CUDA kernels: 36 of 37 compiled with __nv_bfloat16* (dt_kernels.cu remaining)
Rust types: ALL CudaSlice<f32> → CudaSlice<half::bf16> across ml + ml-core
Build: all kernels now compiled with common_device_functions.cuh (BF16 helpers)
BF16 math wrappers: bf16_sqrt, bf16_log, bf16_exp, bf16_pow, bf16_fabs, bf16_fmax,
bf16_fmin, bf16_cos, bf16_zero, bf16_one, bf16(), atomicAddBF16
NOT YET COMPILING — ml-core boundary errors (Vec<f32> → Vec<half::bf16>)
and dt_kernels.cu float*__nv_bfloat16 ambiguity remain.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- cudarc features: removed "nvrtc" (no longer used, all kernels precompiled)
- cudarc features: added "f16" (enables DeviceRepr + ValidAsZeroBits for half::bf16)
- CudaSlice<half::bf16> now fully functional with all cudarc operations
- Foundation for full BF16 tensor core conversion (next session)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace scattered VRAM-based if/else chains with a declarative TOML profile
system. GPU profiles (rtx3050, a100, h100, default) are selected by device
name and embedded at compile time via include_str! for zero-filesystem
fallback in CI/containers, with filesystem and env var overrides.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Final cleanup:
- 61 test files + 5 example files: candle imports replaced
- 8 testing/integration files: migrated to cudarc/ml-core types
- 3 services/trading_service test files: migrated
- Root Cargo.toml: candle-core, candle-nn removed from [workspace.dependencies]
- crates/ml/Cargo.toml: candle-nn dependency removed
- testing/e2e/Cargo.toml: candle-core dependency removed
Zero active candle_core/candle_nn/candle_optimisers code references remain.
Zero candle dependency declarations in any Cargo.toml.
Remaining "candle" strings are exclusively in doc comments.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CUDA infrastructure in ml-core (cuda_compile, gpu/capabilities, gpu/l2_cache)
previously accessed cudarc types through candle_core::cuda_backend::cudarc --
a transitive re-export that coupled low-level CUDA driver calls to Candle.
Changes:
- Add cudarc 0.17 as direct optional dep (gated behind cuda feature)
- Replace all candle_core::cuda_backend::cudarc paths with direct cudarc::
- Generalize OOM detection to accept &dyn Debug (was &CandleError)
- Add generic computation_error_to_common_error() alongside legacy alias
Candle references: 163 -> 72 (remaining are autograd-dependent: Tensor,
Var, GradStore, VarBuilder -- cannot be replaced with cudarc raw ops)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove ALL #[cfg(feature = "cuda")] guards (~400+ occurrences)
- Remove ALL #[cfg_attr(not(feature = "cuda"), ignore)] test annotations (~250)
- Make cuda default feature in 9 ML crates (ml, ml-core, ml-dqn, ml-ppo, etc.)
- Convert nvrtc JIT compilation to precompiled nvcc (searchsorted, prefix_sum)
- Move compile_ptx_for_device() to ml-core for shared access
- Delete dead CPU code: multi_step.rs, self_supervised_pretraining.rs,
training_guard_gpu_tests.rs, CPU PER buffer paths, CPU Q-diagnostics
- Replace unwrap_or(Device::Cpu) with hard errors everywhere
- Remove dead is_cuda() else branches in DQN/PPO/hyperopt trainers
- Change config defaults from "cpu" to "cuda" (rainbow, tlob, pipeline)
- Port IQL value network to GPU kernel (5 CUDA entry points)
- Port HER goal relabeling to GPU kernel (warp-per-sample)
- Wire DSR GPU-to-CPU sync in training loop
- cfg!(feature = "cuda") → true in inference_validator
Zero warnings, zero errors across entire workspace.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move TFT, Mamba-2, Liquid, TGGN, TLOB, KAN, xLSTM, and Diffusion model
implementations to ml-supervised. Bridge files (UnifiedTrainable adapters,
Checkpointable impls) stay in ml. Delete AsyncDataLoader (replaced by
StreamingDbnLoader + simple .chunks() batching). Remove empty ml-infra
scaffold — the remaining ml modules are too tightly coupled for clean
extraction, so ml stays as the orchestration facade.
- ml-supervised: 234 tests, 0 failures
- ml: 1687 tests, 0 failures
- Workspace: 0 compilation errors
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move 24 PPO source files + flow_policy/ from ml into standalone
ml-ppo crate. The ml crate's ppo module is now a thin re-export layer
(`pub use ml_ppo::*`) plus two bridge files (trainable_adapter.rs,
stress_testing.rs) that depend on ml-internal types.
Key changes:
- ml-ppo: 25 modules (incl flow_policy subdir), 198 tests, standalone
- ml: depends on ml-ppo, re-exports via ppo/mod.rs
- Import rewrites: crate::common::action → ml_core::action_space,
crate::dqn::{mixed_precision,xavier_init} → ml_core::*,
crate::gradient_accumulation → ml_core::gradient_accumulation
- ml tests: 1929 pass (down from 2127 — 198 moved to ml-ppo)
- Workspace: 0 errors
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move 53 DQN source files + gpu_replay_buffer from ml into standalone
ml-dqn crate. The ml crate's dqn module is now a thin re-export layer
(`pub use ml_dqn::*`) plus two bridge files (trainable_adapter.rs,
stress_testing.rs) that depend on ml-internal types.
Key changes:
- ml-dqn: 45 modules, 334 tests, compiles standalone
- ml: depends on ml-dqn, re-exports via dqn/mod.rs
- DQN.config: pub(crate) → pub for cross-crate access
- DQNAgent checkpoint methods moved into ml-dqn
- gpu_replay_buffer moved from cuda_pipeline/ to ml-dqn
- 8 dead-code files removed (never declared in mod.rs)
Net: -30,648 lines from ml crate. Workspace: 0 errors.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move Sharpe ratio metrics and SIMD performance module to ml-core.
These only depend on MLError which is already in ml-core.
Add approx = "0.5" to ml-core dev-dependencies for Sharpe tests.
Skip observability/ (needs prometheus dep — stays in ml).
Test counts: 271 ml-core + 2487 ml = 2758 total, 0 failures
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move 6 shared modules (~2.3K lines) from ml to ml-core:
- trading_action.rs (TradingAction enum)
- action_space.rs (action masking, re-exports)
- xavier_init.rs (Xavier/Glorot weight initialization)
- mixed_precision.rs (AMP utilities, FP16/BF16)
- order_router.rs (deterministic order routing)
- portfolio_tracker.rs (portfolio state tracking)
With TradingAction now in ml-core alongside FactoredAction, the
FactoredActionExt extension trait is eliminated entirely —
from_trading_action()/to_trading_action() become inherent methods
on FactoredAction. This removes the need for `use FactoredActionExt`
imports in reward.rs, gae.rs, and trajectories.rs.
Test counts: 250 ml-core + 2508 ml = 2758 total, 0 failures
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move optimizers, gradient_accumulation, gradient_utils, cuda_compat,
tensor_ops, and gpu (device config, capabilities, memory profiling) to
ml-core. These are shared compute primitives used by all models.
Also commit module files for core types (common, config, error, model,
traits, types) that were moved from ml to ml-core in task 5a but left
staged without being committed.
Notable changes:
- resolve_batch_size() stays in ml (new batch_size_resolver module)
because it depends on memory_optimization::auto_batch_size which
has not yet moved to ml-core
- FactoredAction legacy bridge converted from inherent impl to
extension trait (FactoredActionLegacy) since FactoredAction is now
defined in ml-core, not ml
- candle-optimisers added to ml-core dependencies (needed by Adam)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move all inline type definitions from ml/src/lib.rs to ml-core:
MLError, MLResult, Trade, MarketRegime, HealthStatus, Features,
MLModel trait, ModelRegistry, ParallelExecutor, LatencyOptimizer,
TrainingMetrics, ValidationMetrics, InferenceResult, ModelMetadata.
Dedup: consolidate ConfigError{reason}/ConfigurationError(msg) into
single ConfigError(String) tuple variant (was 2 variants, 174 refs).
Cleanup: convert create_hft_* free functions to associated methods
(HFTPerformanceProfile::ultra_low_latency(), ParallelExecutor::hft()).
ml facade re-exports via `pub use ml_core::*`.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Re-export HardwareTimestamp through data crate instead of ml/risk
depending directly on trading_engine. Reduces coupling between
the ML pipeline and the trading engine.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Copied from api_gateway, removed REST handlers (port 8080),
added tonic-web + CORS for grpc-web browser access.
Binary renamed: api-gateway → api
Changes:
- Package name: api-gateway → api
- Deleted src/handlers/ (REST ML endpoints on port 8080)
- Added tonic-web 0.13 + tower-http CORS layer
- Server::builder().accept_http1(true) for grpc-web
- CORS_ORIGINS env var (default http://localhost:5173)
- Metrics server on port 9091 (axum) preserved
- All 95 lib tests pass, 0 clippy warnings
- Added services/api to workspace members
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rename 7 service binaries from snake_case to kebab-case to match
K8s deployment names. Update Cargo.toml package/bin names, K8s
manifest S3 paths and commands, and cross-crate dependency keys.
- api_gateway → api-gateway
- trading_service → trading-service
- broker_gateway_service → broker-gateway
- ml_training_service → ml-training-service
- backtesting_service → backtesting-service
- trading_agent_service → trading-agent-service
- data_acquisition_service → data-acquisition-service
broker-gateway gets an explicit [lib] name = "broker_gateway_service"
since its new package name maps to broker_gateway (not the original
broker_gateway_service used in source code). All other services map
correctly with Rust's automatic hyphen-to-underscore conversion.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Change PercentileScaler, RewardNormalizer, CompositeReward, and
PPORewardShaper public APIs from f64 to f32 — matching what callers
actually pass. Internal EMA/percentile math stays f64 for precision.
Keep data_loader SMA/EMA/MACD accumulators in f64 throughout (was
f64→f32→f64 per step), cast to f32 only at output boundary.
Remove dead _transition computation in rainbow_agent_impl and unused
bigdecimal deps from broker_gateway_service and trading_service.
2704 tests pass, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
fxt 2.0 overhaul: consolidated proto/, absorbed monitoring into API Gateway,
new 15-command CLI with gRPC, MCP server mode, TUI cockpit framework.
159 files changed, net -11,835 lines.
Embed DQNConfig architecture hash (SHA-256 of state_dim, num_actions,
hidden_dims, dueling/distributional/noisy/IQN flags) in safetensors
file header on save. Validate hash on load to catch shape mismatches
before touching the VarMap - prevents silent corruption from loading
checkpoints trained with different network architectures.
All 5 save paths (trainable_adapter, trainer serialize_model,
DQNAgentType::save_checkpoint, RegimeConditionalDQN per-head) now
embed metadata. All 3 load paths (DQN::load_from_safetensors,
trainable_adapter::load_checkpoint, ensemble adapter) validate.
No backward compatibility: checkpoints without metadata are rejected
with a clear error message to re-train.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove from workspace members, delete K8s deployment, remove nginx routing.
Training metrics and system health now served directly by API Gateway.
Changes:
- Cargo.toml: remove services/monitoring_service from workspace members
- services/monitoring_service/: deleted entirely (6 files)
- infra/k8s/services/monitoring-service.yaml: deleted
- infra/k8s/network-policies/monitoring-service.yaml: deleted
- infra/k8s/network-policies/api-gateway.yaml: remove egress rule to monitoring-service
- infra/k8s/network-policies/web-gateway.yaml: remove egress rule to monitoring-service
- infra/k8s/services/api-gateway.yaml: remove stale MONITORING_SERVICE_URL env var
- infra/k8s/gitlab/tailscale-proxy.yaml: redirect monitor.fxhnt.ai to api-gateway:50051
- .gitlab-ci.yml: remove monitoring_service from compile, copy, and deploy loops
- services/trading_service/src/services/monitoring.rs: update stale comments to
reference api_gateway (not monitoring_service) as training metrics host
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- trade_ml.rs: Replace 3 mock data fallbacks (submit, predictions,
performance) with proper error propagation. Commands now fail
honestly when the API Gateway is unreachable instead of silently
returning fake data. Mark 3 integration tests as #[ignore].
- monitoring_service: Add tonic-health with set_serving for
MonitoringServiceServer. Enables grpc_health_probe readiness checks.
- ml_training_service: Add tonic-health with set_serving for
MlTrainingServiceServer. Wired into both TLS and non-TLS paths.
- data_acquisition_service: Add tonic-health with set_serving for
DataAcquisitionServiceServer.
- ml/cuda_streams: Fix pre-existing unused variable clippy warning.
All 8 services now have standard gRPC health checking enabled.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Prometheus client queries training/hyperopt/GPU/K8s metrics, gRPC service
exposes unary + server-streaming RPCs, 4 unit tests, zero clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Match the actual K8s broker-gateway service port discovered during
live validation of fxt broker check.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove entire trading_engine/src/compliance/ directory (9 files, 16,068 LOC)
and 18 associated test files (18,069 LOC). Comprehensive audit confirmed
zero external callers for all types (ISO 27001, SOX, MiFID II, best
execution, automated reporting). Remove unused cron dependency.
Independent compliance modules in risk/ and risk-data/ are preserved.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
