OFICalculator + MicrostructureState fed per-tick. Bar close snapshots
and broadcasts 20 features via tokio::watch channel. Intermediate
snapshots available for speculative compute between bars.
Databento live client integration deferred (requires crate dep).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Integration test now loads imbalance_bar_threshold and ewma_alpha from
config/training/dqn-smoketest.toml. Single source of truth for all
config values. Production threshold lowered to 0.5 for maximum bar yield.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
First load: full MBP-10 decompression (~450s for 19G).
Subsequent loads: bincode deserialize (<1s).
Cache key: hash(dir + symbol + threshold + alpha).
Auto-invalidates when any source .dbn file is newer than cache.
Cache failures are non-fatal — falls through to recomputation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CUDA kernels: 36 of 37 compiled with __nv_bfloat16* (dt_kernels.cu remaining)
Rust types: ALL CudaSlice<f32> → CudaSlice<half::bf16> across ml + ml-core
Build: all kernels now compiled with common_device_functions.cuh (BF16 helpers)
BF16 math wrappers: bf16_sqrt, bf16_log, bf16_exp, bf16_pow, bf16_fabs, bf16_fmax,
bf16_fmin, bf16_cos, bf16_zero, bf16_one, bf16(), atomicAddBF16
NOT YET COMPILING — ml-core boundary errors (Vec<f32> → Vec<half::bf16>)
and dt_kernels.cu float*__nv_bfloat16 ambiguity remain.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- cudarc features: removed "nvrtc" (no longer used, all kernels precompiled)
- cudarc features: added "f16" (enables DeviceRepr + ValidAsZeroBits for half::bf16)
- CudaSlice<half::bf16> now fully functional with all cudarc operations
- Foundation for full BF16 tensor core conversion (next session)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace scattered VRAM-based if/else chains with a declarative TOML profile
system. GPU profiles (rtx3050, a100, h100, default) are selected by device
name and embedded at compile time via include_str! for zero-filesystem
fallback in CI/containers, with filesystem and env var overrides.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Final cleanup:
- 61 test files + 5 example files: candle imports replaced
- 8 testing/integration files: migrated to cudarc/ml-core types
- 3 services/trading_service test files: migrated
- Root Cargo.toml: candle-core, candle-nn removed from [workspace.dependencies]
- crates/ml/Cargo.toml: candle-nn dependency removed
- testing/e2e/Cargo.toml: candle-core dependency removed
Zero active candle_core/candle_nn/candle_optimisers code references remain.
Zero candle dependency declarations in any Cargo.toml.
Remaining "candle" strings are exclusively in doc comments.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CUDA infrastructure in ml-core (cuda_compile, gpu/capabilities, gpu/l2_cache)
previously accessed cudarc types through candle_core::cuda_backend::cudarc --
a transitive re-export that coupled low-level CUDA driver calls to Candle.
Changes:
- Add cudarc 0.17 as direct optional dep (gated behind cuda feature)
- Replace all candle_core::cuda_backend::cudarc paths with direct cudarc::
- Generalize OOM detection to accept &dyn Debug (was &CandleError)
- Add generic computation_error_to_common_error() alongside legacy alias
Candle references: 163 -> 72 (remaining are autograd-dependent: Tensor,
Var, GradStore, VarBuilder -- cannot be replaced with cudarc raw ops)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove ALL #[cfg(feature = "cuda")] guards (~400+ occurrences)
- Remove ALL #[cfg_attr(not(feature = "cuda"), ignore)] test annotations (~250)
- Make cuda default feature in 9 ML crates (ml, ml-core, ml-dqn, ml-ppo, etc.)
- Convert nvrtc JIT compilation to precompiled nvcc (searchsorted, prefix_sum)
- Move compile_ptx_for_device() to ml-core for shared access
- Delete dead CPU code: multi_step.rs, self_supervised_pretraining.rs,
training_guard_gpu_tests.rs, CPU PER buffer paths, CPU Q-diagnostics
- Replace unwrap_or(Device::Cpu) with hard errors everywhere
- Remove dead is_cuda() else branches in DQN/PPO/hyperopt trainers
- Change config defaults from "cpu" to "cuda" (rainbow, tlob, pipeline)
- Port IQL value network to GPU kernel (5 CUDA entry points)
- Port HER goal relabeling to GPU kernel (warp-per-sample)
- Wire DSR GPU-to-CPU sync in training loop
- cfg!(feature = "cuda") → true in inference_validator
Zero warnings, zero errors across entire workspace.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move TFT, Mamba-2, Liquid, TGGN, TLOB, KAN, xLSTM, and Diffusion model
implementations to ml-supervised. Bridge files (UnifiedTrainable adapters,
Checkpointable impls) stay in ml. Delete AsyncDataLoader (replaced by
StreamingDbnLoader + simple .chunks() batching). Remove empty ml-infra
scaffold — the remaining ml modules are too tightly coupled for clean
extraction, so ml stays as the orchestration facade.
- ml-supervised: 234 tests, 0 failures
- ml: 1687 tests, 0 failures
- Workspace: 0 compilation errors
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move 24 PPO source files + flow_policy/ from ml into standalone
ml-ppo crate. The ml crate's ppo module is now a thin re-export layer
(`pub use ml_ppo::*`) plus two bridge files (trainable_adapter.rs,
stress_testing.rs) that depend on ml-internal types.
Key changes:
- ml-ppo: 25 modules (incl flow_policy subdir), 198 tests, standalone
- ml: depends on ml-ppo, re-exports via ppo/mod.rs
- Import rewrites: crate::common::action → ml_core::action_space,
crate::dqn::{mixed_precision,xavier_init} → ml_core::*,
crate::gradient_accumulation → ml_core::gradient_accumulation
- ml tests: 1929 pass (down from 2127 — 198 moved to ml-ppo)
- Workspace: 0 errors
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move 53 DQN source files + gpu_replay_buffer from ml into standalone
ml-dqn crate. The ml crate's dqn module is now a thin re-export layer
(`pub use ml_dqn::*`) plus two bridge files (trainable_adapter.rs,
stress_testing.rs) that depend on ml-internal types.
Key changes:
- ml-dqn: 45 modules, 334 tests, compiles standalone
- ml: depends on ml-dqn, re-exports via dqn/mod.rs
- DQN.config: pub(crate) → pub for cross-crate access
- DQNAgent checkpoint methods moved into ml-dqn
- gpu_replay_buffer moved from cuda_pipeline/ to ml-dqn
- 8 dead-code files removed (never declared in mod.rs)
Net: -30,648 lines from ml crate. Workspace: 0 errors.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move Sharpe ratio metrics and SIMD performance module to ml-core.
These only depend on MLError which is already in ml-core.
Add approx = "0.5" to ml-core dev-dependencies for Sharpe tests.
Skip observability/ (needs prometheus dep — stays in ml).
Test counts: 271 ml-core + 2487 ml = 2758 total, 0 failures
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move 6 shared modules (~2.3K lines) from ml to ml-core:
- trading_action.rs (TradingAction enum)
- action_space.rs (action masking, re-exports)
- xavier_init.rs (Xavier/Glorot weight initialization)
- mixed_precision.rs (AMP utilities, FP16/BF16)
- order_router.rs (deterministic order routing)
- portfolio_tracker.rs (portfolio state tracking)
With TradingAction now in ml-core alongside FactoredAction, the
FactoredActionExt extension trait is eliminated entirely —
from_trading_action()/to_trading_action() become inherent methods
on FactoredAction. This removes the need for `use FactoredActionExt`
imports in reward.rs, gae.rs, and trajectories.rs.
Test counts: 250 ml-core + 2508 ml = 2758 total, 0 failures
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move optimizers, gradient_accumulation, gradient_utils, cuda_compat,
tensor_ops, and gpu (device config, capabilities, memory profiling) to
ml-core. These are shared compute primitives used by all models.
Also commit module files for core types (common, config, error, model,
traits, types) that were moved from ml to ml-core in task 5a but left
staged without being committed.
Notable changes:
- resolve_batch_size() stays in ml (new batch_size_resolver module)
because it depends on memory_optimization::auto_batch_size which
has not yet moved to ml-core
- FactoredAction legacy bridge converted from inherent impl to
extension trait (FactoredActionLegacy) since FactoredAction is now
defined in ml-core, not ml
- candle-optimisers added to ml-core dependencies (needed by Adam)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move all inline type definitions from ml/src/lib.rs to ml-core:
MLError, MLResult, Trade, MarketRegime, HealthStatus, Features,
MLModel trait, ModelRegistry, ParallelExecutor, LatencyOptimizer,
TrainingMetrics, ValidationMetrics, InferenceResult, ModelMetadata.
Dedup: consolidate ConfigError{reason}/ConfigurationError(msg) into
single ConfigError(String) tuple variant (was 2 variants, 174 refs).
Cleanup: convert create_hft_* free functions to associated methods
(HFTPerformanceProfile::ultra_low_latency(), ParallelExecutor::hft()).
ml facade re-exports via `pub use ml_core::*`.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Re-export HardwareTimestamp through data crate instead of ml/risk
depending directly on trading_engine. Reduces coupling between
the ML pipeline and the trading engine.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Copied from api_gateway, removed REST handlers (port 8080),
added tonic-web + CORS for grpc-web browser access.
Binary renamed: api-gateway → api
Changes:
- Package name: api-gateway → api
- Deleted src/handlers/ (REST ML endpoints on port 8080)
- Added tonic-web 0.13 + tower-http CORS layer
- Server::builder().accept_http1(true) for grpc-web
- CORS_ORIGINS env var (default http://localhost:5173)
- Metrics server on port 9091 (axum) preserved
- All 95 lib tests pass, 0 clippy warnings
- Added services/api to workspace members
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rename 7 service binaries from snake_case to kebab-case to match
K8s deployment names. Update Cargo.toml package/bin names, K8s
manifest S3 paths and commands, and cross-crate dependency keys.
- api_gateway → api-gateway
- trading_service → trading-service
- broker_gateway_service → broker-gateway
- ml_training_service → ml-training-service
- backtesting_service → backtesting-service
- trading_agent_service → trading-agent-service
- data_acquisition_service → data-acquisition-service
broker-gateway gets an explicit [lib] name = "broker_gateway_service"
since its new package name maps to broker_gateway (not the original
broker_gateway_service used in source code). All other services map
correctly with Rust's automatic hyphen-to-underscore conversion.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Change PercentileScaler, RewardNormalizer, CompositeReward, and
PPORewardShaper public APIs from f64 to f32 — matching what callers
actually pass. Internal EMA/percentile math stays f64 for precision.
Keep data_loader SMA/EMA/MACD accumulators in f64 throughout (was
f64→f32→f64 per step), cast to f32 only at output boundary.
Remove dead _transition computation in rainbow_agent_impl and unused
bigdecimal deps from broker_gateway_service and trading_service.
2704 tests pass, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
fxt 2.0 overhaul: consolidated proto/, absorbed monitoring into API Gateway,
new 15-command CLI with gRPC, MCP server mode, TUI cockpit framework.
159 files changed, net -11,835 lines.
Embed DQNConfig architecture hash (SHA-256 of state_dim, num_actions,
hidden_dims, dueling/distributional/noisy/IQN flags) in safetensors
file header on save. Validate hash on load to catch shape mismatches
before touching the VarMap - prevents silent corruption from loading
checkpoints trained with different network architectures.
All 5 save paths (trainable_adapter, trainer serialize_model,
DQNAgentType::save_checkpoint, RegimeConditionalDQN per-head) now
embed metadata. All 3 load paths (DQN::load_from_safetensors,
trainable_adapter::load_checkpoint, ensemble adapter) validate.
No backward compatibility: checkpoints without metadata are rejected
with a clear error message to re-train.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove from workspace members, delete K8s deployment, remove nginx routing.
Training metrics and system health now served directly by API Gateway.
Changes:
- Cargo.toml: remove services/monitoring_service from workspace members
- services/monitoring_service/: deleted entirely (6 files)
- infra/k8s/services/monitoring-service.yaml: deleted
- infra/k8s/network-policies/monitoring-service.yaml: deleted
- infra/k8s/network-policies/api-gateway.yaml: remove egress rule to monitoring-service
- infra/k8s/network-policies/web-gateway.yaml: remove egress rule to monitoring-service
- infra/k8s/services/api-gateway.yaml: remove stale MONITORING_SERVICE_URL env var
- infra/k8s/gitlab/tailscale-proxy.yaml: redirect monitor.fxhnt.ai to api-gateway:50051
- .gitlab-ci.yml: remove monitoring_service from compile, copy, and deploy loops
- services/trading_service/src/services/monitoring.rs: update stale comments to
reference api_gateway (not monitoring_service) as training metrics host
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- trade_ml.rs: Replace 3 mock data fallbacks (submit, predictions,
performance) with proper error propagation. Commands now fail
honestly when the API Gateway is unreachable instead of silently
returning fake data. Mark 3 integration tests as #[ignore].
- monitoring_service: Add tonic-health with set_serving for
MonitoringServiceServer. Enables grpc_health_probe readiness checks.
- ml_training_service: Add tonic-health with set_serving for
MlTrainingServiceServer. Wired into both TLS and non-TLS paths.
- data_acquisition_service: Add tonic-health with set_serving for
DataAcquisitionServiceServer.
- ml/cuda_streams: Fix pre-existing unused variable clippy warning.
All 8 services now have standard gRPC health checking enabled.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Prometheus client queries training/hyperopt/GPU/K8s metrics, gRPC service
exposes unary + server-streaming RPCs, 4 unit tests, zero clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Match the actual K8s broker-gateway service port discovered during
live validation of fxt broker check.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove entire trading_engine/src/compliance/ directory (9 files, 16,068 LOC)
and 18 associated test files (18,069 LOC). Comprehensive audit confirmed
zero external callers for all types (ISO 27001, SOX, MiFID II, best
execution, automated reporting). Remove unused cron dependency.
Independent compliance modules in risk/ and risk-data/ are preserved.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
OTLP fix: opentelemetry-otlp 0.27 bundled tonic 0.12 while the workspace
uses tonic 0.14, making with_channel() impossible (type mismatch). Upgrading
to otel-otlp 0.31 aligns both on tonic 0.14, enabling explicit Channel
construction that respects http:// scheme (no spurious TLS negotiation).
API migrations (otel 0.27→0.31):
- TracerProvider → SdkTracerProvider
- with_batch_exporter(exporter, runtime) → with_batch_exporter(exporter)
- Resource::new(vec![...]) → Resource::builder().with_service_name().build()
- global::shutdown_tracer_provider() removed (Drop-based shutdown)
- opentelemetry-otlp feature "tonic" → "grpc-tonic"
DCGM fix: remove runtimeClassName: nvidia from DaemonSet — Scaleway Kapsule
GPU pools use nvidia runtime as default containerd handler. The RuntimeClass
CRD is only created by the full GPU Operator, not the device plugin alone.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add a reusable GrpcMetricsLayer that can be applied to any tonic server
via Server::builder().layer() to automatically emit three Prometheus
metric families for every gRPC handler: started_total, handled_total
(with status code), and handling_seconds histogram.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Web-gateway routing:
- Point TRADING_SERVICE_URL at api-gateway (proto mismatch fix)
Web-gateway uses foxhunt.tli.TradingService proto but was connecting
directly to trading-service which implements trading.TradingService.
api-gateway already proxies Subscribe* → Stream* correctly.
GitLab KAS:
- Disable gitlab_kas in appConfig to stop sidekiq NotifyGitPushWorker
errors (KAS pod was already disabled but Rails still tried to connect)
Trading service monitoring (3 stubs → real):
- AcknowledgeAlert: real alert lookup + state mutation in shared store
- GetActiveAlerts: returns actual active alerts from in-memory store
- StreamAlerts: now persists generated alerts (capped at 1000 entries)
Trading service ML streams (2 stubs → real):
- StreamModelMetrics: emits real inference_count, error_count, latency
per model every N seconds from the RuntimeModelInfo registry
- StreamSignalStrength: emits per-symbol signal aggregation from model
ensemble weights and latency confidence
Backtesting service:
- stop_backtest: real CancellationToken cancellation (was no-op)
Tokens stored per-backtest, execute_backtest wraps strategy call
in tokio::select! for immediate cancellation
Deleted 7 empty placeholder files:
- 4 Wave D regime stubs (dynamic_stops, ensemble, performance_tracker,
position_sizer) — comment-only files, never wired
- 2 Wave 3 feature stubs (microstructure, statistical)
- 1 PPO stub (unified_ppo.rs — empty struct definitions)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace all `.to_string().parse::<f32/f64>()` patterns with
`num_traits::ToPrimitive` methods (`.to_f32()`, `.to_f64()`).
Each string roundtrip heap-allocated per conversion — fatal in
DQN hot loop (300K+ bars × epochs). Decimal stays as canonical
financial type; conversions happen at GPU/float boundaries only.
Also fixes blocking_read() in async context (risk_integration.rs).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
