- cudarc features: removed "nvrtc" (no longer used, all kernels precompiled)
- cudarc features: added "f16" (enables DeviceRepr + ValidAsZeroBits for half::bf16)
- CudaSlice<half::bf16> now fully functional with all cudarc operations
- Foundation for full BF16 tensor core conversion (next session)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Final cleanup:
- 61 test files + 5 example files: candle imports replaced
- 8 testing/integration files: migrated to cudarc/ml-core types
- 3 services/trading_service test files: migrated
- Root Cargo.toml: candle-core, candle-nn removed from [workspace.dependencies]
- crates/ml/Cargo.toml: candle-nn dependency removed
- testing/e2e/Cargo.toml: candle-core dependency removed
Zero active candle_core/candle_nn/candle_optimisers code references remain.
Zero candle dependency declarations in any Cargo.toml.
Remaining "candle" strings are exclusively in doc comments.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove ALL #[cfg(feature = "cuda")] guards (~400+ occurrences)
- Remove ALL #[cfg_attr(not(feature = "cuda"), ignore)] test annotations (~250)
- Make cuda default feature in 9 ML crates (ml, ml-core, ml-dqn, ml-ppo, etc.)
- Convert nvrtc JIT compilation to precompiled nvcc (searchsorted, prefix_sum)
- Move compile_ptx_for_device() to ml-core for shared access
- Delete dead CPU code: multi_step.rs, self_supervised_pretraining.rs,
training_guard_gpu_tests.rs, CPU PER buffer paths, CPU Q-diagnostics
- Replace unwrap_or(Device::Cpu) with hard errors everywhere
- Remove dead is_cuda() else branches in DQN/PPO/hyperopt trainers
- Change config defaults from "cpu" to "cuda" (rainbow, tlob, pipeline)
- Port IQL value network to GPU kernel (5 CUDA entry points)
- Port HER goal relabeling to GPU kernel (warp-per-sample)
- Wire DSR GPU-to-CPU sync in training loop
- cfg!(feature = "cuda") → true in inference_validator
Zero warnings, zero errors across entire workspace.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The ml crate had `default = ["minimal-inference", "cuda"]` which pulled
in cudarc/candle-kernels requiring nvcc. CI test-gate runs on
ci-builder-cpu (no CUDA) so `cargo clippy --workspace` always panicked
with "Failed to execute nvcc: No such file or directory".
CUDA is now opt-in only — compile-and-train template already passes
`--features ml/cuda` explicitly for GPU builds.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three fixes for GPU PER hot path:
1. IQN quantile loss used empty CPU weights Vec instead of GPU-resident
weights_tensor_cached — caused CUDA_ERROR_ILLEGAL_ADDRESS from
uninitialized GPU memory. Now uses cached GPU tensor matching C51
and standard DQN paths.
2. GpuPrioritized add()/add_batch() replaced with StagedGpuBuffer:
add() stages on CPU (Vec::push, zero GPU ops), sample() batch-flushes
staging→GPU in one DMA before sampling. Production path (insert_batch_tensors)
bypasses staging entirely — GPU→GPU with zero CPU.
3. All 9 ML sub-crates default to cuda feature so `cargo test -p ml-dqn`
exercises GPU code paths on CUDA workstations. CI service crates use
default-features=false, unaffected.
Test results: 350 passed (was 343+7 failed), 0 failed, 1 ignored.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ml-supervised, ml-ensemble, ml-labeling, ml-explainability, ml-hyperopt
all had default = ["cuda"] which pulled cudarc into the CPU services
build, causing compile-services to fail with "nvcc not found".
Changed all to default = [] and wired ml/Cargo.toml cuda feature to
propagate to all 8 sub-crates (was only 3: ml-core, ml-dqn, ml-ppo).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove cuda from default features in ml-core, ml-dqn, ml-ppo, ml
- Propagate cuda feature from ml → ml-core/ml-dqn/ml-ppo
- CI compile-training already uses --features ml/cuda explicitly
- Fix MaxDD log format: {:.1}% → {:.3}% (was rounding 0.033% to 0.0%)
- Suppress unused_labels/unused_variables warnings for cfg(cuda) code
- Add CALLBACK_ENDPOINT env to ml-training-service deployment
- Fix Grafana active_workers query to use sum() with fallback
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move TFT, Mamba-2, Liquid, TGGN, TLOB, KAN, xLSTM, and Diffusion model
implementations to ml-supervised. Bridge files (UnifiedTrainable adapters,
Checkpointable impls) stay in ml. Delete AsyncDataLoader (replaced by
StreamingDbnLoader + simple .chunks() batching). Remove empty ml-infra
scaffold — the remaining ml modules are too tightly coupled for clean
extraction, so ml stays as the orchestration facade.
- ml-supervised: 234 tests, 0 failures
- ml: 1687 tests, 0 failures
- Workspace: 0 compilation errors
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move 24 PPO source files + flow_policy/ from ml into standalone
ml-ppo crate. The ml crate's ppo module is now a thin re-export layer
(`pub use ml_ppo::*`) plus two bridge files (trainable_adapter.rs,
stress_testing.rs) that depend on ml-internal types.
Key changes:
- ml-ppo: 25 modules (incl flow_policy subdir), 198 tests, standalone
- ml: depends on ml-ppo, re-exports via ppo/mod.rs
- Import rewrites: crate::common::action → ml_core::action_space,
crate::dqn::{mixed_precision,xavier_init} → ml_core::*,
crate::gradient_accumulation → ml_core::gradient_accumulation
- ml tests: 1929 pass (down from 2127 — 198 moved to ml-ppo)
- Workspace: 0 errors
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move 53 DQN source files + gpu_replay_buffer from ml into standalone
ml-dqn crate. The ml crate's dqn module is now a thin re-export layer
(`pub use ml_dqn::*`) plus two bridge files (trainable_adapter.rs,
stress_testing.rs) that depend on ml-internal types.
Key changes:
- ml-dqn: 45 modules, 334 tests, compiles standalone
- ml: depends on ml-dqn, re-exports via dqn/mod.rs
- DQN.config: pub(crate) → pub for cross-crate access
- DQNAgent checkpoint methods moved into ml-dqn
- gpu_replay_buffer moved from cuda_pipeline/ to ml-dqn
- 8 dead-code files removed (never declared in mod.rs)
Net: -30,648 lines from ml crate. Workspace: 0 errors.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move all inline type definitions from ml/src/lib.rs to ml-core:
MLError, MLResult, Trade, MarketRegime, HealthStatus, Features,
MLModel trait, ModelRegistry, ParallelExecutor, LatencyOptimizer,
TrainingMetrics, ValidationMetrics, InferenceResult, ModelMetadata.
Dedup: consolidate ConfigError{reason}/ConfigurationError(msg) into
single ConfigError(String) tuple variant (was 2 variants, 174 refs).
Cleanup: convert create_hft_* free functions to associated methods
(HFTPerformanceProfile::ultra_low_latency(), ParallelExecutor::hft()).
ml facade re-exports via `pub use ml_core::*`.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add QuestDB ILP sink for training metrics, update Prometheus scrape
configs, and fix network policies for monitoring stack connectivity.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Re-export HardwareTimestamp through data crate instead of ml/risk
depending directly on trading_engine. Reduces coupling between
the ML pipeline and the trading engine.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Embed DQNConfig architecture hash (SHA-256 of state_dim, num_actions,
hidden_dims, dueling/distributional/noisy/IQN flags) in safetensors
file header on save. Validate hash on load to catch shape mismatches
before touching the VarMap - prevents silent corruption from loading
checkpoints trained with different network architectures.
All 5 save paths (trainable_adapter, trainer serialize_model,
DQNAgentType::save_checkpoint, RegimeConditionalDQN per-head) now
embed metadata. All 3 load paths (DQN::load_from_safetensors,
trainable_adapter::load_checkpoint, ensemble adapter) validate.
No backward compatibility: checkpoints without metadata are rejected
with a clear error message to re-train.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- MultiGpuConfig::detect() probes CUDA ordinals 0..8, returns None
on single-GPU/CPU setups
- shard_indices() splits dataset across devices with remainder handling
- NcclGradientSync (behind `nccl` feature flag) for all-reduce averaging
- Feature: `nccl = ["cuda"]` — requires NCCL library on system
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Split the build pipeline: one compile-services job builds all 8 service
binaries with PVC-backed sccache, saves as artifacts. Then 9 Kaniko jobs
just package pre-built binaries into slim runtime images (~30s each).
Before: 9 parallel Kaniko jobs each doing full cargo build --release
(~20min each, no sccache, 9x duplicated dep compilation)
After: 1 compile job with sccache (~5min cached) + 9 package jobs (~30s)
- Add compile stage between test and build
- Add Dockerfile.runtime (minimal debian + pre-built binary)
- Add Dockerfile.web-gateway-runtime (Node dashboard + pre-built binary)
- Keep Dockerfile.training via Kaniko (needs CUDA dev image for H100)
- Remove all SCCACHE_BUCKET build-args from service builds
- Use dir:// context for Kaniko (only sends build-out/ dir, not full repo)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Mass cleanup of crates/ml/:
- Delete 121 dead/broken/superseded example files (-44,098 lines)
- Delete 6 QAT test files (-4,201 lines) — QAT is disabled at runtime
(trainer.rs falls back to FP32 with warning)
- Delete 2 stale markdown files in examples/
- Delete orphaned src/bin/train_tft.rs (unimplemented stub)
- Clean up Cargo.toml: remove stale [[example]] entries, add missing ones
- Fix stale binary references in log_size_test.rs
- Add infra consistency test (35 checks across Dockerfile/train.sh/Cargo.toml)
Remaining examples (7): train_baseline_rl, train_baseline_supervised,
evaluate_baseline, hyperopt_baseline_rl, hyperopt_baseline_supervised,
download_baseline, cuda_test
All 2390 lib tests pass. All examples compile. 35/35 infra checks pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move 17 library crates into crates/, CLI binary into bin/fxt,
consolidate 10 test crates into testing/, split config crate
from deployment config files.
Root directory reduced from 38+ to ~17 directories.
All Cargo.toml paths and build.rs proto refs updated.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
