Commit Graph

11 Commits

Author SHA1 Message Date
jgrusewski
09c515e3e9 fix(clippy): ZERO errors across entire workspace — CI ready
Final 31 ml crate fixes: unsafe_code allows, unused vars prefixed,
boolean simplification, dead code removal, integer suffix, drop cleanup.

cargo fix auto-removed ~30 unused imports from ml crate.

Total clippy cleanup: 278 errors → 0 across all ML crates.
Full workspace: `cargo clippy --workspace --lib -- -D warnings` = 0 errors.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 01:04:09 +01:00
jgrusewski
c89a1dbf29 fix(ml): migrate remaining files — hyperopt, validation, data_loaders, inference
25+ files fixed:
- hyperopt/adapters: duplicate Arc imports, Device→MlDevice
- validation: regime_analysis rewritten for GpuTensor, ppo_adapter NativeDType
- data_loaders: all 3 loaders migrated to GpuTensor::from_host
- tft/training: MlDevice, GpuAdamW, StreamTensor, CPU loss tracking
- training_pipeline: AdamW→GpuAdamW, NativeDevice fixes
- features/multi_timeframe: removed to_candle_tensor
- inference: ModelForward trait replaces candle_nn::Module
- lib.rs: removed cuda_compat re-export
- ppo/mod.rs: fixed re-export

~95 errors remain in: inference.rs, flash_attention, validation/ppo_adapter,
hyperopt adapter method signatures (blocked on trainable adapter trait).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-18 09:57:24 +01:00
jgrusewski
0d62bdba2e refactor(ml): eliminate candle from all 104 src files
Zero candle_core/candle_nn imports in ml/src/. Three-agent parallel migration:

- cuda_pipeline/ (19 files): Tensor→CudaSlice, Device→Arc<CudaStream>,
  VarMap→GpuVarStore, cudarc import path fixed
- trainers/ + adapters (45 files): DQN/PPO/TFT trainers, 10 ensemble
  adapters, 11 hyperopt adapters — all migrated to MlDevice, GpuTensor,
  GpuVarStore, GpuAdamW
- model dirs + infra (40 files): 10 trainable adapters, preprocessing,
  inference, transformers, validation, benchmarks

61 test/example files still reference candle — next commit.
candle-nn still in Cargo.toml (needed by tests until migrated).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-18 00:24:35 +01:00
jgrusewski
450c23a6d0 refactor(cuda): eliminate all CPU fallbacks — CUDA mandatory across ML stack
- Remove ALL #[cfg(feature = "cuda")] guards (~400+ occurrences)
- Remove ALL #[cfg_attr(not(feature = "cuda"), ignore)] test annotations (~250)
- Make cuda default feature in 9 ML crates (ml, ml-core, ml-dqn, ml-ppo, etc.)
- Convert nvrtc JIT compilation to precompiled nvcc (searchsorted, prefix_sum)
- Move compile_ptx_for_device() to ml-core for shared access
- Delete dead CPU code: multi_step.rs, self_supervised_pretraining.rs,
  training_guard_gpu_tests.rs, CPU PER buffer paths, CPU Q-diagnostics
- Replace unwrap_or(Device::Cpu) with hard errors everywhere
- Remove dead is_cuda() else branches in DQN/PPO/hyperopt trainers
- Change config defaults from "cpu" to "cuda" (rainbow, tlob, pipeline)
- Port IQL value network to GPU kernel (5 CUDA entry points)
- Port HER goal relabeling to GPU kernel (warp-per-sample)
- Wire DSR GPU-to-CPU sync in training loop
- cfg!(feature = "cuda") → true in inference_validator

Zero warnings, zero errors across entire workspace.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 21:01:28 +01:00
jgrusewski
b4178952d4 fix(ml): BF16/F32 boundary alignment, GPU-resident ops across all ML crates
- Cast input to weight dtype in DQN residual, rmsnorm, noisy_layers
- Set use_gpu=true in QNetworkConfig defaults and all config sites
- Resolve BF16 boundary mismatches in attention, curiosity, branching,
  distributional_dueling across ml-dqn
- GPU-resident regime ops with BF16 boundary casts, eliminate .expect() in CUDA paths
- Eliminate all Device::Cpu fallbacks — GPU-only across 10 ML crates
- PPO: cast logits to F32 before softmax, cast batch tensors to training dtype
- Gradient collapse detection for RegimeConditionalDQN
- Wire halt_grad_collapse from CUDA guard kernel to halt training
- Dead neuron detection uses active network VarMap + squeeze factored readback
- Increment gradient_logging_step in GPU PER path
- Gradient collapse warmup guards use original buffer_size
- Cap training steps per epoch + tracing migration
- Replace Tensor::all() with sum_all() for pinned Candle compatibility

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-15 11:59:31 +01:00
jgrusewski
e4870b17b9 fix: tune log levels across workspace — demote noisy warn to debug/trace
Reduce log noise for non-critical operational paths: connection retries,
expected fallbacks, graceful degradation, and optional feature absence.
Keeps warn/error for genuine failures requiring attention.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-14 11:35:15 +01:00
jgrusewski
c0c44a5f17 feat(dqn): GPU-native regime classification with 42-dim feature vector
Expand FeatureVector from 40 to 42 dimensions by including ADX(14) at
index 40 and CUSUM direction at index 41 from the existing CPU feature
extraction pipeline. This eliminates proxy-based regime classification
and enables GPU-native regime detection via tensor narrow/comparison ops.

Key changes:
- extraction.rs: wire RegimeADXFeatures + RegimeCUSUMFeatures into
  extract_current_features_v2(), output 42 features per bar
- regime_conditional.rs: classify_regime_masks_gpu() creates per-regime
  mask tensors entirely on GPU (ADX > 0.25 = trending, |CUSUM| > 0.7 =
  volatile, else ranging). Zero CPU roundtrip in training hot path.
- trainer.rs/config.rs: state_dim 43→45 (no OFI), 51→53 (with OFI),
  aligned dims unchanged (48/56). GPU batch insertion for all 3 heads.
- CUDA header: MARKET_DIM 40→42
- walk_forward.rs: FEATURE_DIM 40→42
- 42 files updated, all [f64;40]→[f64;42] propagated across workspace

Test results: ml=874/0, ml-dqn=354/0, ml-features=282/0, ml-core=274/0
Real data GPU smoke tests: 7/7 passed (OHLCV + OFI + trade enrichment)
Hyperopt baseline RL: 2 trials completed on local RTX 3050 Ti

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 12:16:05 +01:00
jgrusewski
c6c550a2be feat(ml): real trade data pipeline for VPIN/Kyle's Lambda + offline RL
Wire Databento Schema::Trades into OFI feature extraction so VPIN and
Kyle's Lambda use real buy/sell classification instead of tick-rule proxy.

Trade data pipeline:
- trades_loader.rs: DbnTrade struct, load_trades_sync(), binary-search
  get_trades_for_bar() for O(log n) time-aligned trade windowing
- ofi_calculator.rs: feed_trade() accumulates real buy/sell pressure
  into VPIN, Kyle's Lambda, and trade imbalance calculators
- data_loading.rs: loads trades from --trades-data-dir, feeds per-bar
  trades to OFI calculator before calculate()
- download-trades-job.yaml: K8s job for ES.FUT trades from Databento
- job-template.yaml: sync trades data from MinIO + --trades-data-dir arg

Offline RL (CQL/IQL):
- experience_dataset.rs: bincode save/load for pre-collected datasets
- iql.rs: Implicit Q-Learning (Kostrikov 2021) — expectile value network,
  advantage-weighted action extraction
- CLI: --offline, --dataset-path, --collect-dataset flags

Cleanup:
- Remove FeatureVector51/MarketFeatureVector type aliases → FeatureVector
- Fix stale dimension comments across 18 files (54→43/51)
- Fix feature_dim default (54→43)

2758 tests pass, 0 compile errors, 0 clippy warnings.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 19:14:46 +01:00
jgrusewski
a35a564f45 fix(ml): DQN hyperopt overhaul — DSR reward, dead features, C51/noisy/network fixes
22-task overhaul (6 phases) for DQN training quality:
- Differential Sharpe Ratio (DSR) reward replacing raw PnL
- Remove 11 dead features (3 regime + 8 OFI): state_dim 54→43, feature_dim 51→40
- C51 v_min/v_max aligned to DSR Q-value range (±25.0)
- IQN batch path fix — consistent network for train+inference
- RMSNorm in distributional-dueling network
- Noisy layer sigma_init wired through config
- Rainbow config unified with DSR/C51/noisy defaults
- All dimension constants, comments, CUDA buffers updated
- 2747 lib tests passing, 0 failures, 0 warnings

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 12:08:39 +01:00
jgrusewski
cc4e0c5a2d refactor: remove trading_engine dep from ml and risk crates
Re-export HardwareTimestamp through data crate instead of ml/risk
depending directly on trading_engine. Reduces coupling between
the ML pipeline and the trading engine.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 23:19:38 +01:00
jgrusewski
9c3d741a08 refactor: restructure repo — crates/, bin/, testing/ layout
Move 17 library crates into crates/, CLI binary into bin/fxt,
consolidate 10 test crates into testing/, split config crate
from deployment config files.

Root directory reduced from 38+ to ~17 directories.
All Cargo.toml paths and build.rs proto refs updated.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 11:56:00 +01:00