Commit Graph

6 Commits

Author SHA1 Message Date
jgrusewski
9bee8e9483 fix: data_loader finds futures-baseline + mamba2 benchmark #[ignore]
1. RealDataLoader::new_from_workspace() now searches multiple paths:
   - test_data/futures-baseline (primary, where data actually lives)
   - data/cache/futures-baseline (CI cache)
   - test_data/real/databento (legacy path)
   Fixes 4 data_loader test failures from wrong path assumption.

2. test_full_mamba2_benchmark marked #[ignore] — runs 2+ epochs of
   Mamba2 training, too slow for unit test suite.

Final test results:
  ml: 855 passed, 0 failed, 11 ignored (13.5s)
  ml-core: 302 passed, 0 failed (3.3s)
  ml-dqn: 359 passed, 0 failed (0.8s)
  ml-ppo: 168 passed, 0 failed (0.3s)
  smoke_e2e: PASSED (0.66s)
  clippy: 0 errors

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 11:38:52 +01:00
jgrusewski
05aa19ffe9 fix: unignore 14 tests — local DBN data, nvidia-smi, GPU benchmarks
Removed #[ignore] from tests that have local infrastructure:
- 3 data_loader tests: auto-detect test_data/real/databento/ via workspace
- 3 memory_profiler tests: nvidia-smi at /usr/bin/nvidia-smi
- 4 benchmark tests (TFT, Mamba2, DQN, PPO): GPU + DBN data available
- 1 inference test: model loading (slow but should run)
- 3 DQN performance smoke tests: GPU available

PPO benchmark: fixed data_path to test_data/real/databento/6E.FUT
Sequential: added vars_mut() accessor

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-18 15:54:26 +01:00
jgrusewski
b4178952d4 fix(ml): BF16/F32 boundary alignment, GPU-resident ops across all ML crates
- Cast input to weight dtype in DQN residual, rmsnorm, noisy_layers
- Set use_gpu=true in QNetworkConfig defaults and all config sites
- Resolve BF16 boundary mismatches in attention, curiosity, branching,
  distributional_dueling across ml-dqn
- GPU-resident regime ops with BF16 boundary casts, eliminate .expect() in CUDA paths
- Eliminate all Device::Cpu fallbacks — GPU-only across 10 ML crates
- PPO: cast logits to F32 before softmax, cast batch tensors to training dtype
- Gradient collapse detection for RegimeConditionalDQN
- Wire halt_grad_collapse from CUDA guard kernel to halt training
- Dead neuron detection uses active network VarMap + squeeze factored readback
- Increment gradient_logging_step in GPU PER path
- Gradient collapse warmup guards use original buffer_size
- Cap training steps per epoch + tracing migration
- Replace Tensor::all() with sum_all() for pinned Candle compatibility

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-15 11:59:31 +01:00
jgrusewski
7ab5d4fa71 fix(ml): BUG #40 — epsilon stuck at 1.0 with noisy nets (100% random actions)
When use_noisy_nets=true (the conservative() default), epsilon never
decayed from 1.0 because (1) the trainer skipped update_epsilon() and
(2) DQNAgentType::set_epsilon() was a no-op for RegimeConditional agents.
This caused ALL training actions to be random — Q-values were learned but
never used for action selection.

Fix: set stored epsilon to 0.0 at training start when noisy nets are on.
Exploration is provided by NoisyLinear weight perturbation + the separate
noisy_epsilon_floor (5% safety floor for 45-action spaces).

Also fixes:
- Zstd-compressed .dbn file detection via magic bytes (0x28B52FFD)
- Test data path resolution using ancestors().find() for workspace root
- Test assertions updated for epsilon < 0.01 with noisy nets

2698 lib tests pass, 0 clippy warnings.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 23:13:25 +01:00
jgrusewski
93104e8545 refactor(ml): eliminate f64↔f32 round-trips and dead deps across ML crate
Change PercentileScaler, RewardNormalizer, CompositeReward, and
PPORewardShaper public APIs from f64 to f32 — matching what callers
actually pass. Internal EMA/percentile math stays f64 for precision.

Keep data_loader SMA/EMA/MACD accumulators in f64 throughout (was
f64→f32→f64 per step), cast to f32 only at output boundary.

Remove dead _transition computation in rainbow_agent_impl and unused
bigdecimal deps from broker_gateway_service and trading_service.

2704 tests pass, 0 clippy warnings.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 18:10:04 +01:00
jgrusewski
2fbb19d9df refactor(ml): rename real_data_loader to data_loader
The real_ prefix was misleading — there is no fake data loader.
Mechanical rename across 18 source files, no logic changes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 15:41:23 +01:00