Commit Graph

2012 Commits

Author SHA1 Message Date
jgrusewski
befa7e2f55 perf(gpu): shared memory weight caching, GPU reductions, fix staging flush dim
CUDA kernel: cooperative_load_tile() + matvec_leaky_relu_shmem() cache weight
matrices in shared memory (SRAM), reducing HBM traffic 32x (1.76TB→55GB).
Block size increased to 256 threads for better SHMEM utilization.

Trainer hot-path: replace to_vec2 full-tensor CPU readbacks with GPU-native
flatten_all/min/max/mean_all (4 scalar readbacks). Merge two Q-diagnostic
forward passes into one. Deferred max_priority flush (1 GPU→CPU sync/epoch
instead of ~8/batch).

Fix StagedGpuBuffer::flush() dimension mismatch: used raw experience
state.len() (45) instead of gpu.state_dim() (48 aligned), causing
slice_scatter crash [500,48] vs [n,45] when CPU experiences flush into
GPU-aligned replay buffer.

Fix hardcoded state_dim=48 in optimal_n_episodes() — now uses dynamic
state_dim from network config (correct for OFI-enabled 56-dim states).

Dynamic episode auto-scaling guard: only auto-scale when gpu_n_episodes≥128
(production), respecting explicit test overrides (gpu_n_episodes=2).

Tests: 874 ml + 392 ml-dqn = 1266 pass, 0 fail.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 22:24:33 +01:00
jgrusewski
3865354d8f perf(gpu): dynamic episode scaling + BF16 cross-entropy stabilization
GPU experience collector:
- Add GpuHardwareInfo with SM count detection from device name lookup
  (H100=132, A100=108, L40S=142, RTX 4090=128, etc.)
- optimal_n_episodes() scales to GPU: sm_count × 2 warps × 32 threads,
  capped by 15% free VRAM budget, 256-aligned for block scheduling
- Remove hardcoded .min(256) cap in trainer — auto-scales from 128 to 8192
- MAX_EPISODES_LIMIT raised from 256 to 4096

Numerical stability:
- BF16 cross-entropy epsilon: 1e-8 → 1e-4 (below BF16 precision floor
  1e-8 rounds to zero, making log-stabilization a no-op → -inf → NaN)
- Add log_probs.clamp(-20, 0) guard against -inf × 0 = NaN in loss

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 21:16:47 +01:00
jgrusewski
356f1d295f fix(train): remove duplicate annotations block from rebase artifact
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 21:08:02 +01:00
jgrusewski
fb18e0f1dc fix(dqn): prevent BF16 softmax overflow, increase batch size for H100
BF16's 7-bit mantissa overflows on exp(50+) in softmax → Inf → NaN.
Cast distributional logits to F32 before softmax in both dueling and
rainbow network heads, then cast back to original dtype.

Increase default batch_size 128→1024 (standard), 128→256 (conservative),
512→2048 (aggressive) to saturate H100 tensor cores. AutoBatchSizer
already caps to VRAM ceiling for smaller GPUs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 21:05:42 +01:00
jgrusewski
e89fbc2b4d fix(metrics): wire training pod scraping to Grafana dashboard
Three root causes for "no metrics" on the training dashboard:

1. Dashboard template variables ($model, $fold) sourced from
   foxhunt_training_current_epoch which isn't emitted until the first
   epoch completes. Switch to foxhunt_training_step which fires from
   step 500 onward.

2. train.sh pod template missing Prometheus annotations
   (prometheus.io/scrape, port, path). Also add the
   app.kubernetes.io/component label to the eval manifest so
   evaluation pods are discoverable too.

3. DQN and PPO trainers only called set_epoch() at the END of each
   epoch. Move the call to the TOP of the epoch loop so the gauge
   exists from the first training iteration.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 21:02:29 +01:00
jgrusewski
9d6a04ab4a fix(grafana): add 20 missing metric panels to training dashboard
Remove phantom foxhunt_training_total_epochs reference (never registered)
and replace with foxhunt_training_step from the recent metrics commit.
Add full coverage for all 52 Tier-1 Prometheus metrics across 5 new rows:
RL diagnostics (Q-overestimation, PPO entropy/KL/advantage), training
health (NaN/grad explosion/feature errors, checkpoint ops, data load
latency), epoch returns, supervised eval (accuracy/precision/recall/F1),
and hyperopt details (mode, trial epoch, failed trials).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 21:02:29 +01:00
jgrusewski
47d4959588 fix(train): correct --output to --output-dir arg, add Prometheus annotations
The train_baseline_rl binary expects --output-dir, not --output.
Also adds prometheus.io scrape annotations to job pod templates
so training metrics appear in Grafana.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 20:53:36 +01:00
jgrusewski
ee12f1d9e4 perf(gpu): zero-sync gradient clipping — eliminate all GPU→CPU barriers from hot path
clip_grad_norm now returns a GPU-resident Tensor instead of (f64, f64),
keeping backward → clip → optimizer.step fully pipelined on GPU with
zero cuStreamSynchronize stalls. The unconditional multiply trick
(scale = min(max_norm/(norm+eps), 1.0)) avoids the conditional branch
that previously required reading the norm to CPU.

Key changes:
- gradient_utils::clip_grad_norm: return Tensor, unconditional GPU multiply
- Handle BF16 mixed-precision via per-gradient to_dtype cast
- adam.rs: backward_step_with_monitoring returns Tensor (zero sync)
- gradient_accumulation: delegate to gradient_utils (DRY, same GPU path)
- dqn.rs: single sync boundary after ALL GPU work queued

1746 tests passing (ml-core 286, ml-dqn 388, ml-ppo 198, ml 874).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 20:26:45 +01:00
jgrusewski
db65eb56a5 perf(gpu): eliminate GPU→CPU sync barriers from training hot path
clip_grad_norm: accumulate squared norms on GPU-resident scalar, single
to_scalar() at end (was 16-20 per-param syncs per step × 2917 steps/epoch).

check_gradients_finite: same GPU-accumulation pattern, single sync.

gpu_replay_buffer sample_proportional/rank_based: generate random targets
via rand(0,1)*total_sum on GPU, normalize weights via broadcast_div
(eliminates 2 to_vec0 syncs per sample call).

gpu_replay_buffer update_priorities_gpu: replace CPU loop of 50 individual
slice_scatter calls with single batched index_add delta trick.

dqn NaN detection (every 500 steps): accumulate 3 NaN counts on GPU,
single to_scalar for total; detailed breakdown only if NaN found.

dqn dead neuron detection (every 1000 steps): accumulate dead count on
GPU-resident scalar (was to_vec0 per parameter tensor, ~16-20 syncs).

Net: ~22 GPU→CPU syncs + 50 micro-kernels per training step → 2 syncs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 20:02:16 +01:00
jgrusewski
0355fb17bc perf(metrics): expose step-level DQN metrics to Prometheus gauges
Move Q-value stats, gradient norm, and training step from tracing::info!
(stdout-only, block-buffered, invisible mid-epoch) to Prometheus gauges
(atomic set, scrapable every 15s). Downgrade formatted step logs to
debug! level to eliminate allocation/serialization overhead in the hot path.

New gauge: foxhunt_training_step (current step within epoch)
Updated: foxhunt_training_q_value_mean/max, foxhunt_training_gradient_norm
now update every 500 steps instead of only at epoch boundaries.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 18:44:29 +01:00
jgrusewski
7751f7615a fix(ci): unblock CPU service builds and fix H100 BF16 regime classification
Three fixes validated by 20/20 hyperopt trials on H100 (zero OOM):

1. Workspace default-features: ml-core, ml-dqn, ml-ppo, ml-supervised
   workspace deps now have default-features=false. Prevents cudarc
   (which requires nvcc) from leaking into CPU service builds via
   Cargo feature unification. CI compile-services was failing with
   "Failed to execute nvcc: No such file or directory" (exit 101).

2. BF16 comparison fix: Candle's gt()/le() don't support BF16 operands.
   Cast ADX/CUSUM features to F32 before threshold comparison in
   regime classification. Previous approach (cast threshold to BF16)
   failed due to Candle broadcast_as reverting dtype.

3. CI pipeline: expand ML change detection to all 14 sub-crates,
   add component:compile labels for sccache network policy matching,
   bump training runtime to CUDA 12.6 + Ubuntu 24.04 (glibc 2.39).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 18:11:04 +01:00
jgrusewski
379c0bee37 fix(dqn): cast F32 constants to input dtype for BF16 mixed precision on H100
On H100 with BF16 mixed precision enabled, Candle's comparison ops (gt,
lt, le, ge) and arithmetic ops require matching dtypes. F32 threshold
constants compared against BF16 state tensors caused every training step
to fail with "dtype mismatch in cmp, lhs: BF16, rhs: F32".

Fixes:
- regime_conditional.rs: ADX/CUSUM thresholds cast to adx.dtype()
- quantile_regression.rs: Huber kappa, zero, and half tensors match
  input dtype (both quantile_huber_loss variants)
- rainbow_network.rs: LeakyReLU slope and ELU alpha/one tensors cast
  to x.dtype()
- continuous_ppo.rs: Huber delta/half tensors cast to abs_diff.dtype()

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 17:13:46 +01:00
jgrusewski
fe223b3843 fix(dqn): eliminate OOM in hyperopt by gating GPU features on small GPUs
Three root causes fixed:
1. GPU PER + experience collector (cudarc) created dual CUDA allocator
   fragmentation on GPUs ≤8 GB, making training impossible even at
   batch_size=1. Added `use_gpu_replay_buffer` config flag; both GPU PER
   and experience collector now disabled when VRAM ≤8192 MB.

2. Search space bounds were WIDENED instead of capped — max_batch_size
   returning 4096 replaced the original 512 upper bound, and
   max_hidden_dim_base_full returning 3072 replaced the original 1024.
   Fixed with min() to only narrow, never widen.

3. VRAM estimator assumed GPU features always active, overcharging when
   they're disabled on small GPUs. Now conditional: when
   replay_buffer_capacity=0 (proxy for GPU PER disabled), collector/cudarc
   costs are zero and fragmentation multiplier drops from 3× to 1.5×.

Additional small-GPU guard: GPUs ≤8 GB get clamped search space
(batch≤128, hidden≤512, atoms≤51, buffer≤50K) to fit 3 regime heads
+ C51 + noisy nets + dueling in limited VRAM.

Validated: 7/7 trials complete on RTX 3050 Ti 4 GB, zero OOM, best trial
Sharpe 9.8 with 50.6% win rate. Previous runs had 100% OOM failure rate.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 16:35:38 +01:00
jgrusewski
06d0a8d618 Merge branch 'feature/dqn-branching'
perf(dqn): GPU-native branching action decomposition (eliminates cudaSync stall)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 13:36:01 +01:00
jgrusewski
1b9f09fce1 perf(dqn): eliminate GPU→CPU roundtrip in branching action decomposition
Replace `actions_tensor.to_vec1()` → CPU loop → `Tensor::from_vec` with
GPU-native `decompose_actions_batch_gpu` using F32 affine+floor arithmetic.
Removes the last `cudaDeviceSynchronize` stall in the branching training
hot path.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 13:26:44 +01:00
jgrusewski
3a37454ea5 Merge branch 'feature/dqn-branching' 2026-03-09 13:17:50 +01:00
jgrusewski
89c3fb89d9 feat(dqn): Branching DQN with full GPU Rainbow parity (7 fixes)
Bring Branching Dueling Q-Network (Tavakoli 2018) to full Rainbow parity
with the existing GPU hotpath. 3 independent advantage heads (exposure=5,
order=3, urgency=3) decompose the 45-action space into learnable branches.

H1 - CUDA fallback: gate GpuExperienceCollector when use_branching=true
     (fused kernel hardcodes NUM_ACTIONS=5, incompatible with 45 factored)
H2 - Per-branch C51 distributional: each branch outputs [batch, n_d, atoms]
     log-softmax, loss = avg of D cross-entropies vs projected Bellman target
M1 - NoisyNet: MaybeNoisyLinear enum in branch heads, reset_noise/disable_noise
     wired through select_action, compute_loss, and set_eval_mode
M2 - Regime-conditional IS weights: Trending=1.2, Ranging=0.8, Volatile=0.6
     applied to branching loss via ADX/CUSUM features at state[40:41]
M3 - State dim alignment: align_dim_for_tensor_cores() in from_dqn_params()
     for H100 HMMA dispatch (8-byte alignment)
L1 - Fill simulator: splitmix64 replaces golden ratio hash (chi-squared tested)
L2 - Hyperopt 29D: branch_hidden_dim [64,256] added to PSO search space

Config plumbing: branch_hidden_dim, v_min/v_max/num_atoms, use_distributional,
use_noisy, noisy_sigma_init all flow from DQNConfig → BranchingConfig.

10 files, +3207/-125 lines, 33 branching tests + 387 ml-dqn + 284 ml-core pass.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 13:17:06 +01:00
jgrusewski
7f3066e695 feat(ml): enable BF16 mixed precision by default on CUDA
training_dtype() now returns BF16 on all CUDA devices, enabling full
tensor-core utilization. F32 is used only at boundaries (scalar
extraction, loss computation, softmax). VRAM estimator updated to
account for BF16 byte sizes, C51/QR atoms, dueling streams, NoisyNet
param doubling, and GPU PER buffer pre-allocation.

Changes:
- mixed_precision.rs: training_dtype() returns BF16 on CUDA
- curiosity.rs: F32 cast before scalar extraction
- network.rs: F32 output at NetworkLayers forward boundary
- traits.rs: estimate_trial_vram_mb_full() with BF16-aware sizing
- dqn.rs adapter: uses full estimator with worst-case architecture

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 12:48:55 +01:00
jgrusewski
3c3812d993 Merge branch 'worktree-gpu-hotpath-audit' 2026-03-09 12:17:05 +01:00
jgrusewski
c0c44a5f17 feat(dqn): GPU-native regime classification with 42-dim feature vector
Expand FeatureVector from 40 to 42 dimensions by including ADX(14) at
index 40 and CUSUM direction at index 41 from the existing CPU feature
extraction pipeline. This eliminates proxy-based regime classification
and enables GPU-native regime detection via tensor narrow/comparison ops.

Key changes:
- extraction.rs: wire RegimeADXFeatures + RegimeCUSUMFeatures into
  extract_current_features_v2(), output 42 features per bar
- regime_conditional.rs: classify_regime_masks_gpu() creates per-regime
  mask tensors entirely on GPU (ADX > 0.25 = trending, |CUSUM| > 0.7 =
  volatile, else ranging). Zero CPU roundtrip in training hot path.
- trainer.rs/config.rs: state_dim 43→45 (no OFI), 51→53 (with OFI),
  aligned dims unchanged (48/56). GPU batch insertion for all 3 heads.
- CUDA header: MARKET_DIM 40→42
- walk_forward.rs: FEATURE_DIM 40→42
- 42 files updated, all [f64;40]→[f64;42] propagated across workspace

Test results: ml=874/0, ml-dqn=354/0, ml-features=282/0, ml-core=274/0
Real data GPU smoke tests: 7/7 passed (OHLCV + OFI + trade enrichment)
Hyperopt baseline RL: 2 trials completed on local RTX 3050 Ti

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 12:16:05 +01:00
jgrusewski
d12273e164 fix(dqn): GPU PER monitoring funcs used empty CPU experiences, causing cuBLAS crash
Four monitoring functions (estimate_avg_q_value_with_early_stopping,
collect_qvalue_statistics, compute_q_gap_for_epoch, compute_per_action_q_values)
iterated over batch_sample.experiences to build CPU tensors for forward passes.
When GPU PER (GpuPrioritized) is active, experiences is always vec![] — all data
lives on GPU tensors in gpu_batch. This created zero-element tensors with non-zero
shapes, triggering CUBLAS_STATUS_INVALID_VALUE on the next forward pass.

Fix: all four functions now check for gpu_batch.states and use it directly,
falling back to CPU experiences only for non-GPU buffers. Also removes debug
eprintln probes, wires new_on_device for DQN/RegimeConditionalDQN construction,
and adds GPU-aware smoke tests (33 pass, 874 total, 0 failures).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 10:47:46 +01:00
jgrusewski
0f45537e6a perf(dqn): GPU searchsorted kernel for PER sampling via CustomOp2
Eliminates the CPU roundtrip in PER binary search: previously the
cumsum tensor (~400KB for 100K buffer) was downloaded to CPU, searched
in a loop, then indices uploaded back. Now a CUDA kernel runs one
thread per target with O(log n) binary search — zero DMA.

- Add searchsorted_kernel.cu (40-line CUDA binary search kernel)
- Implement SearchSorted as Candle CustomOp2 with CPU fallback
- Wire into sample_proportional() and sample_rank_based()
- Fix all clippy warnings in gpu_replay_buffer.rs (const fn, shadow,
  doc backticks, to_owned, div_ceil, module_name_repetitions)
- Fix wildcard_enum_match_arm in mixed_precision.rs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 08:33:28 +01:00
jgrusewski
5191fa022a perf(ml): replace remaining CPU tensor allocations with GPU-native ops
Eliminate all remaining from_vec(vec![const; N]) patterns and CPU
tensor roundtrips in the training hot path:

- gamma discount: from_vec → Tensor::full (DQN + distributional C51)
- C51 atoms: from_vec(computed Vec) → arange+affine, 4 instances
  across forward pass and C51 loss paths (online/standard/double-DQN)
- IQN cosine embedding indices: from_vec → arange+reshape
- IQN uniform quantiles: from_vec → arange+affine (τ_i = (i+0.5)/N)
- PPO clip epsilon: from_vec+ones+sub/add+tensor_clamp → scalar clamp
  (eliminates 6 intermediate tensor allocations, 2 call sites)
- Dead neuron detection: flatten+to_vec1+CPU loop → abs().le().sum_all()
  (single scalar per parameter tensor instead of full weight transfer)

548 tests pass (350 ml-dqn + 198 ml-ppo), ml crate compiles clean.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 23:22:39 +01:00
jgrusewski
801781a472 perf(ml): eliminate CPU tensor ops from GPU training hot path
Replace expensive CPU↔GPU data transfers with GPU-native operations:

- check_gradients_finite: sum_all() scalar readback (4B per tensor)
  replaces flatten_all()+to_vec1() that copied entire gradients to CPU
  (up to 200MB per step across 9 call sites in DQN+PPO)
- Huber loss: affine() replaces 3× Tensor::from_vec(vec![const; N])
  CPU Vec allocations in the inner loss computation loop
- update_priorities_gpu: per-index slice_scatter (~1KB DMA) replaces
  full-buffer to_vec1()+from_vec() roundtrip (1.2MB DMA per step)
- PER sampling: single from_vec + GPU to_dtype replaces duplicate
  from_vec calls and CPU type conversion roundtrips
- RegimeConditional: log warning on silent GPU batch drops

822 tests pass (350 ml-dqn, 274 ml-core, 198 ml-ppo), 0 clippy.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 23:12:54 +01:00
jgrusewski
b616d024ad fix(dqn): IQN GPU PER weights, staged GPU buffer, CUDA default in all ML crates
Three fixes for GPU PER hot path:

1. IQN quantile loss used empty CPU weights Vec instead of GPU-resident
   weights_tensor_cached — caused CUDA_ERROR_ILLEGAL_ADDRESS from
   uninitialized GPU memory. Now uses cached GPU tensor matching C51
   and standard DQN paths.

2. GpuPrioritized add()/add_batch() replaced with StagedGpuBuffer:
   add() stages on CPU (Vec::push, zero GPU ops), sample() batch-flushes
   staging→GPU in one DMA before sampling. Production path (insert_batch_tensors)
   bypasses staging entirely — GPU→GPU with zero CPU.

3. All 9 ML sub-crates default to cuda feature so `cargo test -p ml-dqn`
   exercises GPU code paths on CUDA workstations. CI service crates use
   default-features=false, unaffected.

Test results: 350 passed (was 343+7 failed), 0 failed, 1 ignored.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 22:31:49 +01:00
jgrusewski
e35ead5f3e fix(dqn): wire GPU PER into hot path, fix IS-weight ordering and IQN CVaR sort
- Wire GpuReplayBuffer into DQN constructor with OOM fallback to CPU PER
- Fix P0: GPU PER priorities never updated in single-batch train_step()
  (result.indices was empty CPU Vec; GPU tensors td_errors_gpu/indices_gpu
  were silently dropped)
- Fix IS-weight ordering: apply weights AFTER Huber loss, not before
  (weighting before nonlinear Huber shifts quadratic/linear regime boundary)
- Fix IQN CVaR: add .contiguous() before sort_last_dim (341/341 tests pass)
- Defer loss scalar readback to after backward pass (piggyback on grad flush)
- Add next_states to ExperienceBatch with episode-aware shift computation
- Add insert_batch_tensors() for direct GPU tensor insertion into replay buffer
- Make Q-value estimation periodic (every 50 steps) to reduce forward passes
- Fix CPU fallback path types (u8 action, i32 fixed-point reward, timestamp)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 21:50:26 +01:00
jgrusewski
c208217791 docs: add ML crate split phase 2 design (domain-aligned extraction)
Plan to reduce ml crate from 91K to ~12K LOC by extracting trainers
into model sub-crates, hyperopt adapters into ml-hyperopt, and
infrastructure into existing sub-crates. ml becomes an orchestration
layer owning inference, model factory, and training pipeline.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 19:39:48 +01:00
jgrusewski
7e5af20373 fix(ci): make CUDA non-default in 5 remaining ml sub-crates
ml-supervised, ml-ensemble, ml-labeling, ml-explainability, ml-hyperopt
all had default = ["cuda"] which pulled cudarc into the CPU services
build, causing compile-services to fail with "nvcc not found".

Changed all to default = [] and wired ml/Cargo.toml cuda feature to
propagate to all 8 sub-crates (was only 3: ml-core, ml-dqn, ml-ppo).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 19:35:55 +01:00
jgrusewski
eec5494344 fix(dqn): tighten v_range, replace val_loss proxy, add per-action Q diagnostics
- v_range search bounds [5,30] → [2,10] to match EMA-normalized reward
  range [-0.45,+0.32] (max discounted return ≈1.2)
- Replace val_loss proxy (max_Q - reward)^2 with Sharpe-based validation
  metric: returns -Sharpe so lower = better, economically meaningful
- Add compute_per_action_q_values(): samples 200 states from replay
  buffer, logs per-action Q averages (S100/S50/Flat/L50/L100) at each
  epoch — exposes whether agent differentiates exposure actions
- Remove dead max_q_values computation (leftover from old proxy)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 19:29:17 +01:00
jgrusewski
31d6d63d3f feat(dashboard): add Source selector (Live/CI History/All) for training metrics
Adds a `source` template variable to the training dashboard that controls
which Prometheus job labels are queried:
- Live: only `training-pods` (running pods scraped directly)
- CI History: only `.*_baseline.*` (completed CI runs via pushgateway)
- All: both sources combined

All 37 panel queries now use `job=~"$source"` for consistent filtering.
Active Workers panel stays hardcoded to `job="training-pods"` since it
only makes sense for live pods. Template variable queries (model/fold
dropdowns) also respect the source selector.

Default is "Live" — dashboard shows only the currently running training
session with no pushgateway stale data contamination.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 19:28:38 +01:00
jgrusewski
9a36dfc0ae fix(dashboard): filter training metrics to live pods, exclude stale pushgateway data
Stat panels (Active Workers, Current Epoch, Trial Progress, Elapsed, Best
Objective) showed conflicting values from pushgateway stale gauges overlapping
with live training pod metrics. Scoped all stat/gauge panels and template
variable queries to job="training-pods" so the dashboard reflects only the
currently running training session. Time-series panels remain unfiltered via
$model template variable scoping — dropdown only lists live models, so
historical pushgateway data is naturally excluded without explicit filtering.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 19:22:43 +01:00
jgrusewski
3d68e9feaf fix(ci): make CUDA non-default to unblock CPU service builds
- Remove cuda from default features in ml-core, ml-dqn, ml-ppo, ml
- Propagate cuda feature from ml → ml-core/ml-dqn/ml-ppo
- CI compile-training already uses --features ml/cuda explicitly
- Fix MaxDD log format: {:.1}% → {:.3}% (was rounding 0.033% to 0.0%)
- Suppress unused_labels/unused_variables warnings for cfg(cuda) code
- Add CALLBACK_ENDPOINT env to ml-training-service deployment
- Fix Grafana active_workers query to use sum() with fallback

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 19:12:03 +01:00
jgrusewski
dfab77ad6f ci: re-trigger after sccache purge + enable log archiving
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 18:39:44 +01:00
jgrusewski
c384ab4b43 fix(infra): replace gitlab.com/prometheus annotations with prometheus.io
Prometheus was configured to scrape pods via prometheus.io/scrape
annotations, but all services and training jobs used gitlab.com/
prefix from legacy GitLab-managed Prometheus — causing Prometheus
to never discover any foxhunt pods. This resulted in stale/missing
metrics on the Grafana training dashboard.

12 files updated across services/, gpu-overlays/, training/, and
monitoring/ (node-exporter, dcgm-exporter cleanup).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 18:39:24 +01:00
jgrusewski
9c249e2a27 fix(dqn): symmetric distributional support + near-zero output init to unfreeze Q-values
Root cause: C51 distributional DQN Q-values frozen at ~7.0 for all 30 epochs.
Asymmetric hyperopt support (v_min=-6.7, v_max=17.6) caused init Q ≈ midpoint
= 5.48, creating a self-reinforcing equilibrium that gradient signal couldn't
escape. Additionally, log_q_values() queried the wrong (untrained) network.

Three fixes:
1. Enforce symmetric support in hyperopt: v_min=-v_range, v_max=+v_range
   (midpoint always 0, Q starts unbiased)
2. Near-zero init for distributional output layers (value_out, advantage_out):
   0.01× Xavier scale → softmax ≈ uniform → Q ≈ midpoint regardless of support
3. Fix log_q_values() to use self.forward() (unified dispatch through
   dist_dueling → dueling → standard) instead of self.q_network.forward()

Tests: 350 ml-dqn + 841 ml + 3 ml-core = 1194 passed, 0 failed

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 18:11:22 +01:00
jgrusewski
8c0bed4d2d fix(infra): update training job template to use ci-training-h100 pool
The nodeSelector pointed to non-existent 'ci-training' pool. The actual
Scaleway pool is 'ci-training-h100', matching the CI pipeline template.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 17:20:15 +01:00
jgrusewski
e89fce5eb2 fix(cuda): per-thread EMA reward normalization in GPU experience kernel
GPU kernel produced raw percentage PnL rewards (~3.5e-4) which were
~10,000x smaller than Q-values (~7.0), making rewards invisible in the
Bellman backup. Q-values froze at initialization.

Port the CPU RewardNormalizer algorithm (EMA mean/variance, z-score
normalization, [-3,3] clamp) directly into the CUDA kernel hot path
as per-thread register state. Each thread maintains its own running
mean/variance with configurable decay rate (reward_norm_alpha, default
0.01 = ~100-step window). EMA resets on episode boundaries.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 16:49:11 +01:00
jgrusewski
121962b7c5 Merge branch 'feature/ml-crate-split' — split ml monolith into 21 sub-crates
21 sub-crates extracted from the 260K-line ml monolith, reducing it to ~90K.
Deletes ~7.5K lines of dead code. Zero compilation errors, zero warnings.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 15:23:14 +01:00
jgrusewski
1f1ba40eaa fix: resolve rebase artifacts — import paths and MLError variant
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 15:21:46 +01:00
jgrusewski
b95ee9d336 refactor(ml): remove dead code across 8 sub-crates — delete 742 lines
Remove unused struct fields, dead methods, unreachable code across
ml-dqn, ml-supervised, ml-features, ml-ppo, ml-checkpoint, ml-labeling,
ml-observability, and ml-universe. Gate test-only infra behind #[cfg(test)].

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 15:17:22 +01:00
jgrusewski
58f4f26113 refactor(ml): extract regime-detection, explainability, paper-trading
- ml-regime-detection (1.2K lines): feature_classifier, hmm modules.
  Depends on ml-core + ml-dqn (RegimeType). 23 tests passing.

- ml-explainability (329 lines): integrated_gradients module.
  Depends on ml-core + candle-core. 4 tests passing.

- ml-paper-trading (389 lines): broker, pnl_tracker modules.
  Depends on ml-ensemble (TradeAction, TradeSignal). 8 tests passing.

Total: 21 sub-crates extracted from ml monolith.
ml reduced from ~260K to ~90K lines (65% extracted).
All tests: 841 ml + 35 in new sub-crates = 876 passing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 15:17:22 +01:00
jgrusewski
4676fe79e2 refactor(ml): extract observability, stress-testing, security into sub-crates
- ml-observability (1.2K lines): alerts, dashboards, metrics modules.
  Depends on ml-core + common (ModelType). 4 tests passing.

- ml-stress-testing (1.3K lines): load_generator, market_simulator,
  performance_analyzer modules. Depends on ml-core + common + config.
  5 tests passing.

- ml-security (1.4K lines): anomaly_detector, prediction_validator
  modules. Depends on ml-core + ml-ensemble (EnsembleDecision,
  ModelVote, TradingAction). 17 tests passing.

Total: 18 sub-crates extracted from ml monolith.
Workspace: 0 errors, ml tests 876 + 26 in new sub-crates.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 15:17:22 +01:00
jgrusewski
b7597a7543 refactor(ml): extract universe, backtesting, asset-selection into sub-crates
- ml-universe (1.7K lines): correlation, liquidity, momentum, volatility
  modules. Only depends on ml-core (MLError, PRECISION_FACTOR).
  6 tests passing.

- ml-backtesting (1.1K lines): action_loader, barrier_backtest, report
  modules. Only depends on ml-core (MLError). Discovered and included
  previously undeclared report.rs module. 10 tests passing.

- ml-asset-selection (1.2K lines): scorer, selector modules with
  AssetClass, AssetUniverse, PredictabilityScorer, ActiveSetSelector.
  Only depends on ml-core (MLError). 33 tests passing.

All three replaced with thin facade re-exports in ml — existing
`use ml::universe::*` / `use ml::backtesting::*` /
`use ml::asset_selection::*` paths continue to work.

Total: 15 sub-crates extracted from ml monolith.
Workspace: 0 errors, ml tests 902 passed + 49 in sub-crates.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 15:17:22 +01:00
jgrusewski
aa19b42255 refactor(ml): move UnifiedTrainable to ml-core + delete 6K dead deployment code
- Move UnifiedTrainable trait, TrainingMetrics, CheckpointMetadata, and
  checkpoint helpers from ml to ml-core (zero new dependencies — ml-core
  already had candle-core + serde_json)
- Wrap Mamba2SSM in Mamba2TrainableAdapter to satisfy orphan rule (trait
  in ml-core, type in ml-supervised — all other 9 models already used
  wrapper pattern)
- Make Mamba2SSM::validate() pub for cross-crate adapter access
- Delete 5 permanently disabled deployment modules (cfg(any()) — never
  compiled): registry, hot_swap, validation, monitoring, endpoints
  (-5,842 lines)
- Delete 2 undeclared dead files in training/: dqn_trainer.rs,
  transformer_trainer.rs (-138 lines)
- Fix pre-existing compute_loss test shape mismatch in mamba adapter

UnifiedTrainable in ml-core unblocks future trainers/ extraction (17.8K
lines) since model-specific trainers can now depend on ml-core for the
trait without pulling in the full ml monolith.

14 files changed, +128 -6,497 (net -6,369 lines)
Tests: 274 ml-core + 948 ml = 1,222 passed, 0 failed

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 15:17:22 +01:00
jgrusewski
d313486dc2 refactor(ml): split monolith into 9 sub-crates + delete dead code
Extract 9 new sub-crates from the ml monolith to enable parallel
compilation across the workspace:

New crates (this commit):
- ml-features (282 tests): feature engineering, 21 modules
- ml-labeling (45 tests): triple barrier, meta-labeling, fractional diff
- ml-ensemble (116 tests): ensemble coordination, voting, confidence
- ml-hyperopt (47 tests): core PSO/TPE optimizer, parameter space
- ml-checkpoint (41 tests): checkpoint persistence, compression, signing
- ml-regime (68 tests): CUSUM, Bayesian changepoint, regime classification
- ml-data-validation (67 tests): FDR correction, CPCV, data quality
- ml-risk (33 tests): neural VaR, Kelly criterion, circuit breakers
- ml-validation (43 tests): statistical validation, walk-forward, DSR

Extended existing crates:
- ml-dqn: added evaluation/ (backtesting engine, metrics, reports)
  and checkpoint implementation
- ml-supervised: added checkpoint implementations
- ml-core: added shared types needed by new sub-crates

Pattern: each module in ml/ becomes a thin facade (pub use subcrate::*)
with bridge modules staying in ml for cross-model adapter code.

Dead code deleted (~7K lines):
- 13 undeclared files in microstructure/ (never compiled)
- 7 undeclared files + tests/ in risk/ (never compiled)
- parquet_io, cache_service, cache_storage, minio_integration (unused)
- extraction_wave_d_impl.rs (bare fn outside impl block)

All 2,746 sub-crate tests + 951 ml tests pass.
Full workspace builds clean.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 15:17:22 +01:00
jgrusewski
3db7f4828b refactor(ml): extract 8 supervised models into ml-supervised crate (task 8)
Move TFT, Mamba-2, Liquid, TGGN, TLOB, KAN, xLSTM, and Diffusion model
implementations to ml-supervised. Bridge files (UnifiedTrainable adapters,
Checkpointable impls) stay in ml. Delete AsyncDataLoader (replaced by
StreamingDbnLoader + simple .chunks() batching). Remove empty ml-infra
scaffold — the remaining ml modules are too tightly coupled for clean
extraction, so ml stays as the orchestration facade.

- ml-supervised: 234 tests, 0 failures
- ml: 1687 tests, 0 failures
- Workspace: 0 compilation errors

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 15:16:08 +01:00
jgrusewski
58d0f535df refactor(ml): extract PPO module into ml-ppo crate (task 7)
Move 24 PPO source files + flow_policy/ from ml into standalone
ml-ppo crate. The ml crate's ppo module is now a thin re-export layer
(`pub use ml_ppo::*`) plus two bridge files (trainable_adapter.rs,
stress_testing.rs) that depend on ml-internal types.

Key changes:
- ml-ppo: 25 modules (incl flow_policy subdir), 198 tests, standalone
- ml: depends on ml-ppo, re-exports via ppo/mod.rs
- Import rewrites: crate::common::action → ml_core::action_space,
  crate::dqn::{mixed_precision,xavier_init} → ml_core::*,
  crate::gradient_accumulation → ml_core::gradient_accumulation
- ml tests: 1929 pass (down from 2127 — 198 moved to ml-ppo)
- Workspace: 0 errors

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 15:16:08 +01:00
jgrusewski
88f0b3ec23 refactor(ml): extract DQN module into ml-dqn crate (task 6)
Move 53 DQN source files + gpu_replay_buffer from ml into standalone
ml-dqn crate. The ml crate's dqn module is now a thin re-export layer
(`pub use ml_dqn::*`) plus two bridge files (trainable_adapter.rs,
stress_testing.rs) that depend on ml-internal types.

Key changes:
- ml-dqn: 45 modules, 334 tests, compiles standalone
- ml: depends on ml-dqn, re-exports via dqn/mod.rs
- DQN.config: pub(crate) → pub for cross-crate access
- DQNAgent checkpoint methods moved into ml-dqn
- gpu_replay_buffer moved from cuda_pipeline/ to ml-dqn
- 8 dead-code files removed (never declared in mod.rs)

Net: -30,648 lines from ml crate. Workspace: 0 errors.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 15:16:08 +01:00
jgrusewski
d2f3b6c799 refactor(ml): move metrics/ + performance.rs to ml-core (5g)
Move Sharpe ratio metrics and SIMD performance module to ml-core.
These only depend on MLError which is already in ml-core.

Add approx = "0.5" to ml-core dev-dependencies for Sharpe tests.
Skip observability/ (needs prometheus dep — stays in ml).

Test counts: 271 ml-core + 2487 ml = 2758 total, 0 failures

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 15:13:41 +01:00
jgrusewski
4843b4a2a6 refactor(ml): move shared infrastructure to ml-core + eliminate FactoredActionExt (5e)
Move 6 shared modules (~2.3K lines) from ml to ml-core:
- trading_action.rs (TradingAction enum)
- action_space.rs (action masking, re-exports)
- xavier_init.rs (Xavier/Glorot weight initialization)
- mixed_precision.rs (AMP utilities, FP16/BF16)
- order_router.rs (deterministic order routing)
- portfolio_tracker.rs (portfolio state tracking)

With TradingAction now in ml-core alongside FactoredAction, the
FactoredActionExt extension trait is eliminated entirely —
from_trading_action()/to_trading_action() become inherent methods
on FactoredAction. This removes the need for `use FactoredActionExt`
imports in reward.rs, gae.rs, and trajectories.rs.

Test counts: 250 ml-core + 2508 ml = 2758 total, 0 failures

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 15:13:41 +01:00