gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,927 Commits 13 Branches 0 Tags
b10cdebfb17201c1395615d7be63ebbea720623e
Commit Graph

6 Commits

Author SHA1 Message Date
jgrusewski
864808e056 fix(ml): add ensure_training_dtype boundary casts to all model forward methods 
After converting all VarBuilder sites from F32 to training_dtype (BF16 on
Ampere+ GPUs), input tensors from callers remain F32, causing dtype
mismatches in matmul/add/mul operations. This commit adds systematic
boundary casts across all 10 model architectures:

- Input boundary: ensure_training_dtype() at each model forward() entry
- Output boundary: to_dtype(F32) at each model forward() exit
- Internal intermediates: hidden state init, gradient extraction,
  positional encodings, causal masks, SSM state matrices, B-spline
  basis values all cast to match computation dtype
- Ensemble adapters: ensure_training_dtype after Tensor::from_vec

36 files, +293/-72 lines. 2640 tests pass, 0 failures, 0 clippy warnings.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-03 17:42:41 +01:00
jgrusewski
dc8ee1f630 feat(ml): BF16 VarBuilder for all 8 supervised models 
Replace DType::F32 with training_dtype(&device) in VarBuilder::from_varmap
calls across TFT, Mamba2, Liquid/CfC, KAN, xLSTM, Diffusion, TGGN, TLOB.
61 sites changed across 25 files (production constructors + test helpers).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-03 16:23:03 +01:00
jgrusewski
e12cdfa005 feat(ml): diffusion trainer GPU-accumulated loss + batched grad norm 
Replace per-param gradient norm extraction with batched Tensor::stack
pattern (N GPU syncs → 1). Remove per-batch loss scalar extraction
from backward(), defer to epoch-level accumulation. Apply GPU-accumulated
validation loss (stack + mean_all instead of per-batch to_scalar).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-02 17:30:19 +01:00
jgrusewski
2aedc2ae1a feat(ml): comprehensive GPU saturation audit — 58 fixes across all 10 models 
Phase 1 — Fix broken models (P0):
- Diffusion: wire optimizer_step to actually apply gradients (was no-op)
- TLOB: connect forward pass to projection layers (was Tensor::zeros)
- Mamba2: F64→F32 migration across 5 files (~30x faster on L40S tensor cores)

Phase 2 — Eliminate hot-path GPU sync stalls:
- Mamba2: keep dt on GPU in discretize_ssm (4 functions, no CPU round-trip)
- TFT: gate attention weight logging to eval only (8 syncs/forward eliminated)
- Mamba2: defer loss scalar after backward (pipeline stall removed)
- Mamba2: delete dead gradient clipping (4N wasted GPU syncs removed)

Phase 3 — Enable BF16 for supervised models:
- Flip mixed_precision defaults to true in 4 config locations
- Fix cuda_layer_norm to support BF16/F16 via F32 intermediate

Phase 4 — Raise hyperopt bounds for datacenter GPUs:
- 7 adapters with VRAM-aware tiers (TFT, Liquid, TGGN, KAN, xLSTM,
  Diffusion, TLOB) — L40S gets full hidden_dim range
- Fix L40S tier boundary (was excluded at <48000, now >=40000)

Phase 5 — Update memory estimates:
- 10 param_count estimates updated (DQN 200K→12M, TFT 2M→50M, etc.)
- Fix power-of-two rounding (was wasting up to 49% of budget)
- Correct MODEL_OVERHEAD_MB in DQN/PPO/TFT adapters

Phase 6 — Fix per-epoch CPU bottlenecks:
- PPO: deduplicate double advantage normalization (correctness fix)
- PPO: GPU tensor reward normalization + explained variance
- Fuse per-parameter grad norm to single GPU sync (xLSTM, KAN, TGGN)

Phase 7 — Data pipeline:
- GpuBufferPool: use from_slice (eliminate staging buffer copy)

Phase 8 — Correctness:
- TFT: remove broken .detach() in forward_checkpointed (restore gradients)
- Update stale RTX 3050 Ti doc references

33 files changed, 2451 tests pass, 0 clippy warnings.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-02 15:41:23 +01:00
jgrusewski
265bd2441c fix(ml,ci): zero-dim guards on all 10 models, eliminate warnings, unblock CI parallelism 
- Add dimension validation in DQN, PPO, Mamba2, TGGN, TLOB, Liquid,
  KAN, xLSTM, Diffusion constructors (fail-fast on zero-dim inputs
  that would cause CUDA_ERROR_INVALID_VALUE at runtime)
- Add num_unknown_features > 0 guard to TFT (temporal input required)
- Fix 12 dead-code/unused warnings in test compilation
- Remove opt-level=3 and codegen-units=1 from target rustflags
  (was forcing O3 + single-thread codegen on dev/test builds)
- Remove hardcoded jobs=16 cap (cargo now auto-detects CPU count)
- Switch linker to clang+lld (2-5x faster linking)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-26 17:38:43 +01:00
jgrusewski
9c3d741a08 refactor: restructure repo — crates/, bin/, testing/ layout 
Move 17 library crates into crates/, CLI binary into bin/fxt,
consolidate 10 test crates into testing/, split config crate
from deployment config files.

Root directory reduced from 38+ to ~17 directories.
All Cargo.toml paths and build.rs proto refs updated.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-25 11:56:00 +01:00