gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,866 Commits 13 Branches 0 Tags
7b60fd5f8636ce144abbbc04631328993c942e60
Commit Graph

7 Commits

Author SHA1 Message Date
jgrusewski
2aedc2ae1a feat(ml): comprehensive GPU saturation audit — 58 fixes across all 10 models 
Phase 1 — Fix broken models (P0):
- Diffusion: wire optimizer_step to actually apply gradients (was no-op)
- TLOB: connect forward pass to projection layers (was Tensor::zeros)
- Mamba2: F64→F32 migration across 5 files (~30x faster on L40S tensor cores)

Phase 2 — Eliminate hot-path GPU sync stalls:
- Mamba2: keep dt on GPU in discretize_ssm (4 functions, no CPU round-trip)
- TFT: gate attention weight logging to eval only (8 syncs/forward eliminated)
- Mamba2: defer loss scalar after backward (pipeline stall removed)
- Mamba2: delete dead gradient clipping (4N wasted GPU syncs removed)

Phase 3 — Enable BF16 for supervised models:
- Flip mixed_precision defaults to true in 4 config locations
- Fix cuda_layer_norm to support BF16/F16 via F32 intermediate

Phase 4 — Raise hyperopt bounds for datacenter GPUs:
- 7 adapters with VRAM-aware tiers (TFT, Liquid, TGGN, KAN, xLSTM,
  Diffusion, TLOB) — L40S gets full hidden_dim range
- Fix L40S tier boundary (was excluded at <48000, now >=40000)

Phase 5 — Update memory estimates:
- 10 param_count estimates updated (DQN 200K→12M, TFT 2M→50M, etc.)
- Fix power-of-two rounding (was wasting up to 49% of budget)
- Correct MODEL_OVERHEAD_MB in DQN/PPO/TFT adapters

Phase 6 — Fix per-epoch CPU bottlenecks:
- PPO: deduplicate double advantage normalization (correctness fix)
- PPO: GPU tensor reward normalization + explained variance
- Fuse per-parameter grad norm to single GPU sync (xLSTM, KAN, TGGN)

Phase 7 — Data pipeline:
- GpuBufferPool: use from_slice (eliminate staging buffer copy)

Phase 8 — Correctness:
- TFT: remove broken .detach() in forward_checkpointed (restore gradients)
- Update stale RTX 3050 Ti doc references

33 files changed, 2451 tests pass, 0 clippy warnings.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-02 15:41:23 +01:00
jgrusewski
d63190b9b9 feat(ml): GPU full saturation — remove artificial caps and enable VRAM-aware scaling 
Phase 1: Remove artificial caps
- TFT benchmark: VRAM-scaled batch sizes (4/16/32/64) replacing hardcoded max_batch=4
- Liquid CUDA: VRAM-aware config defaults (batch 256-2048, pool 10% VRAM)
- DQN trainer: remove double-clamp between AutoBatchSizer and HardwareBudget

Phase 2: Mixed precision in training
- DQN agent: add BF16/FP16 dtype casting in forward_with_gradients and
  forward_without_gradients (training was bypassing forward_mixed)

Phase 3: Reduce CPU round-trips
- DQN trainer: flat buffer select_actions_batch (eliminate Vec<Vec<f32>>)
- DQN trainer: early-skip experience extraction (avoid .to_vec() on invalid)
- EpochPrefetcher: AtomicBool is_ready() so callers can detect completion

Phase 4: Adaptive scaling
- HardwareBudget: tiered safety factor (0.70-0.85 by GPU size)
- AutoBatchSizer: VRAM-proportional batch_overhead_mb (1.5% instead of fixed 250MB)

2451 tests pass, 0 clippy warnings.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-02 11:42:38 +01:00
jgrusewski
14d405f112 refactor(ml): remove legacy base_model_memory_mb from BatchSizeConfig 
Single model_memory_mb field now serves as the sole source of truth for
base model memory. Removes the redundant base_model_memory_mb field,
legacy dual-path branches in calculate_optimal_batch_size() and
max_safe_batch_size(), and the associated legacy test.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-28 20:58:04 +01:00
jgrusewski
6804f0facf fix(dqn): let PSO control batch_size, use VRAM ceiling instead of AutoBatchSizer override 
AutoBatchSizer was computing an "optimal" batch_size (128 on L4) and
overriding the PSO-chosen value. On an L4 24GB this meant 3.5% VRAM
utilization per trial. Now the trainer only clamps if the batch_size
would actually OOM, letting PSO explore the full [64, 4096] range.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-28 20:48:43 +01:00
jgrusewski
979061c80c fix: downgrade remaining DQN training warn noise to debug/info 
Portfolio value <= 0 during early exploration is expected behavior,
not a warning. GPU memory utilization advisory is informational.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-27 23:48:19 +01:00
jgrusewski
ff8a275d47 fix(ml): widen flaky QAT checkpoint test range + use PVC sccache 
- Widen observer checkpoint test bounds from 4σ to 5σ — random
  calibration data can exceed 4.0 with 100 batches of normal(0,1)
- Override SCCACHE_BUCKET="" in .rust-base so check/test jobs use
  PVC-backed SCCACHE_DIR instead of S3

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-25 23:55:35 +01:00
jgrusewski
9c3d741a08 refactor: restructure repo — crates/, bin/, testing/ layout 
Move 17 library crates into crates/, CLI binary into bin/fxt,
consolidate 10 test crates into testing/, split config crate
from deployment config files.

Root directory reduced from 38+ to ~17 directories.
All Cargo.toml paths and build.rs proto refs updated.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-25 11:56:00 +01:00