gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,595 Commits 13 Branches 0 Tags
ab218121b671e0d06bb9c8bf36d0956c68bbca19
Commit Graph

5 Commits

Author SHA1 Message Date
jgrusewski
a454a26a5c feat(ml): GPU saturation backlog — PPO rollout sync, VRAM-aware dims, KAN GPU splines 
Three remaining GPU bottlenecks from the saturation backlog:

1. PPO rollout GPU sync stalls (3→1 per step):
   - Merged sample_action to return (action_idx, probs_vec), eliminating
     duplicate to_vec1 sync on action probabilities
   - Batched critic forward after rollout loop — single GPU→CPU sync
     replaces per-step critic.forward() calls (2048 syncs → 1)
   - Safe indexing throughout (clippy deny rules)

2. VRAM-aware default network dimensions:
   - Added detect_vram_mb() with GPU_MEMORY_MB env var override for K8s
   - Added vram_scaled_hidden_dims() with 4 tiers (CPU/<8GB/16GB/40GB+)
   - DQN: [256,256] → [2048,1024,512] on L40S/H100
   - PPO: hidden_dim_base 128 → 1024 on L40S/H100
   - Wired into train_baseline_rl.rs for non-hyperopt training runs

3. KAN B-spline GPU lookup table:
   - Pre-compute basis values on 1024-point grid at layer construction
   - GPU evaluation via gather + linear interpolation (replaces recursive
     Cox-de Boor CPU bounce: 32K recursive calls → 2 GPU gathers)
   - Fallback to CPU path when grid not pre-computed

5 files changed, +687/-43, 2476 tests pass, 0 clippy warnings.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-02 15:41:23 +01:00
jgrusewski
2aedc2ae1a feat(ml): comprehensive GPU saturation audit — 58 fixes across all 10 models 
Phase 1 — Fix broken models (P0):
- Diffusion: wire optimizer_step to actually apply gradients (was no-op)
- TLOB: connect forward pass to projection layers (was Tensor::zeros)
- Mamba2: F64→F32 migration across 5 files (~30x faster on L40S tensor cores)

Phase 2 — Eliminate hot-path GPU sync stalls:
- Mamba2: keep dt on GPU in discretize_ssm (4 functions, no CPU round-trip)
- TFT: gate attention weight logging to eval only (8 syncs/forward eliminated)
- Mamba2: defer loss scalar after backward (pipeline stall removed)
- Mamba2: delete dead gradient clipping (4N wasted GPU syncs removed)

Phase 3 — Enable BF16 for supervised models:
- Flip mixed_precision defaults to true in 4 config locations
- Fix cuda_layer_norm to support BF16/F16 via F32 intermediate

Phase 4 — Raise hyperopt bounds for datacenter GPUs:
- 7 adapters with VRAM-aware tiers (TFT, Liquid, TGGN, KAN, xLSTM,
  Diffusion, TLOB) — L40S gets full hidden_dim range
- Fix L40S tier boundary (was excluded at <48000, now >=40000)

Phase 5 — Update memory estimates:
- 10 param_count estimates updated (DQN 200K→12M, TFT 2M→50M, etc.)
- Fix power-of-two rounding (was wasting up to 49% of budget)
- Correct MODEL_OVERHEAD_MB in DQN/PPO/TFT adapters

Phase 6 — Fix per-epoch CPU bottlenecks:
- PPO: deduplicate double advantage normalization (correctness fix)
- PPO: GPU tensor reward normalization + explained variance
- Fuse per-parameter grad norm to single GPU sync (xLSTM, KAN, TGGN)

Phase 7 — Data pipeline:
- GpuBufferPool: use from_slice (eliminate staging buffer copy)

Phase 8 — Correctness:
- TFT: remove broken .detach() in forward_checkpointed (restore gradients)
- Update stale RTX 3050 Ti doc references

33 files changed, 2451 tests pass, 0 clippy warnings.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-02 15:41:23 +01:00
jgrusewski
56374a43ac feat(ml): VRAM-aware hidden dimension scaling for datacenter GPUs 
DQN and PPO trainers now resolve hidden_dim_base from GPU VRAM when no
explicit override is given, so L4/L40S/H100 GPUs use proportionally
larger networks instead of being stuck at RTX 3050 Ti defaults (256).

- Add resolve_hidden_dim_base() tiered lookup (256/512/768/1024 by VRAM)
- Add network_param_count() for accurate model size estimation
- DQN: pre-compute hidden dims, use real param count for batch sizing
- PPO: add hidden_dim_base field, VRAM resolution in PpoTrainer::new()
- PPO hyperopt: raise hidden_dim_base ceiling from 2048 to 4096
- TFT hyperopt: expand hidden_sizes from [128,256,512] to 5 tiers
- Update stale estimates (DQN 50K→200K, PPO 100K→400K params)
- Fix pre-existing clippy lints in prefetch.rs and ppo.rs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-02 11:02:56 +01:00
jgrusewski
14d405f112 refactor(ml): remove legacy base_model_memory_mb from BatchSizeConfig 
Single model_memory_mb field now serves as the sole source of truth for
base model memory. Removes the redundant base_model_memory_mb field,
legacy dual-path branches in calculate_optimal_batch_size() and
max_safe_batch_size(), and the associated legacy test.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-28 20:58:04 +01:00
jgrusewski
9c3d741a08 refactor: restructure repo — crates/, bin/, testing/ layout 
Move 17 library crates into crates/, CLI binary into bin/fxt,
consolidate 10 test crates into testing/, split config crate
from deployment config files.

Root directory reduced from 38+ to ~17 directories.
All Cargo.toml paths and build.rs proto refs updated.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-25 11:56:00 +01:00