gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
4,135 Commits 13 Branches 0 Tags
82dca76daebd7a5f13b1f24780a2eaa2d4e058bb
Commit Graph

5 Commits

Author SHA1 Message Date
jgrusewski
904185004c feat: L40S GPU profile + auto-derive cuda-compute-cap from GPU pool 
argo-train.sh now auto-selects cuda-compute-cap based on --gpu-pool:
  - ci-training-h100* → sm_90 (Hopper)
  - ci-training-l40s  → sm_89 (Ada Lovelace)

Added config/gpu/l40s.toml:
  - batch_size=4096 (between H100's 8192 and A100's 2048)
  - buffer_size=300K (scaled for 48GB VRAM)
  - gpu_timesteps_per_episode=2000 (bandwidth-limited)
  - gpu_n_episodes=2048 (scaled from H100's 4096)

GPU profile loader maps "L40S" → "l40s" (was "a100" fallback).

Also fixed pre-existing test drift: num_atoms=52 in h100.toml/a100.toml
was 51 in test expectations (padding alignment for C51 kernels).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-20 16:59:06 +02:00
jgrusewski
2c8967ad96 feat: compute-sanitizer support in Argo training workflow 
Usage: ./scripts/argo-train.sh dqn --baseline --epochs 2 --sanitizer memcheck
       ./scripts/argo-train.sh dqn --baseline --epochs 1 --sanitizer synccheck

Tools: memcheck (OOB, uninitialized), racecheck (data races),
synccheck (__syncthreads divergence/deadlocks).
10-100x slower — use with 1-2 epochs for debugging.
Detects exact kernel + line causing GPU hang.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-17 12:24:23 +02:00
jgrusewski
445c2197f2 feat: unified Argo train workflow — replace 7 templates with 1 
New: train-template.yaml with smart caching:
  - ensure-binary: checks /data/bin/{sha}/ on PVC, compiles only on cache miss
  - ensure-fxcache: runs precompute only if cache doesn't exist
  - gpu-warmup: parallel autoscale during compile
  - hyperopt → train-best → evaluate → upload-results

Deleted 7 redundant templates:
  - compile-and-train-template.yaml
  - train-dqn-template.yaml
  - train-baseline-rl-template.yaml
  - train-supervised-template.yaml
  - training-workflow-template.yaml
  - precompute-features-template.yaml
  - train-ppo-template.yaml

Deleted: scripts/argo-precompute.sh (absorbed into ensure-fxcache step)
Rewritten: scripts/argo-train.sh (single template, --sha for commit pinning)
Updated: kustomization.yaml

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-10 20:36:03 +02:00
jgrusewski
729e8b5fae feat: argo-train.sh --baseline flag (skips hyperopt) + --cache-dir 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-03 23:00:14 +02:00
jgrusewski
6ba52425ea feat(infra): Argo workflow templates, drop cuDNN, GPU hotpath fixes 
- Add compile-and-deploy, train-dqn/ppo/supervised WorkflowTemplates
- Add Argo Events (EventSource, Sensor, Service) for webhook triggers
- Add NetworkPolicy for compile-and-deploy pods (MinIO/DNS/API egress)
- Add convenience scripts: argo-compile-deploy.sh, argo-train.sh
- Drop cuDNN feature flags from all 9 ML crates (zero conv ops in codebase)
- Switch training runtime base to nvidia/cuda:12.9.1-runtime (saves ~800MB)
- Delete unused selective_scan.cu (16KB, zero Rust callers)
- Fix GPU hotpath violations in ml-core (NVTX, gradient utils, capabilities)
- Fix clippy warnings in ml-dqn (VarMap backticks, const fn)
- Add DQN GPU smoketest, backtest evaluator signal adapter fixes

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-12 01:44:03 +01:00