gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(ml-alpha): eval-path cfc_step launch config — block-per-batch (Phase B fix-up)

Browse Source 
Phase B commit 1 (cfc_step block-per-batch refactor) updated the
TRAINING-path cfg_cfc to grid=(B,1,1) but missed the same change in
evaluate_batched. The eval path silently kept the legacy grid=(1,1,1)
launch config, which against the refactored kernel meant only block 0
ran — batches 1..B-1 got GARBAGE h_new (whatever was in scratch
memory), which propagated through CfC + GRN to produce garbage probs,
which BCE-eval'd to chance-level AUC.

Caught by t6z89-vs-txftz cluster A/B at L40S:
  baseline (t6z89, pre-Phase-B): mean_auc=0.7428 / h6000=0.7211 @ E0
  broken  (txftz, Phase B):      mean_auc=0.4973 / h6000=0.5136 @ E0
  train_loss matched within noise (0.6232 vs 0.6258) — the smoking gun
  for "training works, eval is broken".

Why the perception_overfit smokes didn't catch it: those tests check
that loss converges on a synthetic up-ramp signal, exercising only the
training path. Eval is exercised by `evaluate_works_after_*` smokes
but those use n_batch=1, where grid=(1,1,1) and grid=(B,1,1)=(1,1,1)
are bit-identical — the bug only manifests at B>1.

Fix: eval cfg_cfc → grid=(b_sz, 1, 1), matching training. Plus an
explanatory comment so future eyes don't repeat the mistake.

Phase B perf gains are unchanged (training path was correct). Only
eval's wall time may grow slightly because of the now-correct
per-batch parallelism doing the work it should have done.

Follow-up: re-submit cluster A/B vs t6z89 to confirm AUC trajectory
recovers to ±0.005 of baseline.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-18 00:23:42 +02:00
parent a478ba3d84
commit 73925b15d3
1 changed files with 8 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
crates/ml-alpha/src/trainer/perception.rs
View File

@@ -2475,10 +2475,15 @@ impl PerceptionTrainer {
let dt_s = self.cfg.decision_stride.max(1) as f32;
let n_in_i = HIDDEN_DIM as i32;
let n_hid_i = HIDDEN_DIM as i32;
let block_dim = 128u32;
let grid_dim = (HIDDEN_DIM as u32).div_ceil(block_dim);
// Phase B fix-up: block-per-batch (Phase B), same launch contract as
// training's K-loop. BEFORE this fix the eval path used the legacy
// grid=(1,1,1) config which only processed batch 0 — batches
// 1..B-1 got GARBAGE h_new. Caught by t6z89-vs-txftz cluster A/B
// (val_loss=0.83, mean_auc=chance) on 2026-05-17.
let cfg_cfc = LaunchConfig {
grid_dim: (grid_dim, 1, 1), block_dim: (block_dim, 1, 1), shared_mem_bytes: 0,
grid_dim: (b_sz as u32, 1, 1),
block_dim: (HIDDEN_DIM as u32, 1, 1),
shared_mem_bytes: 0,
};
let cfg_grn_fwd = LaunchConfig {
grid_dim: (b_sz as u32, 1, 1),