From 33b03a86a625be21a1418b9ef9942ea91b096dd6 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Mon, 13 Apr 2026 20:13:39 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20IQL=20H100=20hardening=20spec=20?= =?UTF-8?q?=E2=80=94=20drift=20readiness,=20p5=20floor,=20buffer-relative?= =?UTF-8?q?=20staleness?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.6 (1M context) --- .../2026-04-13-iql-h100-hardening-design.md | 131 ++++++++++++++++++ 1 file changed, 131 insertions(+) create mode 100644 docs/superpowers/specs/2026-04-13-iql-h100-hardening-design.md diff --git a/docs/superpowers/specs/2026-04-13-iql-h100-hardening-design.md b/docs/superpowers/specs/2026-04-13-iql-h100-hardening-design.md new file mode 100644 index 000000000..40fd598a4 --- /dev/null +++ b/docs/superpowers/specs/2026-04-13-iql-h100-hardening-design.md @@ -0,0 +1,131 @@ +# IQL H100 Hardening Design + +## Goal + +Fix three production concerns before H100 baseline: readiness activates too fast on large batches, per-sample C51 support drops samples silently, staleness decay is ineffective at scale. All fixes self-calibrating, zero new hyperparameters. + +## Fix 1: V(s) Inter-Batch Drift Readiness + +Replace CV-based readiness (`(cv_initial - cv_current) / cv_initial`) with two-timescale EMA drift detection on V(s) mean. + +### Kernel changes (`iql_adv_sigma_ema_update`) + +The kernel currently computes advantage sigma EMA + CV readiness. Replace the readiness computation: + +``` +// Compute v_mean from v_out_buf (grid-stride loop in same single-thread kernel) +v_mean = sum(v_out_buf[0..B]) / B + +// Two-timescale EMA +v_ema_fast = 0.9 * v_ema_fast + 0.1 * v_mean +v_ema_slow = 0.99 * v_ema_slow + 0.01 * v_mean + +// Relative drift +drift = |v_ema_fast - v_ema_slow| / max(|v_ema_slow|, 1e-4) + +// Self-calibrating scale: capture first drift after 10-step warmup +if (step_counter < 10) { readiness = 0; step_counter++; } +else if (drift_scale < 1e-8) { drift_scale = drift; readiness = 0; } +else { readiness = 1 - clamp(drift / drift_scale, 0, 1); } +``` + +### Buffer changes + +Expand `readiness_buf` from `[2]` to `[6]`: +- `[0]` = readiness scalar +- `[1]` = drift_scale (self-calibrated) +- `[2]` = v_ema_fast +- `[3]` = v_ema_slow +- `[4]` = step_counter (as float, cast to int in kernel) +- `[5]` = reserved + +### Kernel signature change + +Add `const float* __restrict__ v_out_buf` and `int batch_size` parameters to `iql_adv_sigma_ema_update` so it can compute v_mean internally. + +### Dead code + +The CV-based readiness in the current kernel (`cv = sigma_ema / |mean|`) is replaced entirely. The `readiness_buf[1]` which stored `initial_cv` is repurposed for `drift_scale`. + +## Fix 2: Frugal p5 Quantile Floor for C51 Support + +### New kernel: `iql_support_floor` + +Single-thread kernel (grid=1, block=1) that runs after `iql_compute_per_sample_support`: + +``` +// Step 1: Frugal p5 estimator update +// Read half_widths from per_sample_support (recoverable as (v_max - v_min) / 2) +for b in 0..batch_size: + hw = (per_sample_support[b*3+1] - per_sample_support[b*3+0]) / 2 + if hw < p5_estimate: + p5_estimate -= p5_estimate * 0.05 // proportional step down + else: + p5_estimate += p5_estimate * 0.05 * (0.05 / 0.95) // asymmetric for p5 + +// Step 2: Floor delta_z for all samples +floor_hw = p5_estimate * 0.1 +floor_dz = (2 * floor_hw) / (num_atoms - 1) +for b in 0..batch_size: + if per_sample_support[b*3+2] < floor_dz: + // Recompute v_min/v_max/delta_z with floored half_width + center = (per_sample_support[b*3+0] + per_sample_support[b*3+1]) / 2 + per_sample_support[b*3+0] = center - floor_hw + per_sample_support[b*3+1] = center + floor_hw + per_sample_support[b*3+2] = floor_dz +``` + +### Buffer changes + +- New: `p5_ema_buf: CudaSlice` size 1, initialized to 1.0 (safe default) +- New kernel function loaded from cubin: `iql_support_floor` + +### Integration + +Called from `submit_aux_ops` immediately after `compute_per_sample_support`, before `compute_branch_scales`. + +## Fix 3: Buffer-Relative Staleness + +### Kernel change (`iql_modulate_td_errors`) + +Replace: +```cuda +float decay = expf(-staleness_lambda * (float)age / fmaxf(staleness_tau, 1.0f)); +``` +With: +```cuda +float age_norm = (float)age / fmaxf((float)capacity, 1.0f); +float decay = expf(-3.0f * age_norm); +``` + +Remove `staleness_lambda` and `staleness_tau` parameters from kernel signature. Remove `float beta` parameter (the advantage temperature is already baked into the advantage weights by `iql_compute_advantage_weights`). + +Wait — `beta` is used for computing K (dynamic clamp). Keep `beta`. Only remove `staleness_lambda` and `staleness_tau`. + +### Rust changes + +- Remove `staleness_lambda` and `staleness_tau` from `GpuIqlConfig` +- Remove corresponding `.arg()` calls in `modulate_td_errors()` launcher +- Remove from `GpuIqlConfig::default()` +- Remove from `FusedTrainingCtx::new` where they're set to `0.001` and `10000.0` + +### Dead code removed + +- `GpuIqlConfig::staleness_lambda` field +- `GpuIqlConfig::staleness_tau` field +- Both default values +- Both `.arg()` calls in the Rust launcher + +## Files Modified + +| File | Changes | +|------|---------| +| `iql_value_kernel.cu` | Rewrite `iql_adv_sigma_ema_update` (drift readiness), new `iql_support_floor` kernel, simplify `iql_modulate_td_errors` | +| `gpu_iql_trainer.rs` | Expand `readiness_buf` to [6], add `p5_ema_buf`, load `iql_support_floor` kernel, update launcher signatures, remove staleness config | +| `fused_training.rs` | Call `support_floor` after `compute_per_sample_support`, remove staleness args from IQL config init | + +## Testing + +1. `test_no_nan_after_graph_capture` — basic smoke +2. `test_production_training_stability` — 10 epochs, check Best Sharpe +3. `test_walk_forward_no_overfitting_50_epochs` — 50 epochs × 3 folds, verify peak epoch moves later and Sharpe holds