docs(sp14): spec — aux→Q wire + Earned Gradient Flow pearl
Designs the SP14 chain on top of SP13 Layer B (HEAD 6657e5626):
1. Sub-project A — stability fixes (3 small bugs found in Smoke A)
- C51 atom-probability floor (ISV-driven from SP4 atom_pos_p99)
- aux_w setter clamp lift [0.05, 0.3] → [0.15, 1.5]
- Stagnation warmup gate at fold boundary
2. Sub-project B — the architectural piece (THIS spec)
- Forward wire: aux_softmax_diff per-bar into direction Q-head input
concat (in_dim+1, fingerprint bump, zero-init new column)
- Earned Gradient Flow pearl — adaptive ISV-driven gradient gating:
* Gate 1 (aux competence) — Schmitt-trigger hysteresis
* Gate 2 (Q-head disagreement) — NEW signal, EMA per-step argmax
mismatch
* ISV-adaptive sigmoid steepness (variance-driven k_aux, k_q)
* Per-epoch warmup ramp
* Anti-gradient-hacking circuit breaker (mesa-opt defense)
- 11 new ISV slots, 3 new GPU kernels, ~1060 LOC total
- HEALTH_DIAG pearl_egf_diag observability line
3. Sub-project C — Adaptive LR (deferred until A+B effects measured)
Motivation from Smoke A diagnostic:
- aux_dir_acc reached 0.61 (signal extraction works)
- val_win_rate stuck 45-48% (no path to action selection)
- WR-flat-while-aux-varies = Q-head directional weights frozen
- 1109 GRAD_CLIP_OUTLIER events (chronic; not noise)
Three parallel diagnostic agents triangulated three interlocking root
causes:
- Slot 375 has zero readers (the wire was scoped but never built)
- C51 raw grad reaches 9.5e6, saturates SP7 budget controller
- aux_w controller muzzled by SP11-era [0.05, 0.3] clamp
The Earned Gradient Flow pearl is a new application of the codebase's
pearl pattern: ISV-driven adaptive controller, but applied to backward-
pass gradient flow instead of forward-pass features. The wire is one-
way (stop-gradient) by default; co-training is earned by both:
(a) aux head demonstrating label competence, AND
(b) Q-head showing it's actually fighting aux signal (informative
disagreement above baseline).
Stability additions hardened against:
- Oscillation around target (Schmitt hysteresis)
- Numerical sigmoid saturation (argument clipping ±30)
- Stale variance EMAs across folds (state-reset-registry)
- Discontinuous warmup transitions (linear ramp)
- Mesa-optimization (gradient-hacking circuit breaker)
- Cold-start sentinel-state spurious gate openings (Pearl-A bootstrap)
Awaiting user review before invoking superpowers:writing-plans for
sub-project B (and a separate small plan for sub-project A).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
@@ -0,0 +1,337 @@
|
||||
# SP14 — Aux→Q Wire + Earned Gradient Flow Pearl
|
||||
|
||||
**Status:** Design (awaiting user review)
|
||||
**Date:** 2026-05-05
|
||||
**Author:** Brainstormed in session post-Smoke A
|
||||
**Parent commit:** `6657e5626` (B1.1b — final piece of SP13 Layer B chain)
|
||||
**Depends on:** SP13 chain (B0 → B0.1 → B1.0 → B1.1a → B1.1b) all landed
|
||||
**Branch target:** `sp11-reward-as-controlled-subsystem`
|
||||
|
||||
---
|
||||
|
||||
## Goal
|
||||
|
||||
Connect the SP13 supervised aux signal to the Q-head's directional decision-making, and fix three structural stability issues that Smoke A surfaced. After SP14 lands, the K=2 softmax CE aux head's directional confidence should actually reach the action-selection path, gradient clipping should drop to baseline frequency, and the aux_w controller should reach its designed dynamic range.
|
||||
|
||||
## Motivation
|
||||
|
||||
Smoke A on `6657e5626` (multi_fold_convergence × 3 folds × 5 epochs) revealed a striking diagnostic pattern:
|
||||
|
||||
- `aux_dir_acc` reached **0.61** in fold 1 — supervised signal extraction works, the data has directional signal, the K=2 softmax CE head finds it
|
||||
- `val_win_rate` stayed glued at **45-48%** across all 15 epochs and all 3 folds, even when aux_dir_acc varied 0.16 → 0.61
|
||||
- `val_dir_dist` was uniform `[short=0.24, hold=0.26, long=0.26, flat=0.24]` — Hold-pricing eliminated the prior Hold dominance ✓
|
||||
- `val_sharpe` oscillated 9-16 — sizing/timing branches respond to fold variation
|
||||
- `GRAD_CLIP_OUTLIER` fired **1109 times** (vs 5 in pre-SP13 baseline) — chronic gradient throttling
|
||||
|
||||
The "WR-flat-while-aux-varies" pattern is a structural tell: directional weights are *frozen* (no path to action selection) while size/timing branches respond normally. Three parallel diagnostic agents triangulated three interlocking root causes, documented in the next section.
|
||||
|
||||
The user's intuition was correct on all three counts: longer runs won't fix chronic clipping (gradient information is being lost at every step); the model isn't adapting because the aux signal can't reach the Q-head; and adaptive LR is the right next adaptation lever *but only after* these structural bugs are fixed.
|
||||
|
||||
## Smoke A diagnostic — three interlocking bugs
|
||||
|
||||
| # | Bug | Evidence | Bug class |
|
||||
|---|---|---|---|
|
||||
| 1 | **Aux→Q wire absent.** Slot 375 (`AUX_DIR_PREDICTION_INDEX`) has zero readers in any Q-head pipeline. The deferred Layer B Task B.T3 Step 2 (Q-head input concat) was scoped but never built. | Exhaustive grep across 120 `.cu` files returns no consumers. Self-aware comments at `gpu_dqn_trainer.rs:13385-13387` and `training_loop.rs:4878-4880` admit "Phase 0b wires the consumers — deferred." | Architectural |
|
||||
| 2 | **C51 distributional Q-loss gradient blow-up.** Raw c51 grad reaches `9.5×10⁶` in fold 2; SP7 budget controller saturates at `1e-4` minimum but post-scale grad is still `~10³`, dominating IQN by 50-500×. | HEALTH_DIAG `grad_split_bwd c51=9522607` at fold 0 ep 1; `c51_budget` collapses to `1e-4` and pins. Pre-SP13 baseline had c51 grad 0.006-0.5. | Numerical |
|
||||
| 3 | **aux_w controller muzzled by stale clamp.** P0b's deficit+stagnation controller computes raw aux_w in `[0.15, 1.5]`, but `set_aux_weight` at `gpu_dqn_trainer.rs:14722` re-clamps with the SP11-era `[0.05, 0.3]` cap. Deficit-amplification term is silently dead. Fold 2 stagnation drops aux_w 45% below working value. | Trace at line 14720-14723. Aux_w reaches floor 0.164 in fold 2 vs ceiling 0.3 in folds 0/1 (clamp masks reality). | Configuration |
|
||||
|
||||
Plus two related issues:
|
||||
- **Stagnation gate fires inappropriately at fold boundary** because Pearl-A first-observation bootstrap makes short-EMA = long-EMA at fold reset, giving `improvement = 0` by construction.
|
||||
- **Producer kernel writes batch-mean scalar** to slot 375 instead of per-bar tile (Plan B.T3 sketched per-bar). For per-step Q-head decisions the batch-mean shape is structurally wrong, but the trainer-side `aux_nb_softmax_buf` is already `[B, K]` per-bar — so the wire reads from the source buffer directly, sidestepping the slot-375 shape question.
|
||||
|
||||
## Decomposition into three sub-projects
|
||||
|
||||
| Sub-project | Scope | Size | Brainstorm needed? |
|
||||
|---|---|---|---|
|
||||
| **A — Stability fixes** | Bugs 2, 3, plus stagnation warmup gate | ~3 files, ~50 LOC | No — direct to writing-plans |
|
||||
| **B — Wire + Earned Gradient Flow pearl** | Bug 1 + the new gradient-gating pearl | 12-15 files, ~1200 LOC | Yes — this spec |
|
||||
| **C — Adaptive LR (per-loss-group)** | Bug 6 (the user's open question) | TBD | Deferred until A+B's effect is measured |
|
||||
|
||||
A and B are independent and parallel-safe. A goes straight to a plan; B is the focus of this spec. C is intentionally deferred — only revisit if the post-SP14 30-epoch validation shows WR not climbing past 50%.
|
||||
|
||||
## Sub-project A — stability fixes (this spec defers to plan)
|
||||
|
||||
A is small enough that brainstorming would be theatrical. The plan will land:
|
||||
|
||||
1. **C51 atom-probability floor** in `c51_grad_kernel.cu`: `p_atom = fmaxf(p_atom, ATOM_FLOOR)` before the `−log(p)/p · ∂p/∂z` divide. Floor source: ISV-driven from existing SP4 `atom_pos_p99` per-branch producer. Per `pearl_isv_for_adaptive_bounds`, the floor must be ISV-driven not hardcoded.
|
||||
2. **`set_aux_weight` clamp lift** at `gpu_dqn_trainer.rs:14722`: change `clamp(0.05, 0.3)` to `clamp(AUX_W_BASE * AUX_W_HARD_FLOOR_RATIO, AUX_W_BASE * AUX_W_HARD_CEIL_RATIO)` = `clamp(0.15, 1.5)`, importing constants from `sp13_isv_slots.rs`.
|
||||
3. **Stagnation warmup gate** in `compute_aux_w_p0b` at `training_loop.rs:96-135`: skip stagnation logic when `epochs_in_fold < 1`. Pearl-A first-observation makes short=long at fold reset, so `improvement = 0` by construction in the first epoch — the stagnation gate fires on a non-stagnation. Add `epochs_in_fold > 0` guard.
|
||||
|
||||
Validation: Smoke A re-run on commit landing A should show GRAD_CLIP_OUTLIER count drop from 1109 → ~50 or less, and aux_w trace should show values in `[0.15, 1.5]` range during deficit-amplification phases.
|
||||
|
||||
A and B can land in either order. If A lands first (simpler), B's smoke validation has cleaner gradient signals. If B lands first, A's smoke shows whether the wire alone moves WR. Recommendation in plan: **A first** as a cleanup commit, then B as the main feature work.
|
||||
|
||||
## Sub-project B — Aux→Q wire + Earned Gradient Flow pearl
|
||||
|
||||
This is the architectural piece. Two coupled changes that must land atomically per `feedback_no_partial_refactor`:
|
||||
|
||||
1. **Forward wire**: direction Q-head input concat gains 1 column from `aux_softmax_diff = aux_nb_softmax_buf[b, 1] - aux_nb_softmax_buf[b, 0]` ∈ [−1, +1]
|
||||
2. **Backward gating pearl**: gradient flow back through that wire is gated by an adaptive scalar `α_grad ∈ [0, 1]` driven by ISV signals
|
||||
|
||||
### B.1 — Forward wire design
|
||||
|
||||
**Input shape change:** direction Q-head's input layer was `[h_s2]` (trunk features, shape `[B, SH2]`). Becomes `[h_s2 || aux_softmax_diff]` shape `[B, SH2 + 1]`. The new column is per-bar, one float per batch row.
|
||||
|
||||
**Source of per-bar signal:** read directly from `aux_nb_softmax_buf` (existing `CudaSlice<f32>` shape `[B, K=2]`, populated by `aux_next_bar_forward` post-B1.1a). The diff `softmax[:, 1] - softmax[:, 0]` is bounded structurally in `[−1, +1]` because softmax components are non-negative and sum to 1. Per `pearl_bounded_modifier_outputs_require_structural_activation`, this is the right structural bound — no runtime clamp needed.
|
||||
|
||||
**Slot 375 fate:** kept as-is (batch-mean scalar) for HEALTH_DIAG observability. The existing `aux_pred_to_isv_tanh_kernel` continues to write `mean(softmax[1] - softmax[0])` for diagnostic purposes. The wire bypasses slot 375 and reads `aux_nb_softmax_buf` directly per-bar — slot 375 becomes pure diagnostic, not a critical data path.
|
||||
|
||||
**Q-head SGEMM weight matrix:** `direction Q-head W` grows from `[SH2, branch_0_size]` to `[SH2 + 1, branch_0_size]`. The new row of weights is initialized to **zero** at trainer construction. Rationale: model starts ignoring the new input (no perturbation to existing learned behavior), then learns to use it through gradient descent. Xavier-init would inject noise on day 0 that the trunk would have to learn to denoise.
|
||||
|
||||
**Param tensor layout:** the direction Q-head weight tensor is one of the indices in `compute_param_sizes()` — implementer to identify exactly which index against current code at HEAD `6657e5626` per `feedback_trust_code_not_docs`. Its size grows by `branch_0_size`. Update `compute_param_sizes()` and the corresponding `layout_fingerprint_seed` entry. Per `pearl_build_rs_rerun_if_env_changed` and Invariant 8, fingerprint must bump (rename the entry suffix to indicate the new shape, e.g., `PARAM_W_DIR_QHEAD → PARAM_W_DIR_QHEAD_AUX1` or similar — implementer chooses convention matching the codebase).
|
||||
|
||||
**Forward kernel modifications:** the direction Q-head's forward SGEMM (likely in `gpu_dqn_trainer.rs` near the per-branch forward block) needs its `K` (input dim) bumped by 1 and an extra concat operation that copies the per-bar `softmax_diff` into the input vector. Implementer to choose between (a) a new fused concat-SGEMM kernel, or (b) a separate concat into a scratch buffer followed by the existing SGEMM. (a) is more efficient but more invasive; (b) reuses existing paths.
|
||||
|
||||
**Symmetry decision:** does the magnitude / order / urgency Q-head also see `softmax_diff`? **No, v1 keeps it direction-only.** The aux signal is directional — adding it to mag/ord/urg branches dilutes the signal and adds compute without clear benefit. If post-validation magnitude-conditional-on-direction shows benefit, that's a v2 lever.
|
||||
|
||||
### B.2 — Earned Gradient Flow pearl
|
||||
|
||||
The forward wire is one-way information flow by default. Backward gradient flow through the wire is **earned** by the aux head demonstrating label competence and the Q-head demonstrating it's not already aligned with aux. This is the new pearl — adaptive ISV-driven gating on backward-pass gradient flow, a new application of the codebase's existing Pearl pattern (which previously gated forward-pass features only).
|
||||
|
||||
#### B.2.1 — Mathematical form
|
||||
|
||||
```
|
||||
α_grad = σ(k_aux · sgn_open_1 · (aux_dir_acc_short_ema − threshold_1))
|
||||
× σ(k_q · (q_disagreement_short_ema − q_disagreement_baseline))
|
||||
× post_warmup_gate
|
||||
|
||||
where:
|
||||
σ(x) = 1 / (1 + exp(-clip(x, -30, 30))) # numerically safe sigmoid
|
||||
k_aux, k_q = ISV-adaptive sigmoid steepness
|
||||
sgn_open_1, threshold_1 = state-dependent (Schmitt trigger; see B.2.4)
|
||||
post_warmup_gate = linear ramp over first epoch of each fold
|
||||
```
|
||||
|
||||
`α_grad` is a single batch-mean scalar in `[0, 1]`. It scales the backward gradient column at the wire position (i.e., the new column in the direction Q-head's weight gradient that corresponds to the `softmax_diff` input). When `α_grad = 0`, the aux head receives no gradient from Q-loss (pure stop-gradient). When `α_grad = 1`, full Q-loss gradient flows back through the wire to the aux softmax outputs, then through aux head backward to the trunk.
|
||||
|
||||
#### B.2.2 — Gate 1: aux competence
|
||||
|
||||
**Driver signals:** `aux_dir_acc_short_ema` (ISV[373], existing) and `target_dir_acc` (ISV[372], existing).
|
||||
|
||||
**Behavior:** opens when aux head has demonstrated label accuracy above target. Closes when aux head's accuracy drops back below target.
|
||||
|
||||
**Threshold:** in v1, `threshold_1 = target_dir_acc` (so default behavior is to open at exactly target). The Schmitt-trigger (B.2.4) modifies this dynamically.
|
||||
|
||||
**Why this gate exists:** prevents Q-loss from hijacking aux training before aux has earned its label. The aux head's primary job is supervised directional prediction; co-training is a privilege earned by performing that primary job.
|
||||
|
||||
#### B.2.3 — Gate 2: Q-head disagreement (NEW signal)
|
||||
|
||||
**Driver signals:** `q_disagreement_short_ema` (NEW ISV slot) and `q_disagreement_baseline` (NEW ISV slot, long-EMA of disagreement to provide adaptive baseline).
|
||||
|
||||
**Per-step disagreement signal:**
|
||||
```
|
||||
q_disagreement[t] = (argmax(Q_dir_logits[t]) != argmax(aux_softmax[t])) ? 1.0 : 0.0
|
||||
```
|
||||
EMA'd over batch, then over steps via standard Pearl-A bootstrap pattern.
|
||||
|
||||
**Behavior:** opens when persistent disagreement is meaningfully above baseline. Closed when alignment is high (no co-training needed; aux signal already being consumed). Specifically:
|
||||
- High disagreement → aux signal is informative (Q-head WANTS to fight aux) → co-training would help align them
|
||||
- Low disagreement → Q-head already aligned with aux → no benefit from gradient flow
|
||||
|
||||
**Argmax source:** use the deterministic `argmax(Q_dir_logits)` at both training and eval time — i.e., the Q-head's preferred direction *before* Thompson sampling. This tracks the Q-head's underlying directional belief; the disagreement signal should not be polluted by exploration noise. Thompson-sampled action selection diverges from argmax stochastically (especially early in training when Q values are flat), so using the post-Thompson selection would conflate two signals: directional belief and exploration. We want the belief.
|
||||
|
||||
**Why this gate exists:** open Gate 1 alone would unlock co-training even when Q-head is already aligned with aux — wasted gradient flow. Gate 2 ensures co-training happens only when the system shows it's productive.
|
||||
|
||||
#### B.2.4 — Schmitt-trigger hysteresis on Gate 1
|
||||
|
||||
Single-threshold sigmoid oscillates around target by ±50% if `aux_dir_acc` jitters around target by training noise. Hysteresis prevents this:
|
||||
|
||||
```
|
||||
Two thresholds:
|
||||
threshold_1_open = target_dir_acc + 0.03 # cross above this from below to OPEN
|
||||
threshold_1_close = target_dir_acc - 0.03 # cross below this from above to CLOSE
|
||||
|
||||
State persistence:
|
||||
gate1_open_state ∈ {0, 1} (NEW ISV slot)
|
||||
|
||||
Logic:
|
||||
if gate1_open_state == 0 and aux_dir_acc_short_ema > threshold_1_open:
|
||||
gate1_open_state = 1
|
||||
elif gate1_open_state == 1 and aux_dir_acc_short_ema < threshold_1_close:
|
||||
gate1_open_state = 0
|
||||
|
||||
Sigmoid argument:
|
||||
if gate1_open_state == 1:
|
||||
arg_1 = aux_dir_acc_short_ema − threshold_1_close
|
||||
else:
|
||||
arg_1 = aux_dir_acc_short_ema − threshold_1_open
|
||||
```
|
||||
|
||||
Once open, gate stays open until aux drops 0.06 below the open-threshold (i.e., 0.03 below target). Once closed, stays closed until aux rises 0.06 above the close-threshold. The ±0.03 band is structural (numerical-stability anchor; not adaptive).
|
||||
|
||||
**State reset:** `gate1_open_state` resets to 0 at fold boundary via state-reset-registry. Otherwise stale gate state from fold N pollutes fold N+1.
|
||||
|
||||
#### B.2.5 — ISV-adaptive sigmoid steepness
|
||||
|
||||
Both `k_aux` and `k_q` self-adapt based on the variance of their respective driver signals:
|
||||
|
||||
```
|
||||
k_x = clip(k_base / (1 + variance_ema_x / variance_ref), [k_min, k_max])
|
||||
```
|
||||
|
||||
- **Welford EMA variance** of the driver signal computed per-step
|
||||
- High variance → noisy signal → low k → smoother sigmoid → less reactive to noise
|
||||
- Low variance → stable signal → high k → sharper sigmoid → more decisive
|
||||
|
||||
**Bounds:** `k_min = 1.0`, `k_max = 50.0`. These are structural numerical-stability anchors (Invariant 1 territory). The variance bound `variance_ref` is structural per-signal: e.g., `variance_ref_aux = 0.01` (1% deviation), `variance_ref_q = 0.05`.
|
||||
|
||||
**Per-fold reset:** variance EMAs reset to a sentinel that gives `k = k_base` at fold boundary (e.g., `var_ema = variance_ref` so initial `k = k_base / 2`). Without reset, fold N's variance carries into fold N+1's sigmoid steepness during cold start.
|
||||
|
||||
#### B.2.6 — Per-epoch warmup ramp
|
||||
|
||||
```
|
||||
post_warmup_gate = min(steps_in_fold / WARMUP_STEPS, 1.0)
|
||||
```
|
||||
|
||||
where `WARMUP_STEPS` is one epoch's worth of training steps (structural constant, derived from `config.steps_per_epoch`). This ramps the gate from 0 → 1 linearly over the first epoch of each fold, avoiding Δα discontinuity at the warmup boundary.
|
||||
|
||||
**State source:** `steps_in_fold` is already tracked by the trainer for other purposes; reuse it.
|
||||
|
||||
#### B.2.7 — Anti-gradient-hacking circuit breaker
|
||||
|
||||
The pathological dynamic to defend against: Q-head learns that disagreeing with aux unlocks aux co-training, which lets Q-loss pull aux toward Q-favored features, which makes aux less label-aligned, which → Q-head disagrees more → cycle. This is mesa-optimization risk.
|
||||
|
||||
**Detection:** track `aux_dir_acc_post_open_min` (NEW ISV slot — lowest aux_dir_acc since gate 1 opened in current fold). Reset when gate 1 closes.
|
||||
|
||||
**Trigger:** if `aux_dir_acc_post_open_min < threshold_1_open - 0.05` AND `q_disagreement_short_ema > q_disagreement_baseline + 0.10` simultaneously → suspected gradient hacking → force gate 1 closed for `LOCKOUT_EPOCHS` (structural constant; 2 epochs).
|
||||
|
||||
**Implementation:** `gradient_hack_lockout_remaining` (NEW ISV slot, integer epoch count). When zero, normal gate behavior. When >0, force `gate1_open_state = 0` and decrement at each epoch boundary.
|
||||
|
||||
This is the "smart toddler with circuit breaker" pattern: the model can't game the wire if doing so triggers a forced reset.
|
||||
|
||||
### B.3 — ISV slot allocations
|
||||
|
||||
11 new ISV slots required:
|
||||
|
||||
| Slot name | Purpose | Reset at fold? |
|
||||
|---|---|---|
|
||||
| `Q_DISAGREEMENT_SHORT_EMA_INDEX` | Fast EMA of per-step disagreement | Yes (sentinel 0.5) |
|
||||
| `Q_DISAGREEMENT_LONG_EMA_INDEX` | Slow EMA, baseline source | Yes (sentinel 0.5) |
|
||||
| `Q_DISAGREEMENT_BASELINE_INDEX` | Adaptive baseline (= long-EMA after warmup) | Yes (structural 0.5 sentinel) |
|
||||
| `K_AUX_ADAPTIVE_INDEX` | Current `k_aux` value | Yes (sentinel `k_base`) |
|
||||
| `K_Q_ADAPTIVE_INDEX` | Current `k_q` value | Yes (sentinel `k_base`) |
|
||||
| `AUX_DIR_ACC_VARIANCE_EMA_INDEX` | Welford variance of aux_dir_acc | Yes (sentinel `variance_ref`) |
|
||||
| `Q_DISAGREEMENT_VARIANCE_EMA_INDEX` | Welford variance of disagreement | Yes (sentinel `variance_ref`) |
|
||||
| `GATE1_OPEN_STATE_INDEX` | Schmitt-trigger persistent state | Yes (0 — closed) |
|
||||
| `ALPHA_GRAD_INDEX` | Computed α_grad value (HEALTH_DIAG visibility) | Yes (0) |
|
||||
| `AUX_DIR_ACC_POST_OPEN_MIN_INDEX` | Anti-gradient-hacking detection | Yes (1.0 — no min observed) |
|
||||
| `GRADIENT_HACK_LOCKOUT_REMAINING_INDEX` | Circuit-breaker epoch counter | Yes (0 — no lockout) |
|
||||
|
||||
All slots register in `state_reset_registry.rs` with appropriate sentinels. Slot indices to be allocated in `sp14_isv_slots.rs` (NEW file; mirrors `sp13_isv_slots.rs` pattern).
|
||||
|
||||
### B.4 — Kernels
|
||||
|
||||
Three new GPU kernels:
|
||||
|
||||
1. **`q_disagreement_update_kernel`** — per-step compute argmax mismatch from `Q_dir_logits` and `aux_nb_softmax_buf`, EMA update for short and long, Welford variance update. Block tree-reduce per `feedback_no_atomicadd`. ~80 LOC.
|
||||
2. **`alpha_grad_compute_kernel`** — read all driver signals, evaluate Schmitt-trigger logic for Gate 1, compute both sigmoids, multiply with warmup gate, write `ALPHA_GRAD_INDEX`. Includes adaptive k_aux/k_q computation from variance EMAs. ~60 LOC.
|
||||
3. **`gradient_hack_detect_kernel`** — read aux_dir_acc, gate1_open_state, q_disagreement, decide lockout state, write to `GRADIENT_HACK_LOCKOUT_REMAINING_INDEX` and force-clear `GATE1_OPEN_STATE_INDEX` if triggered. ~40 LOC.
|
||||
|
||||
Plus modifications to:
|
||||
- **Direction Q-head forward kernel** (or new fused concat-SGEMM): in_dim+1, read `softmax_diff` per-bar from `aux_nb_softmax_buf`. ~50-100 LOC.
|
||||
- **Direction Q-head backward kernel**: scale the gradient column at the wire position by `α_grad` before SAXPY into aux head's softmax gradient. ~30 LOC.
|
||||
- **`compute_param_sizes()`** and **`layout_fingerprint_seed()`**: bump direction Q-head weight tensor size by `branch_0_size`, rename layout entry. ~10 LOC.
|
||||
- **Trainer construction**: zero-initialize the new weight column. ~5 LOC.
|
||||
|
||||
### B.5 — HEALTH_DIAG observability
|
||||
|
||||
New per-epoch `pearl_egf_diag` line:
|
||||
```
|
||||
pearl_egf_diag α=0.45 gate1=0.92 gate2=0.49 k_aux=15.3 k_q=8.7
|
||||
q_dis_short=0.523 q_dis_long=0.498 q_dis_baseline=0.50
|
||||
gate1_state=open aux_post_open_min=0.55 lockout_remaining=0
|
||||
warmup=1.0
|
||||
```
|
||||
|
||||
Without these emit fields, diagnosing α_grad behavior in the smoke logs is forensic-only. All 11 ISV slots emit per epoch.
|
||||
|
||||
### B.6 — Hard rules upheld
|
||||
|
||||
- `feedback_no_partial_refactor` — wire (forward) and pearl (backward) land atomically. Cannot ship one without the other.
|
||||
- `feedback_no_atomicadd` — all three new kernels use block tree-reduce. Welford variance specifically benefits from this pattern (Welford's online algorithm is associative and parallelizable).
|
||||
- `feedback_cpu_is_read_only` — all signals computed GPU-side. CPU only reads ISV for HEALTH_DIAG emit.
|
||||
- `feedback_no_stubs` — every new ISV slot has a real producer and a real consumer in this commit. No zero-stub.
|
||||
- `feedback_isv_for_adaptive_bounds` — adaptive bounds (k_aux, k_q, q_disagreement_baseline) live in ISV. Numerical-stability anchors (k_min, k_max, variance_ref, hysteresis ±0.03 band, lockout epochs) are structural constants per Invariant 1.
|
||||
- `feedback_trust_code_not_docs` — implementer verifies all line numbers and structures against current code at HEAD `6657e5626` before editing.
|
||||
- `feedback_no_legacy_aliases` — fingerprint entry rename has no `_DEPRECATED` shim.
|
||||
- `pearl_first_observation_bootstrap` — Pearl-A bootstrap applied to all new EMAs (q_disagreement, variances, post_open_min).
|
||||
- `pearl_bounded_modifier_outputs_require_structural_activation` — softmax IS the structural activation; α_grad's sigmoid composition produces structurally-bounded `[0, 1]`.
|
||||
- `pearl_zscore_normalization_for_magnitude_asymmetric_signals` — variance-driven k adjustment is the appropriate analog.
|
||||
- `pearl_event_driven_reward_density_alignment` — N/A; this is a backward-pass gradient gate, not a per-step reward shaping.
|
||||
- `feedback_no_quickfixes` — the pearl is the canonical answer to "stop-gradient vs full-gradient" tradeoff, not a hack to make Q-head train.
|
||||
|
||||
### B.7 — Validation criteria
|
||||
|
||||
**Smoke A2 (5-epoch L40S, post-SP14):**
|
||||
- `α_grad` emits per epoch in HEALTH_DIAG; observable across all folds
|
||||
- `gate1_open_state` opens (=1) at least once during fold 1 (when aux_dir_acc reached 0.61 in pre-SP14 smoke)
|
||||
- `q_disagreement_short_ema` non-zero (at random init, expect ~0.5; with training, expect drift toward agreement or persistent disagreement)
|
||||
- `aux_dir_acc` reaches target (0.55) in fold 1 — confirms aux head still trains on its label even with new wire
|
||||
- `val_dir_dist` remains balanced (Hold-pricing still works)
|
||||
- `GRAD_CLIP_OUTLIER` count < 100 (down from 1109 — assumes A's C51 floor lands first)
|
||||
- No new pathological behaviors: aux_dir_acc doesn't drop catastrophically; circuit breaker `lockout_remaining` stays at 0 in normal operation
|
||||
|
||||
**30-epoch validation A (post-Smoke-A2-pass):**
|
||||
- `val_win_rate` rises above 0.50 by ep 20-25 — confirms the user's hypothesis that aux signal can move WR when wired
|
||||
- If WR stays at 45-48% across 30 epochs even with α_grad demonstrably opening: hypothesis falsified at this layer; next intervention is on the aux signal richness (sub-project C — adaptive LR, OR longer prediction horizons, OR microstructure feature additions)
|
||||
- `val_sharpe` recovers toward B1.0 baseline (~30) or stabilizes positive — assesses whether the new wire contention with Q-loss is destabilizing
|
||||
|
||||
**Multi-seed validation (post-30-epoch-pass):**
|
||||
- 3-seed × 30-epoch reproducibility check
|
||||
- Validates the WR climb is signal, not seed-luck
|
||||
|
||||
### B.8 — Implementation cost
|
||||
|
||||
| Component | LOC |
|
||||
|---|---|
|
||||
| 11 new ISV slots + constants in `sp14_isv_slots.rs` | ~80 |
|
||||
| `q_disagreement_update_kernel` (NEW) | ~80 |
|
||||
| `alpha_grad_compute_kernel` (NEW) | ~60 |
|
||||
| `gradient_hack_detect_kernel` (NEW) | ~40 |
|
||||
| Direction Q-head forward modification (in_dim+1, concat) | ~100 |
|
||||
| Direction Q-head backward gradient scaling | ~30 |
|
||||
| `compute_param_sizes` + `layout_fingerprint_seed` updates | ~20 |
|
||||
| `state_reset_registry` entries (11 new) | ~50 |
|
||||
| HEALTH_DIAG emit | ~30 |
|
||||
| Trainer construction (zero-init new weight column) | ~10 |
|
||||
| Three orchestrator launchers (kernel launches + arg threading) | ~80 |
|
||||
| Audit doc B section | ~80 |
|
||||
| **Production code total** | **~660** |
|
||||
| GPU oracle unit tests (12-15 tests) | ~400 |
|
||||
| **Grand total** | **~1060** |
|
||||
|
||||
3-5 days of focused work for a fresh implementer with clear brief. Comparable in scope to SP13 B1.1a (which was 175 tool uses for the implementer agent — at the upper bound of what fits in a single session).
|
||||
|
||||
**Decomposition recommendation:** B should land as a single atomic commit per `feedback_no_partial_refactor`. The forward wire and the backward pearl are coupled (gating a non-existent wire is meaningless; an ungated wire bypasses the safety design). However, the spec writer (this doc) anticipates that the implementer may need to decompose internally into sub-commits the way B1 split into B1.0/B1.1a/B1.1b — the structure of "the wire alone gives degraded but valid behavior" is similar.
|
||||
|
||||
## Sub-project C — Adaptive LR (deferred)
|
||||
|
||||
Not in scope for this spec. Addressed only if post-SP14 30-epoch shows WR stuck at 45-48% despite `α_grad` opening and `gate1_open_state` showing healthy behavior. In that case, the next lever is per-loss-group LR (per `pearl_adam_normalizes_loss_weights` — the lever that escapes Adam normalization). Spec'd separately if needed.
|
||||
|
||||
## Open questions for implementer
|
||||
|
||||
These are decisions the implementer should make by reading current code (not pre-decided in this spec):
|
||||
|
||||
1. **Forward concat strategy** — fused concat-SGEMM kernel vs. separate concat-then-SGEMM vs. extending an existing concat kernel like `mag_concat_qdir`. Implementer to evaluate compute cost vs. code clarity.
|
||||
2. **Argmax source for q_disagreement** — Thompson selector output buffer vs. recomputing argmax inline in `q_disagreement_update_kernel`. Implementer to choose the cleaner integration with existing action-selection path.
|
||||
3. **Per-bar α vs batch-mean α** — spec ships batch-mean for v1. If profiling shows the gradient column scaling has high variance per-row, per-row α may be needed. Implementer flags this if observable.
|
||||
4. **Sub-decomposition** — if implementer agent capacity is at risk, may split into "wire only with stop-gradient hardcoded" + "pearl on top" as B1.1a/B1.1b style. Decide based on dispatch session size.
|
||||
|
||||
## References
|
||||
|
||||
- **Smoke A:** `smoke-test-qfxjw` on commit `6657e5626`, 22m 59s, 3-fold × 5-epoch
|
||||
- **Three-agent diagnostic findings:** in conversation 2026-05-05 between Smoke A completion and this spec
|
||||
- **Memory pearls referenced:**
|
||||
- `pearl_first_observation_bootstrap` — sentinel = 0; first observation replaces directly
|
||||
- `pearl_zscore_normalization_for_magnitude_asymmetric_signals` — variance-driven adaptive bounds
|
||||
- `pearl_bounded_modifier_outputs_require_structural_activation` — sigmoid/softmax as structural bounds
|
||||
- `pearl_event_driven_reward_density_alignment` — ruled out; backward-pass gating is not reward shaping
|
||||
- `pearl_adam_normalizes_loss_weights` — drives the C deferral decision
|
||||
- `feedback_isv_for_adaptive_bounds` — adaptive in ISV, structural anchors as constants
|
||||
- `feedback_no_partial_refactor` — atomic commit constraint
|
||||
- **Plan reference:** `docs/superpowers/plans/2026-05-04-sp13-redefine-success-for-predictive-skill.md` §B.T3 — the deferred Q-head input concat sketch (this spec implements that, with the EGF pearl as new addition)
|
||||
- **Prior SP13 chain commits:** `f934ea171` (P0a) → `bdc5cb8bb` (P0b) → `62ab8ed85` (B0) → `6a869ad36` (B0.1) → `75e94858c` (B1.0) → `7d10ea8b3` (B1.1a) → `6657e5626` (B1.1b)
|
||||
|
||||
## Next step
|
||||
|
||||
Spec self-review (placeholder scan, internal consistency, ambiguity check), then user review. After user approval, invoke `superpowers:writing-plans` to produce the implementation plan for sub-project B (and a smaller plan for sub-project A if A is to land first).
|
||||
Reference in New Issue
Block a user