From 976ab4bf1a05365dfc0231cf348a58f73e426aac Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Tue, 5 May 2026 21:43:46 +0200 Subject: [PATCH] =?UTF-8?q?docs(sp14):=20Smoke=20A2-B=20PASSED=20=E2=80=94?= =?UTF-8?q?=20Layer=20A+B=20chain=20validated,=202=20EGF=20bugs=20flagged?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Workflow smoke-test-z2kt7 on commit 26343cd57 succeeded in 22m57s. test result: ok. 1 passed; 0 failed; finished in 473.88s. Positive recovery signal — sharpe_ema trajectory: epoch 1: -24.05 (cold start) epoch 2: -9.12 epoch 3: +6.97 epoch 4: +16.14 (positive territory) Aux head bootstrapped from 6% to ~60% accuracy (above 50% random baseline). Layer B forward wire feeds aux signal into direction Q-head as designed. GRAD_CLIP_OUTLIER count: 455 (vs 1109 pre-fix Smoke A → 59% reduction) A.1's inv_a_std floor lift (1e-6 → 1e-3) bounded the amplifier. EGF GATE BUGS IDENTIFIED (Layer B follow-ups, not kill criteria): L1 — gate1 never opens: Schmitt trigger never fires "open" even when aux_dir_acc reached 0.62 (above target+0.03=0.58). The gate1_state slot (391) reads as 0 throughout the entire smoke. Possible causes: stale aux read, inverted threshold, slot corruption. L2 — post_open_min slot corrupted: Should be in [0, 1] but observed values 9.491, 27.981, 46.102. Slot 394 reads pulling garbage, likely typo or fold-reset misfire. Net effect: EGF is wired but behaviorally inactive — gate1 never opens, gradient_hack circuit breaker never fires, α_smoothed pinned at β_max via rate-limiter holding prior state. Wire-col scale at B.10 effectively passes through 95% of the gradient. Layer A's stability fixes were sufficient for the smoke to pass and produce a positive sharpe trajectory. Layer B's behavioral protection is currently a no-op pending L1 + L2 bug fixes. User's underlying hypothesis VALIDATED: the model learns the directional signal. Aux long_ema climbed from 0.06 to 0.61 in 4 epochs. The path from -24 to +16 sharpe validates the training mechanics enabled by A.1+A.2+A.3 + B forward wire. Recommendation: defer 30-epoch full validation until L1 + L2 are fixed, so EGF actually gates and the val numbers reflect real architectural protection. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/dqn-wire-up-audit.md | 55 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/docs/dqn-wire-up-audit.md b/docs/dqn-wire-up-audit.md index 381e5d218..350ab9cd5 100644 --- a/docs/dqn-wire-up-audit.md +++ b/docs/dqn-wire-up-audit.md @@ -6712,3 +6712,58 @@ B.8/B.9 grew `w_b0fc` to `[adv_h, SH2 + 1]` end-to-end across forward dispatch + **Next**: push to origin → submit Smoke A2-B → monitor with kill criteria above. +## SP14 Smoke A2-B — validation results (2026-05-05) + +**Workflow:** `smoke-test-z2kt7` on commit `26343cd57`. Duration: 22m57s. Status: **Succeeded**. + +**Test result:** `test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 1009 filtered out; finished in 473.88s` + +### Sharpe trajectory (positive recovery signal) + +| Epoch | sharpe_ema | aux_short | aux_long | +|---|---|---|---| +| 1 | -24.05 | 0.30 | 0.06 | +| 2 | -9.12 | 0.62 | 0.61 | +| 3 | +6.97 | 0.605 | 0.610 | +| 4 | +16.14 | 0.533 | 0.520 | + +Sharpe went from -24 → +16 across 4 epochs. Aux head bootstrapped from 6% to ~60% accuracy (above 50% random baseline) — Layer B forward wire is feeding signal into direction Q-head as designed. + +### GRAD_CLIP_OUTLIER count: 455 cumulative (vs 1109 pre-fix Smoke A → 59% reduction) + +A.1's structural floor lift (1e-6 → 1e-3) bounded the inv_a_std amplifier magnitude. The remaining 455 events reflect normal training-noise clipping at moderate magnitudes (17-19), not catastrophic spikes. + +### Layer B EGF gate — bugs identified, Layer B is wired but behaviorally inactive + +**Bug 1: gate1 never opens.** Across the entire smoke, `pearl_egf_diag` shows `gate1=closed` even when aux_dir_acc reached 0.62 (well above the Schmitt open threshold target+0.03=0.58). The Schmitt-trigger logic never fires. Possible causes: +- The kernel reads a stale aux_dir_acc_short value (pre-Pearl-A bootstrap) +- Threshold comparison inverted +- gate1_state slot (391) not being updated correctly across kernel launches + +**Bug 2: post_open_min slot corrupted.** Should be in [0, 1] (tracks lowest aux_dir_acc since gate opened). Observed values: 9.491, 27.981, 46.102. Indicates slot 394 read pulls garbage data — possibly: +- Wrong slot index in alpha_grad_compute_kernel or gradient_hack_detect_kernel reading from 394 +- Sentinel mismatch (post_open_min sentinel is 1.0; if reset isn't firing, slot accumulates from a different source) +- Kernel writes to slot 394 but with a wrong scaling + +**Net effect:** Because gate1 never fires "open", the gradient_hack_detect circuit breaker never triggers (no lockout); α_smoothed sits at β_max=0.95 because α_raw stays near 0 but the rate limiter holds the prior open-state value. The wire-col scale at B.10 effectively passes through 95% of the gradient. **EGF is wired but not actively protecting** in this smoke. + +### Layer B follow-ups (post-Smoke A2-B) + +| ID | Bug | Fix path | +|---|---|---| +| L1 | gate1 Schmitt never opens | Add HEALTH_DIAG of gate1_state value (currently only "open"/"closed" string); verify aux_short read matches the value alpha_grad sees; check threshold comparison | +| L2 | post_open_min values 9-46 | Audit slot 394 reads/writes for typo or wrong-slot collision; verify fold-reset registry sets sentinel 1.0 correctly | + +These are NOT kill criteria — Layer A's stability fixes were sufficient for the smoke to pass and produce a positive sharpe trajectory. Layer B's behavioral protection is currently a no-op pending bug fixes. + +### What this means for the user's hypothesis + +User asked at the start of this work: **does the model learn the directional signal?** Answer: yes. Aux head's long_ema climbed from 0.06 (epoch 1, near-zero) to 0.61 (epoch 4, well above 50% baseline). The Layer A+B chain enables this via: +- A.1: cap inv_a_std grad amplifier at 1000 (was 1e6) +- A.2: clamp aux_w to SP13 P0b range (was masking deficit signal) +- A.3: stagnation gate skips epoch 0 of fold (was triggering on cold-start) +- B forward wire: aux_softmax_diff feeds direction Q-head input +- B backward (currently un-gated due to bugs): gradient flows freely to aux + +The path from -24 sharpe to +16 sharpe in 4 epochs validates the underlying training mechanics. **30-epoch full validation should be deferred** until L1 + L2 are fixed, so the EGF pearl actually does work and the val numbers reflect genuine architectural improvement. +