gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

docs(sp22): H6 Phase 2 smoke verdict — FALSIFIED + Phase 3 framing

Browse Source 
Workflow `train-bw28b` on sp20-aux-h-fixed @ 71eab9a25 terminated at
epoch=1 end after 56m wall-clock per
`feedback_kill_runs_on_anomaly_quickly`.

Epoch 1: 490399 trades, 246196 wins, 244203 losses, PF=0.946 — WR =
50.20% (vs Phase 1 = 50.21%, Δ = -0.01pp). Squarely in the runbook's
pre-declared falsification band.

Mechanistic finding (stronger than just "WR didn't move"): the
action distribution is essentially bit-identical across Phase 1
([0,1] / sentinel 0.5) and Phase 2 ([-1,+1] / sentinel 0.0)
encodings — drift < 0.5% on every action bin (run-to-run noise
floor). The policy made the SAME action choices regardless of slot
121's encoding. State[121] has zero behavioral effect on action
selection in either encoding. `pred_tanh = 0.66` in both phases
confirms the aux head IS producing strongly directional predictions
and the bridge IS conducting them — the policy is just ignoring
them entirely.

Hypothesis refinement (vs Phase 2 spec's "encoder can't extract
directional alpha in 3 epochs"): the encoder's weights for state[121]
are effectively zero. This dim was added by H6 with only 3 epochs of
training, while the first 121 dims have had thousands of training
steps to develop meaningful weights. Slot 121's gradient leverage is
dwarfed by the trained dims regardless of input magnitude. This is a
new-dim cold-start weight-init problem, not an encoding problem.

Implication: amplitude scaling (Phase 2 spec's fallback suggestion)
won't help — encoder weights are already near-zero, gradient
propagation through them stays near-zero regardless of input scale.
The deeper fix routes aux signal through a path that BYPASSES the
cold-encoder problem.

Phase 2 wiring stays merged per `feedback_no_functionality_removal`:
the recentered encoding is the better choice on principle (matches
`pearl_first_observation_bootstrap`) even when the bridge isn't
producing measurable WR effect.

Pivot to H6 Phase 3 (combined per
`pearl_no_deferrals_for_complementary_fixes`):
- (α) Bypass-head: small linear head `aux_dir_prob → Q_dir_bias`
  summed into Q_dir output post-encoder (parallel skip connection).
- (β) Aux→Q-target shaping: inject aux conviction into the Bellman
  target at trade-close events. Event-driven per
  `pearl_event_driven_reward_density_alignment`; bypasses the
  encoder entirely via the training-signal path.

Distinct mechanisms, non-overlapping refactor scopes — pearl
prescribes one atomic plan.

Refs
────
- docs/plans/2026-05-12-sp22-h6-phase2-recenter.md (Phase 2 spec)
- pearl_first_observation_bootstrap (Phase 2 encoding rationale)
- pearl_event_driven_reward_density_alignment (β motivation)
- pearl_no_deferrals_for_complementary_fixes (combined plan)
- feedback_no_functionality_removal (keep Phase 2 wiring merged)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-12 23:50:09 +02:00
parent 71eab9a253
commit f7fc83879e
1 changed files with 93 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

93
docs/dqn-wire-up-audit.md
View File

@@ -16574,3 +16574,96 @@ Verdict criteria (per spec, to be evaluated post-smoke):
directional alpha from one state slot in 3 epochs. Pivot to
amplitude scaling (multiply state[121] write by a scalar > 1) or
deeper hypothesis.
#### Phase 2 smoke result (2026-05-12) — FALSIFIED, mechanistically strong
Workflow `train-bw28b` dispatched on `sp20-aux-h-fixed` @ `71eab9a25`
(Phase 2 recentering + orthogonal test-action-constant fix).
Terminated at `epoch=1` end after 56m wall-clock per
`feedback_kill_runs_on_anomaly_quickly` (saved ~1h35m of L40S time).
**Epoch 1 outcome:**
```
GPU trades=490399 wins=246196 losses=244203 PF=0.946
```
WR = 246196 / 490399 = **50.20%** (vs Phase 1 = 50.21%, Δ = -0.01pp).
**Side-by-side telemetry comparison (Phase 1 vs Phase 2):**
| Signal | Phase 1 ([0,1] / sentinel 0.5) | Phase 2 ([-1,+1] / sentinel 0.0) |
|---|---|---|
| WR | 50.21% | **50.20%** |
| trades | 489,959 | 490,399 |
| PF | 0.947 | 0.946 |
| mean_loss | 2.366 | 2.369 |
| mean_q | -0.0073 | -0.0233 |
| `a_var [d/m/o/u]` | 0.0040 / 0.0003 / 0.0000 / 0.0000 | **0.0041 / 0.0002 / 0.0000 / 0.0000** |
| `v_share [d/m/o/u]` | 0.3934 / 0.3988 / 0.3958 / 0.3982 | **0.3906 / 0.3980 / 0.3972 / 0.3983** |
| `pred_tanh` (aux dir bias) | 0.6626 | 0.6639 |
| `dir_acc_short/long` | 0.2798 / 0.2801 | 0.2796 / 0.2801 |
| `hold_pct_ema` | 0.2004 | 0.2003 |
| `wr_ema` | 0.3911 | 0.3904 |
| `S_Small` exposure | 309,619 (3.8%) | **311,133 (3.8%)** |
| `H_Half` exposure | 33,713 (0.4%) | **33,867 (0.4%)** |
| `H_Full` exposure | 63,340 (0.8%) | **63,593 (0.8%)** |
| `L_Small` exposure | 341,432 (4.2%) | **343,045 (4.2%)** |
**Mechanistic finding (stronger than just "WR didn't move")**:
The **action distribution is essentially bit-identical** across the two
encodings (drift < 0.5% on every action bin — run-to-run noise floor).
This means the policy made the SAME action choices regardless of slot
121's encoding. State[121] has **zero behavioral effect** on action
selection.
`pred_tanh = 0.66` confirms the aux head IS producing strongly
directional predictions in both phases. The bridge IS conducting a
signal. The policy IS receiving it. **The policy is just ignoring it.**
**Hypothesis for why** (refines the Phase 2 spec's "encoder can't
extract directional alpha in 3 epochs"):
The encoder's weights for state[121] are effectively zero. This dim
was added by H6 — only got 3 epochs of training, while the first 121
dims have had thousands of training steps to develop meaningful
weights. Slot 121's gradient leverage is dwarfed by the trained dims
regardless of input magnitude. This is a **new-dim cold-start
weight-init problem**, not an encoding problem.
Implication: simply amplifying state[121]'s amplitude (the Phase 2
spec's first fallback suggestion) won't help if the bottleneck is
encoder weight initialization. The weights are already near-zero and
gradient propagation through them stays near-zero regardless of
input scale.
The deeper fix routes aux signal through a path that **bypasses**
the cold-encoder problem.
**Phase 2 wiring stays merged** per `feedback_no_functionality_removal`:
the recentered encoding is the better choice on principle (matches
`pearl_first_observation_bootstrap`) even when the bridge isn't
producing measurable WR effect. The wiring infrastructure is now in
place for any future fix that needs it.
**Pivot to H6 Phase 3** (combined plan per
`pearl_no_deferrals_for_complementary_fixes`): two non-overlapping
mechanisms that route aux around the cold-encoder:
- **(α) Bypass-head**: small dedicated linear head
`aux_dir_prob → Q_dir_bias` summed into the Q_dir output post-encoder.
Bypasses the main encoder's slow weight initialization for slot 121.
Architecturally a parallel skip connection.
- **(β) Aux→Q-target shaping**: inject aux conviction into the
Bellman target at trade-close events. Event-driven (aligns with
`pearl_event_driven_reward_density_alignment`); skips the encoder
entirely. Aux contribution arrives via training signal instead of
forward activation.
Distinct mechanisms, non-overlapping refactor scopes. Pearl prescribes
combining into one atomic plan.
If combined α+β fails: amplitude scaling or per-dim adaptive LR are
fallbacks; the action-distribution bit-identity finding suggests the
problem is structural, not numerical.