diff --git a/docs/dqn-wire-up-audit.md b/docs/dqn-wire-up-audit.md index ee4a03442..a51778a57 100644 --- a/docs/dqn-wire-up-audit.md +++ b/docs/dqn-wire-up-audit.md @@ -16210,3 +16210,69 @@ configuration. New SP arc needed: - **Validation method**: each hypothesis gets its own atomic smoke with one variable changed; compare against v9 baseline (174 peak val, WR=50.1%). + +## 2026-05-12 — SP22 H6 design: aux→policy state bridge + +### Synthesis from v5-v11 + H1 experiment + +Five layers of evidence: +1. Aux head at H=200 hits 78% directional accuracy within-fold (H1 HD[2]) +2. Aux drops to 49% on new folds (no cross-fold generalization) +3. Aux is on a separate trunk with stop-grad to policy (per + `pearl_separate_aux_trunk_when_shared_starves`) +4. Policy WR pinned at 50.1-50.2% regardless of: epochs (3 vs 10), + H value (1.7 vs 200), cascade controllers (alive vs not) +5. Reward magnitudes: per-bar P&L ~1e-7, cost penalties ~1e-4 — costs + are 1000× the signal (`pearl_event_driven_reward_density_alignment` + predicts exposure-positive bias → exactly WR=50% / PF=1.0 pattern) + +### Three coupled mechanisms + +**Mechanism 1**: aux is walled off from policy (proven directional +capability sits idle — `pearl_separate_aux_trunk_when_shared_starves`). + +**Mechanism 2**: per-step reward density mismatched with event-driven +objective (per `pearl_event_driven_reward_density_alignment`); per-step +cost dominance overwhelms sparse directional alpha. + +**Mechanism 3**: dueling V/A unidentifiability (`project_dueling_va_unidentifiable`); +optimizer can park V at Q(Flat) → Q(Flat) attractor. + +### H6 design + +**Wire the aux head's directional probability into the policy STATE**: + +- Read aux head's "up-probability" output during experience collection +- Write to fresh state slot — use existing padding `[121..128)` in + STATE_DIM=128 layout (no STATE_DIM growth needed) +- Stop-grad on the wire — policy gradient flows through state into + aux trunk MUST be blocked (preserves `pearl_separate_aux_trunk_when_shared_starves`) +- Aux probability at trade-open persists for trade duration (avoids + per-bar flicker; per the per-trade-vs-per-bar caveat) + +### Why this design + +- Addresses Mechanism 1 directly (aux ↔ policy bridge via state) +- Does NOT touch reward composition (orthogonal to Mechanism 2 — follow-up) +- Does NOT touch V/A architecture (orthogonal to Mechanism 3 — follow-up) +- Information-flow only — no new optimizer dynamics +- Information-theoretic: feeds policy a feature it provably can't + compute itself (aux can; policy can't on the same input) + +### Test outcomes + +- WR rises past 50.5% → Mechanism 1 was binding constraint; aux signal + is policy-usable when accessible +- WR stays pinned → Mechanism 2 or 3 dominates; pivot to H3 (reward + density) or fix V/A unidentifiability + +### Engineering scope + +1. State layout: claim slot in padding `[121..128)` for aux_dir_prob +2. Aux trunk export: pull "up" probability per bar from aux forward +3. Experience collection wireup: write aux_dir_prob into per-bar state +4. Stop-grad verification: confirm policy gradient doesn't flow back to aux +5. Trade-open persistence: latch aux_dir_prob into state for trade duration + +Plan doc: `docs/plans/2026-05-12-sp22-wr-plateau-investigation.md` +section H6. diff --git a/docs/plans/2026-05-12-sp22-wr-plateau-investigation.md b/docs/plans/2026-05-12-sp22-wr-plateau-investigation.md index 55f475ab1..9dc7d18cb 100644 --- a/docs/plans/2026-05-12-sp22-wr-plateau-investigation.md +++ b/docs/plans/2026-05-12-sp22-wr-plateau-investigation.md @@ -118,16 +118,52 @@ Compare WR/PF against volume-bar baseline. **Cost**: Significant — different data ingestion path, retrain from scratch, may need hyperparam re-tuning. Longest experiment. -## Experiment order +### H6 (added 2026-05-12 after H1 falsification): Aux→Policy state bridge + +**Theory** (synthesized from H1 result + memory pearls): The aux head +**proved it can extract directional signal at H=200 (78% accuracy +within-fold)**, but `pearl_separate_aux_trunk_when_shared_starves` +walls off the aux trunk from the policy trunk via stop-grad. The +proven directional capability sits idle while the policy can't +extract direction on its own (WR=50.1% across all conditions). + +**The bridge**: Wire the aux head's directional probability into the +policy's STATE as an input feature. Policy gradient flows THROUGH the +feature (uses it) but NOT BACK (stop-grad → aux trunk unaffected). +Uses one of the existing padding slots `[121..128)` in `STATE_DIM=128`. + +**Test**: Add aux probability state slot → run smoke → watch WR. If +WR breaks past 50.5%, mechanism 1 (trunk separation gap) was the +binding constraint. If WR stays pinned, the bottleneck is downstream +of policy state input (Mechanism 2 reward density or Mechanism 3 +V/A unidentifiability). + +**Caveats** (must be scoped into implementation): +1. Aux drops 78%→49% across fold boundaries (within-fold overfit at + aux level). May need cross-fold stabilization or feed calibrated + probability not argmax. +2. Aux predicts per-bar; trades hold ~1.7 bars. Aux output at + trade-open should persist for trade duration to avoid flicker. +3. Reward density (Mechanism 2) might still kill the gradient even + with direction info in state — follow-up scope. + +**Cost**: ~1 day engineering (state layout, experience collection +wireup, aux probability extraction) + 1 smoke ~30min. + +## Experiment order (revised 2026-05-12) Run in this priority order, killing early on each as soon as the hypothesis is conclusively resolved: -1. **H1 (label horizon)** — cheapest, addresses the most likely root cause -2. **H3 (reward shape)** — cheap, complementary to H1 -3. **H2 (Hold disable)** — moderate cost -4. **H4 (MTF features)** — higher engineering cost -5. **H5 (bar resolution)** — last resort, biggest cost +1. ✗ **H1 (label horizon)** — FALSIFIED 2026-05-12. Aux learns at + H=200 (78% acc) but doesn't propagate to policy (trunk separation). +2. **H6 (aux→policy state bridge)** — PRIMARY now. Bridges proven + aux capability into policy. +3. **H3 (reward shape)** — if H6 falsified, address per-step cost + dominance (`pearl_event_driven_reward_density_alignment`). +4. **H2 (Hold disable)** — if H3 inconclusive. +5. **H4 (MTF features)** — higher engineering cost. +6. **H5 (bar resolution)** — last resort. Stop after the first hypothesis that produces WR ≥ 51% as a clear signal.