docs(sp22): H1 falsified + H6 aux→policy state bridge design

H1 result (commit 9adbca826, smoke train-5zmkr, reverted at e8814079d):
  aux_dir_acc HD[2] = 78% with H=200 — aux head LEARNS direction at
  longer horizon. But policy WR stayed pinned at 50.1-50.2% — the
  learned aux signal does NOT propagate to policy (per
  pearl_separate_aux_trunk_when_shared_starves: aux on separate
  trunk with stop-grad to policy).

H6 design (synthesized from all v5-v11 + H1 evidence):

  Wire the aux head's directional probability into the policy STATE
  as an input feature.
    - Policy gradient flows THROUGH the feature (uses it)
    - Stop-grad blocks gradient BACK (aux trunk unaffected, pearl
      preserved)
    - Uses existing padding slot [121..128) in STATE_DIM=128 (no
      layout growth)

  Why this is the structural fix:
    - Aux PROVED directional signal is in the features (78% at H=200)
    - Policy PROVED it can't extract direction (WR=50% across all
      v5-v11 conditions)
    - Bridge connects the two without violating trunk separation
    - Information-theoretic: gives policy a feature it provably
      can't compute itself

  Test outcome interpretation:
    WR > 50.5%  → Mechanism 1 was binding (trunk separation gap)
    WR pinned   → Mechanism 2 (reward density) or Mechanism 3 (V/A
                  unidentifiability) dominates → H3 or V/A fix next

Files changed:
  - docs/plans/2026-05-12-sp22-wr-plateau-investigation.md:
      H6 added as new primary hypothesis after H1; experiment order
      revised
  - docs/dqn-wire-up-audit.md: H6 design entry with three-mechanism
      synthesis

Implementation scope (separate commit):
  1. State layout: claim slot in padding [121..128) for aux_dir_prob
  2. Aux trunk export: pull "up" probability per bar from aux forward
  3. Experience collection: write aux_dir_prob into per-bar state
  4. Stop-grad verification: confirm policy gradient blocked
  5. Trade-open persistence: latch aux_dir_prob for trade duration

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-12 20:58:13 +02:00
parent e8814079d9
commit f50a974fb6
2 changed files with 108 additions and 6 deletions

View File

@@ -16210,3 +16210,69 @@ configuration. New SP arc needed:
- **Validation method**: each hypothesis gets its own atomic smoke
with one variable changed; compare against v9 baseline (174 peak val,
WR=50.1%).
## 2026-05-12 — SP22 H6 design: aux→policy state bridge
### Synthesis from v5-v11 + H1 experiment
Five layers of evidence:
1. Aux head at H=200 hits 78% directional accuracy within-fold (H1 HD[2])
2. Aux drops to 49% on new folds (no cross-fold generalization)
3. Aux is on a separate trunk with stop-grad to policy (per
`pearl_separate_aux_trunk_when_shared_starves`)
4. Policy WR pinned at 50.1-50.2% regardless of: epochs (3 vs 10),
H value (1.7 vs 200), cascade controllers (alive vs not)
5. Reward magnitudes: per-bar P&L ~1e-7, cost penalties ~1e-4 — costs
are 1000× the signal (`pearl_event_driven_reward_density_alignment`
predicts exposure-positive bias → exactly WR=50% / PF=1.0 pattern)
### Three coupled mechanisms
**Mechanism 1**: aux is walled off from policy (proven directional
capability sits idle — `pearl_separate_aux_trunk_when_shared_starves`).
**Mechanism 2**: per-step reward density mismatched with event-driven
objective (per `pearl_event_driven_reward_density_alignment`); per-step
cost dominance overwhelms sparse directional alpha.
**Mechanism 3**: dueling V/A unidentifiability (`project_dueling_va_unidentifiable`);
optimizer can park V at Q(Flat) → Q(Flat) attractor.
### H6 design
**Wire the aux head's directional probability into the policy STATE**:
- Read aux head's "up-probability" output during experience collection
- Write to fresh state slot — use existing padding `[121..128)` in
STATE_DIM=128 layout (no STATE_DIM growth needed)
- Stop-grad on the wire — policy gradient flows through state into
aux trunk MUST be blocked (preserves `pearl_separate_aux_trunk_when_shared_starves`)
- Aux probability at trade-open persists for trade duration (avoids
per-bar flicker; per the per-trade-vs-per-bar caveat)
### Why this design
- Addresses Mechanism 1 directly (aux ↔ policy bridge via state)
- Does NOT touch reward composition (orthogonal to Mechanism 2 — follow-up)
- Does NOT touch V/A architecture (orthogonal to Mechanism 3 — follow-up)
- Information-flow only — no new optimizer dynamics
- Information-theoretic: feeds policy a feature it provably can't
compute itself (aux can; policy can't on the same input)
### Test outcomes
- WR rises past 50.5% → Mechanism 1 was binding constraint; aux signal
is policy-usable when accessible
- WR stays pinned → Mechanism 2 or 3 dominates; pivot to H3 (reward
density) or fix V/A unidentifiability
### Engineering scope
1. State layout: claim slot in padding `[121..128)` for aux_dir_prob
2. Aux trunk export: pull "up" probability per bar from aux forward
3. Experience collection wireup: write aux_dir_prob into per-bar state
4. Stop-grad verification: confirm policy gradient doesn't flow back to aux
5. Trade-open persistence: latch aux_dir_prob into state for trade duration
Plan doc: `docs/plans/2026-05-12-sp22-wr-plateau-investigation.md`
section H6.

View File

@@ -118,16 +118,52 @@ Compare WR/PF against volume-bar baseline.
**Cost**: Significant — different data ingestion path, retrain from
scratch, may need hyperparam re-tuning. Longest experiment.
## Experiment order
### H6 (added 2026-05-12 after H1 falsification): Aux→Policy state bridge
**Theory** (synthesized from H1 result + memory pearls): The aux head
**proved it can extract directional signal at H=200 (78% accuracy
within-fold)**, but `pearl_separate_aux_trunk_when_shared_starves`
walls off the aux trunk from the policy trunk via stop-grad. The
proven directional capability sits idle while the policy can't
extract direction on its own (WR=50.1% across all conditions).
**The bridge**: Wire the aux head's directional probability into the
policy's STATE as an input feature. Policy gradient flows THROUGH the
feature (uses it) but NOT BACK (stop-grad → aux trunk unaffected).
Uses one of the existing padding slots `[121..128)` in `STATE_DIM=128`.
**Test**: Add aux probability state slot → run smoke → watch WR. If
WR breaks past 50.5%, mechanism 1 (trunk separation gap) was the
binding constraint. If WR stays pinned, the bottleneck is downstream
of policy state input (Mechanism 2 reward density or Mechanism 3
V/A unidentifiability).
**Caveats** (must be scoped into implementation):
1. Aux drops 78%→49% across fold boundaries (within-fold overfit at
aux level). May need cross-fold stabilization or feed calibrated
probability not argmax.
2. Aux predicts per-bar; trades hold ~1.7 bars. Aux output at
trade-open should persist for trade duration to avoid flicker.
3. Reward density (Mechanism 2) might still kill the gradient even
with direction info in state — follow-up scope.
**Cost**: ~1 day engineering (state layout, experience collection
wireup, aux probability extraction) + 1 smoke ~30min.
## Experiment order (revised 2026-05-12)
Run in this priority order, killing early on each as soon as the
hypothesis is conclusively resolved:
1. **H1 (label horizon)**cheapest, addresses the most likely root cause
2. **H3 (reward shape)** — cheap, complementary to H1
3. **H2 (Hold disable)** — moderate cost
4. **H4 (MTF features)** — higher engineering cost
5. **H5 (bar resolution)** — last resort, biggest cost
1. **H1 (label horizon)**FALSIFIED 2026-05-12. Aux learns at
H=200 (78% acc) but doesn't propagate to policy (trunk separation).
2. **H6 (aux→policy state bridge)** — PRIMARY now. Bridges proven
aux capability into policy.
3. **H3 (reward shape)** — if H6 falsified, address per-step cost
dominance (`pearl_event_driven_reward_density_alignment`).
4. **H2 (Hold disable)** — if H3 inconclusive.
5. **H4 (MTF features)** — higher engineering cost.
6. **H5 (bar resolution)** — last resort.
Stop after the first hypothesis that produces WR ≥ 51% as a clear signal.