docs(sp22): H1 falsified + H6 aux→policy state bridge design
H1 result (commit9adbca826, smoke train-5zmkr, reverted ate8814079d): aux_dir_acc HD[2] = 78% with H=200 — aux head LEARNS direction at longer horizon. But policy WR stayed pinned at 50.1-50.2% — the learned aux signal does NOT propagate to policy (per pearl_separate_aux_trunk_when_shared_starves: aux on separate trunk with stop-grad to policy). H6 design (synthesized from all v5-v11 + H1 evidence): Wire the aux head's directional probability into the policy STATE as an input feature. - Policy gradient flows THROUGH the feature (uses it) - Stop-grad blocks gradient BACK (aux trunk unaffected, pearl preserved) - Uses existing padding slot [121..128) in STATE_DIM=128 (no layout growth) Why this is the structural fix: - Aux PROVED directional signal is in the features (78% at H=200) - Policy PROVED it can't extract direction (WR=50% across all v5-v11 conditions) - Bridge connects the two without violating trunk separation - Information-theoretic: gives policy a feature it provably can't compute itself Test outcome interpretation: WR > 50.5% → Mechanism 1 was binding (trunk separation gap) WR pinned → Mechanism 2 (reward density) or Mechanism 3 (V/A unidentifiability) dominates → H3 or V/A fix next Files changed: - docs/plans/2026-05-12-sp22-wr-plateau-investigation.md: H6 added as new primary hypothesis after H1; experiment order revised - docs/dqn-wire-up-audit.md: H6 design entry with three-mechanism synthesis Implementation scope (separate commit): 1. State layout: claim slot in padding [121..128) for aux_dir_prob 2. Aux trunk export: pull "up" probability per bar from aux forward 3. Experience collection: write aux_dir_prob into per-bar state 4. Stop-grad verification: confirm policy gradient blocked 5. Trade-open persistence: latch aux_dir_prob for trade duration Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -16210,3 +16210,69 @@ configuration. New SP arc needed:
|
||||
- **Validation method**: each hypothesis gets its own atomic smoke
|
||||
with one variable changed; compare against v9 baseline (174 peak val,
|
||||
WR=50.1%).
|
||||
|
||||
## 2026-05-12 — SP22 H6 design: aux→policy state bridge
|
||||
|
||||
### Synthesis from v5-v11 + H1 experiment
|
||||
|
||||
Five layers of evidence:
|
||||
1. Aux head at H=200 hits 78% directional accuracy within-fold (H1 HD[2])
|
||||
2. Aux drops to 49% on new folds (no cross-fold generalization)
|
||||
3. Aux is on a separate trunk with stop-grad to policy (per
|
||||
`pearl_separate_aux_trunk_when_shared_starves`)
|
||||
4. Policy WR pinned at 50.1-50.2% regardless of: epochs (3 vs 10),
|
||||
H value (1.7 vs 200), cascade controllers (alive vs not)
|
||||
5. Reward magnitudes: per-bar P&L ~1e-7, cost penalties ~1e-4 — costs
|
||||
are 1000× the signal (`pearl_event_driven_reward_density_alignment`
|
||||
predicts exposure-positive bias → exactly WR=50% / PF=1.0 pattern)
|
||||
|
||||
### Three coupled mechanisms
|
||||
|
||||
**Mechanism 1**: aux is walled off from policy (proven directional
|
||||
capability sits idle — `pearl_separate_aux_trunk_when_shared_starves`).
|
||||
|
||||
**Mechanism 2**: per-step reward density mismatched with event-driven
|
||||
objective (per `pearl_event_driven_reward_density_alignment`); per-step
|
||||
cost dominance overwhelms sparse directional alpha.
|
||||
|
||||
**Mechanism 3**: dueling V/A unidentifiability (`project_dueling_va_unidentifiable`);
|
||||
optimizer can park V at Q(Flat) → Q(Flat) attractor.
|
||||
|
||||
### H6 design
|
||||
|
||||
**Wire the aux head's directional probability into the policy STATE**:
|
||||
|
||||
- Read aux head's "up-probability" output during experience collection
|
||||
- Write to fresh state slot — use existing padding `[121..128)` in
|
||||
STATE_DIM=128 layout (no STATE_DIM growth needed)
|
||||
- Stop-grad on the wire — policy gradient flows through state into
|
||||
aux trunk MUST be blocked (preserves `pearl_separate_aux_trunk_when_shared_starves`)
|
||||
- Aux probability at trade-open persists for trade duration (avoids
|
||||
per-bar flicker; per the per-trade-vs-per-bar caveat)
|
||||
|
||||
### Why this design
|
||||
|
||||
- Addresses Mechanism 1 directly (aux ↔ policy bridge via state)
|
||||
- Does NOT touch reward composition (orthogonal to Mechanism 2 — follow-up)
|
||||
- Does NOT touch V/A architecture (orthogonal to Mechanism 3 — follow-up)
|
||||
- Information-flow only — no new optimizer dynamics
|
||||
- Information-theoretic: feeds policy a feature it provably can't
|
||||
compute itself (aux can; policy can't on the same input)
|
||||
|
||||
### Test outcomes
|
||||
|
||||
- WR rises past 50.5% → Mechanism 1 was binding constraint; aux signal
|
||||
is policy-usable when accessible
|
||||
- WR stays pinned → Mechanism 2 or 3 dominates; pivot to H3 (reward
|
||||
density) or fix V/A unidentifiability
|
||||
|
||||
### Engineering scope
|
||||
|
||||
1. State layout: claim slot in padding `[121..128)` for aux_dir_prob
|
||||
2. Aux trunk export: pull "up" probability per bar from aux forward
|
||||
3. Experience collection wireup: write aux_dir_prob into per-bar state
|
||||
4. Stop-grad verification: confirm policy gradient doesn't flow back to aux
|
||||
5. Trade-open persistence: latch aux_dir_prob into state for trade duration
|
||||
|
||||
Plan doc: `docs/plans/2026-05-12-sp22-wr-plateau-investigation.md`
|
||||
section H6.
|
||||
|
||||
@@ -118,16 +118,52 @@ Compare WR/PF against volume-bar baseline.
|
||||
**Cost**: Significant — different data ingestion path, retrain from
|
||||
scratch, may need hyperparam re-tuning. Longest experiment.
|
||||
|
||||
## Experiment order
|
||||
### H6 (added 2026-05-12 after H1 falsification): Aux→Policy state bridge
|
||||
|
||||
**Theory** (synthesized from H1 result + memory pearls): The aux head
|
||||
**proved it can extract directional signal at H=200 (78% accuracy
|
||||
within-fold)**, but `pearl_separate_aux_trunk_when_shared_starves`
|
||||
walls off the aux trunk from the policy trunk via stop-grad. The
|
||||
proven directional capability sits idle while the policy can't
|
||||
extract direction on its own (WR=50.1% across all conditions).
|
||||
|
||||
**The bridge**: Wire the aux head's directional probability into the
|
||||
policy's STATE as an input feature. Policy gradient flows THROUGH the
|
||||
feature (uses it) but NOT BACK (stop-grad → aux trunk unaffected).
|
||||
Uses one of the existing padding slots `[121..128)` in `STATE_DIM=128`.
|
||||
|
||||
**Test**: Add aux probability state slot → run smoke → watch WR. If
|
||||
WR breaks past 50.5%, mechanism 1 (trunk separation gap) was the
|
||||
binding constraint. If WR stays pinned, the bottleneck is downstream
|
||||
of policy state input (Mechanism 2 reward density or Mechanism 3
|
||||
V/A unidentifiability).
|
||||
|
||||
**Caveats** (must be scoped into implementation):
|
||||
1. Aux drops 78%→49% across fold boundaries (within-fold overfit at
|
||||
aux level). May need cross-fold stabilization or feed calibrated
|
||||
probability not argmax.
|
||||
2. Aux predicts per-bar; trades hold ~1.7 bars. Aux output at
|
||||
trade-open should persist for trade duration to avoid flicker.
|
||||
3. Reward density (Mechanism 2) might still kill the gradient even
|
||||
with direction info in state — follow-up scope.
|
||||
|
||||
**Cost**: ~1 day engineering (state layout, experience collection
|
||||
wireup, aux probability extraction) + 1 smoke ~30min.
|
||||
|
||||
## Experiment order (revised 2026-05-12)
|
||||
|
||||
Run in this priority order, killing early on each as soon as the
|
||||
hypothesis is conclusively resolved:
|
||||
|
||||
1. **H1 (label horizon)** — cheapest, addresses the most likely root cause
|
||||
2. **H3 (reward shape)** — cheap, complementary to H1
|
||||
3. **H2 (Hold disable)** — moderate cost
|
||||
4. **H4 (MTF features)** — higher engineering cost
|
||||
5. **H5 (bar resolution)** — last resort, biggest cost
|
||||
1. ✗ **H1 (label horizon)** — FALSIFIED 2026-05-12. Aux learns at
|
||||
H=200 (78% acc) but doesn't propagate to policy (trunk separation).
|
||||
2. **H6 (aux→policy state bridge)** — PRIMARY now. Bridges proven
|
||||
aux capability into policy.
|
||||
3. **H3 (reward shape)** — if H6 falsified, address per-step cost
|
||||
dominance (`pearl_event_driven_reward_density_alignment`).
|
||||
4. **H2 (Hold disable)** — if H3 inconclusive.
|
||||
5. **H4 (MTF features)** — higher engineering cost.
|
||||
6. **H5 (bar resolution)** — last resort.
|
||||
|
||||
Stop after the first hypothesis that produces WR ≥ 51% as a clear signal.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user