From 1c63c32297b5aa66cbc72baf75e39d2bec0feefd Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Sun, 26 Apr 2026 23:41:05 +0200 Subject: [PATCH] spec(dqn): reframe Phase 3 as direction-branch dead-code cleanup MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit User correctly challenged the "band-aid removal" framing. All fixes shipped during the val-Flat-collapse investigation addressed real bugs at their respective layers and should be preserved: - Kelly cap warm-branch (0c9d1ee39): post-decision physics layer. Thompson-independent. KEEP. - Train Return display + Sharpe annualization (non-tau parts of 7a3d88646): display/metric layer. Thompson-independent. KEEP. - Direction Boltzmann tau-floor (tau part of 7a3d88646) + adaptive eps_dir floor (d54b49efc): gates inside direction-branch action selection. Phase 2 replaces direction-branch action selection wholesale (eps-greedy + Boltzmann → Thompson), so these direction-only code paths become structurally unreachable. The latter two are NOT band-aids being removed because Thompson is better. They are dead code being cleaned up because Thompson replaces the surrounding mechanism. Magnitude/order/urgency branches keep their existing eps-greedy + Boltzmann + tau-floor + EPS_FLOOR paths intact. Reframed Phase 3 deliverable: "direction-branch dead-code cleanup" with explicit rationale (per feedback_no_legacy_aliases.md and feedback_no_partial_refactor.md). 0.5 day budget instead of 1. Also clarified eval action selection: argmax of (E[Q_C51]+E[Q_IQN])/2 is correct. Bellman backup is a Q-learning UPDATE rule, not an action-selection rule. Once Q is learned, optimal policy is greedy argmax of learned Q. Online Bellman lookahead at eval would require a forward model of market dynamics — not available, not standard. Co-Authored-By: Claude Opus 4.7 (1M context) --- ...26-distributional-rl-aggregation-design.md | 22 ++++++++++++------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/docs/superpowers/specs/2026-04-26-distributional-rl-aggregation-design.md b/docs/superpowers/specs/2026-04-26-distributional-rl-aggregation-design.md index 1d50c0e6c..9dcba4888 100644 --- a/docs/superpowers/specs/2026-04-26-distributional-rl-aggregation-design.md +++ b/docs/superpowers/specs/2026-04-26-distributional-rl-aggregation-design.md @@ -3,7 +3,10 @@ **Date**: 2026-04-26 **Status**: Design approved, awaiting implementation plans (4 sub-plans) **Origin**: Task #178 — Design proper C51 expected-Q Hold/Flat bias mitigation -**Predecessor symptom-layer fixes (revisited in Phase 3)**: Kelly cap warm-branch (`0c9d1ee39`), eps-floor adaptive boost (`d54b49efc`), ISV-adaptive Boltzmann tau (`7a3d88646`) +**Predecessor fixes from val-Flat-collapse investigation (all KEPT — real fixes for real bugs)**: +- Kelly cap warm-branch (`0c9d1ee39`) — post-decision physics layer +- Train Return display + Sharpe annualization (non-tau parts of `7a3d88646`) — display/metric layer +- Direction-branch Boltzmann tau-floor (tau part of `7a3d88646`) and adaptive eps_dir floor (`d54b49efc`) — gates inside direction-branch action selection. Phase 2 replaces direction-branch action selection wholesale (Thompson replaces eps-greedy + Boltzmann), so these gates become dead code FOR THE DIRECTION BRANCH ONLY. Magnitude/order/urgency keep their existing eps-greedy + Boltzmann + tau-floor + EPS_FLOOR paths intact. Phase 3 cleanup deletes only the direction-branch unreachable code, preserving all logic for other branches. --- @@ -297,19 +300,22 @@ Pearl entry to memory: `pearl_thompson_for_distributional_action_selection.md`. **Time budget**: 2-3 days. -### Phase 3 — Verification + Band-Aid Removal +### Phase 3 — Verification + Direction-Branch Dead-Code Cleanup **Deliverables**: - L4 multi-fold deploy (1 seed × 6 folds × 30 epoch) — long enough to see edge discovery via Thompson exploration - L5 full validation matrix (per Plan 5 Task 5): Tier 1/2/3 PASS -- **Band-aid audit + removal**: explicit task to revisit: - - eps-floor adaptive boost (`d54b49efc`): if Thompson generates sufficient directional exploration, REMOVE - - Boltzmann tau-floor on direction (`7a3d88646`): direction branch no longer uses Boltzmann (Thompson replaces it), so this code path becomes dead — REMOVE - - Kelly cap fix (`0c9d1ee39`): KEEP — addresses different layer (post-decision physics), still needed +- **Direction-branch dead-code cleanup** (NOT band-aid removal — these are real fixes whose direction-branch code paths simply become unreachable after Phase 2): + - **Kelly cap warm-branch fix (`0c9d1ee39`)**: KEEP. Post-decision physics layer; Thompson-independent. Real fix for a real bug; would break Long/Short trades again if reverted. + - **Train Return display + Sharpe annualization fixes (non-tau parts of `7a3d88646`)**: KEEP. Display/metric layer; Thompson-independent. Real fixes for real bugs. + - **Direction-branch Boltzmann tau-floor (tau-floor part of `7a3d88646`)**: code path becomes structurally unreachable for direction once Thompson replaces direction Boltzmann. The tau-floor remains active for magnitude/order/urgency branches (which keep Boltzmann). Cleanup pass: delete direction-only tau-floor code; keep mag/ord/urg paths intact. Not a band-aid removal — dead-code elimination per `feedback_no_legacy_aliases.md` and `feedback_no_partial_refactor.md`. + - **Adaptive eps_dir floor (`d54b49efc`)**: same — gates the random-sample path of direction-branch eps-greedy. Direction branch no longer uses eps-greedy under Thompson; this code becomes structurally unreachable for direction. Static `EPS_FLOOR=0.02` for magnitude/order/urgency branches stays. Cleanup pass: remove direction-only adaptive boost; keep the static EPS_FLOOR for other branches. - nsys regression check (Phase H continues independently; Thompson overhead measured) - Final architecture doc updates -**Time budget**: per Plan 5 existing budget + 1 day for band-aid removal. +**Why "cleanup" not "removal"**: every band-aid fix shipped during the val-Flat-collapse investigation addressed a real bug at its layer and should be preserved. Phase 2 replaces the direction-branch action-selection mechanism wholesale (eps-greedy + Boltzmann → Thompson). The eps_dir adaptive floor and the direction Boltzmann tau-floor are both inside the direction-branch action-selection path that Thompson replaces. Their direction-only code becomes dead — not because they were unnecessary, but because the larger surrounding mechanism changed shape. Dead code that runs but does nothing is a maintenance hazard (same anti-pattern as backward-compatibility shims for removed features). Phase 3 cleans it up. + +**Time budget**: per Plan 5 existing budget + 0.5 day for dead-code cleanup. --- @@ -419,7 +425,7 @@ The single 5-epoch L40S smoke is INSUFFICIENT to verify Thompson — long-term e | Plan A | Phase 0: TDD hypothesis + synthetic edge verification | 2 days | | Plan B | Phase 1: existing-lever audit | 1 day (longer if bug) | | Plan C | Phase 2: Thompson sampling integration | 2-3 days | -| Plan D | Phase 3: long verification + band-aid removal | per Plan 5 + 1 day | +| Plan D | Phase 3: long verification + direction-branch dead-code cleanup | per Plan 5 + 0.5 day | Each plan gets its own writing-plans cycle.