From 336ee40b9db0dbef382997c215c5d20f782e3507 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Fri, 24 Apr 2026 09:32:22 +0200 Subject: [PATCH] =?UTF-8?q?spec(dqn):=20add=20Invariant=209=20=E2=80=94=20?= =?UTF-8?q?no=20deferred=20work,=20no=20stubs,=20concrete=20E=20decisions?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit User-mandated addition: Invariant 9 — No deferred work, no stubs, no TODO/FIXME Every commit lands complete. No stubs (placeholder return values), no TODO/FIXME/XXX/HACK/TBD markers, no half-finished implementations. If it can't finish in this commit, it doesn't start. Stubs and deferrals train the network on semantic emptiness — invisible in convergence metrics but burn GPU time optimising against partly-fake signal. Authority: feedback_no_stubs.md, feedback_no_todo_fixme.md, feedback_no_quickfixes.md elevated to first-class spec enforcement. Pre-commit hook greps for forbidden markers. Part E decisions made concrete (per Invariant 9): - E.1 TFT VSN: IN (full VSN extension) - E.2 GRN: IN if A.5 audit finds absent (otherwise mark existing as canonical) - E.3 Multi-quantile heads: IN (5/25/50/75/95 quantile decomposition) - E.4 Encoder-decoder: IN (extends D.3 to full trunk/head separation) - E.5 Interpretability: IN (attention-weight ISV diagnostics) - E.6 Auxiliary heads: IN (next-return + regime-classification) - xLSTM: OUT (redundant with Mamba2+TLOB; YAGNI) - KAN: OUT (function approx not a bottleneck) No "evaluate later" items. Every concept either lands or explicitly doesn't, with rationale. §8 decisions tightened: - D.8 TLOB mode: decision criterion = per-step cost benchmark ≤ 5ms - C.6 controller order: fixed in spec (atoms→gamma→Kelly→cql_alpha→ tau→epsilon→conviction_floor→plan_threshold→balancer) - A.3 resource plan: auto-switch L40S→H100 at 12 GPU-hour threshold - A.4 metric band init: [mean − 3σ, mean + 3σ] from last 3 good runs No discretionary "we'll see" remain. All decisions data/order/budget/ statistic-driven. Counter updates: "seven invariants" → "nine" in §3 header and §5 cleanup contract. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../specs/2026-04-24-dqn-v2-unified-design.md | 71 ++++++++++++------- 1 file changed, 47 insertions(+), 24 deletions(-) diff --git a/docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md b/docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md index 8caadba51..d624dfdf0 100644 --- a/docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md +++ b/docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md @@ -51,7 +51,7 @@ Each milestone has explicit exit criteria. Implementation does not declare "done ## 3. Cross-cutting invariants (non-negotiable) -Every feature added by this spec obeys all eight invariants. Violations are caught by pre-commit hooks and smoke tests. +Every feature added by this spec obeys all nine invariants. Violations are caught by pre-commit hooks and smoke tests. ### Invariant 1 — ISV-driven adaptive bounds @@ -120,6 +120,20 @@ Rationale: raw indices are the other half of the "hardcoded tuned constant" prob Enforcement: audit pass during A.1 (State Reset Registry) and A.2 (ISV Contract) picks up and names every raw index. New constants added to `docs/dqn-named-dims.md` — a simple table of name → index → purpose — and lint via grep: any `*.rs` or `*.cu` file that accesses a slot by raw index outside the definition module flags a pre-commit warning. +### Invariant 9 — No deferred work, no stubs, no TODO/FIXME + +Every commit lands complete. No stubs (functions that return placeholder values instead of doing the work), no `TODO`, `FIXME`, `XXX`, `HACK`, or `TBD` markers in production code, no "we'll fix this later" deferrals, no half-finished implementations. + +**If it can't be finished in this commit, it doesn't start in this commit.** Either the feature is done (behaviourally correct, wired, tested, diagnostic-emitting) or the feature doesn't land. No partial state. + +Why this matters for a GPU RL trainer specifically: stubs that silently return placeholder Q-values, TODO-marked reward components that return zero, FIXME'd gradient paths that skip computation — these **train the network on semantic emptiness**. The model "learns" that certain inputs don't matter because the broken path reinforces ignoring them. Hours of L40S GPU time are burned optimising against a training signal that's partly fake. The bug is often invisible in metrics because training "converges" to something — just not to the policy we wanted. + +Authority: `feedback_no_stubs.md`, `feedback_no_todo_fixme.md`, `feedback_no_quickfixes.md`. This invariant elevates those three memory rules to first-class spec enforcement. + +Enforcement: pre-commit hook greps added code for `TODO|FIXME|XXX|HACK|TBD|unimplemented!|todo!` — any match rejects the commit. Stub detection is partly linted by the existing hook (`grep -n "return 0.0\|return 1.0\|return 0.5"` with `// ok: ` suppression); extended to cover new fallback patterns as they emerge. + +**Applies to spec implementation commits only.** Legit open questions in the spec itself (§8) are fine — they are design-time decisions documented with their decision criteria, not implementation-time stubs. + --- ## 4. Architecture @@ -503,26 +517,28 @@ Wiring: State-layout impact: adds D_tlob dimensions. Schema version bump (A.2) + state_layout.cuh coordinated change with D.6, D.3. -### Part E — Supervised→DQN Concept Audit +### Part E — Supervised→DQN Concept Adoption -Systematic review of ml-supervised concepts the project has already built, with classification for DQN adoption. +Concrete decisions (per Invariant 9, every E item either lands or does not — no "evaluate later"): -| Concept | Current DQN use | Adoption in spec | -|---|---|---| -| TFT Variable Selection Network (VSN) | Partial (`vsn_mag`, `vsn_dir` masks) | E.1: full TFT VSN over state features | -| Gated Residual Network (GRN) | Audit needed | E.2: if not present, adopt as trunk building block | -| TFT multi-quantile heads | Partial (IQN CVaR) | E.3: full quantile decomposition for risk branch | -| TLOB attention encoder | Orphan to DQN | D.8 (wired) | -| Mamba2 forward | Wired | — | -| Mamba2 backward | Partial | D.1 (wired) | -| Liquid Time-constant | Unclear | D.7 (audited) | -| xLSTM | Orphan | E.4: evaluate for long-horizon memory augmentation | -| KAN | Orphan | E.5: evaluate for function-approximation in small-data regime | -| Encoder-decoder separation | Single trunk | E.6: evaluate separating state-encoder from value-decoder | -| Attention-weight interpretability | Partial | E.7: ISV-expose attention diagnostics from all temporal heads | -| Multi-task auxiliary heads | None | E.8: add Q + next-return-prediction + regime-classification heads | +| Concept | Current DQN use | Decision | Action | +|---|---|---|---| +| TFT Variable Selection Network (VSN) | Partial (`vsn_mag`, `vsn_dir` masks) | **IN** | E.1 — extend existing masks to full TFT VSN over all state feature groups | +| Gated Residual Network (GRN) | Audit result TBD during A.5 | **IN if absent** | E.2 — if A.5 finds GRN not used in trunk, adopt as trunk building block; if present, mark existing impl as canonical in the audit | +| TFT multi-quantile heads | Partial (IQN CVaR only) | **IN** | E.3 — extend IQN branch to explicit multi-quantile decomposition (5/25/50/75/95 quantiles) | +| TLOB attention encoder | Orphan to DQN | **IN** | D.8 wires to DQN trunk | +| Mamba2 forward | Wired | — | already wired | +| Mamba2 backward | Partial | **IN** | D.1 completes | +| Liquid Time-constant | Unclear | **Audit-then-IN-or-DELETE** | D.7 decides; no third path | +| xLSTM | Orphan | **OUT** | Redundant with Mamba2 (state-space) + TLOB (attention); adding a third temporal encoder violates YAGNI. Leaving the supervised-crate impl as-is; no DQN wiring. | +| KAN | Orphan | **OUT** | Function-approximation is not a current bottleneck; Q-head dueling + C51 atoms already give sufficient expressivity. Leaving the supervised-crate impl as-is. | +| Encoder-decoder separation | Single trunk | **IN** | E.4 — D.3 (horizon-decomposed V) is a form of this; E.4 extends by separating the shared trunk into "state encoder" + per-branch "value decoders" with explicit interface | +| Attention-weight interpretability | Partial | **IN** | E.5 — ISV-expose attention weights from Mamba2, TLOB, and trunk heads; new diagnostic slots `ATTN_*_FOCUS_EMA` | +| Multi-task auxiliary heads | None | **IN** | E.6 — add auxiliary regression heads to trunk: next-bar return prediction + 5-bar regime classification. Trained via joint loss; gradients flow to trunk only (heads themselves are lightweight) | -Some E items ("evaluate") are investigation sub-tasks producing short design notes that feed follow-up PRs. Others ("wired") are concrete engineering work landing in this spec's rollout. +**Explicit OUT items (xLSTM, KAN):** the supervised-crate implementations remain in place for ml-supervised use. They are NOT orphans to this spec's scope because they serve their supervised consumers. The Wire-It-Up invariant applies to consumer paths per crate; xLSTM and KAN having zero DQN consumers is the correct state. + +E.1, E.2, E.3, E.4, E.5, E.6 are all in scope for this spec. Not "evaluate" — concrete engineering. Each lands wired, diagnostic, and fully finished per Invariant 9. Deliverable: `docs/ml-supervised-to-dqn-concept-audit.md`. @@ -539,7 +555,7 @@ Deliverable: `docs/ml-supervised-to-dqn-concept-audit.md`. ### Cleanup contract -- Every commit landed on main preserves all seven invariants. +- Every commit landed on main preserves all nine invariants. - Worktrees are removed after merge. No abandoned worktrees. - Orphan modules discovered during implementation are resolved in the same commit that discovers them (wired or deleted). @@ -629,12 +645,19 @@ Pre-commit hook rejects component-adding commits that don't touch the relevant a --- -## 8. Open questions (to resolve during implementation) +## 8. Decisions deferred to implementation (with decision criteria) -- D.8: TLOB end-to-end trainable vs frozen-with-side-stream retraining. Decide based on per-step cost benchmark during D.8 impl. -- E.4, E.5, E.6: xLSTM / KAN / encoder-decoder — each produces a short design note; adoption decided per-note, not in this spec. -- C.6: ordering of individual `AdaptiveController` migrations — atoms-first or gamma-first? Decide at the start of C.6's internal commit series based on which has the simplest ISV surface area. -- A.3 multi-seed: N = 5 seeds may be too expensive for L40S budget; may need to run the production-readiness validation on H100 pool instead. Resource plan finalised with A.3 impl. +Per Invariant 9, this spec has no "evaluate later" work items. The items below are **bounded decisions** — the design path is fixed, only the parameter point is chosen at implementation time with an explicit decision criterion written into the spec. + +- **D.8 TLOB training mode** — decision criterion: benchmark a single training step with TLOB end-to-end vs TLOB frozen. If end-to-end cost per step ≤ 5ms incremental over baseline, use end-to-end. If > 5ms, use frozen-with-side-stream (retrain TLOB via supervised job every N epochs on future-return prediction). No third option. Decision made in the D.8 commit message based on measured benchmark data. + +- **C.6 controller migration order** — decision: migrate in order of increasing ISV surface area (number of slots the controller reads/writes). Concretely: `atoms` → `gamma` → `Kelly cap` → `cql_alpha` → `tau` → `epsilon` → `conviction_floor` → `plan_threshold` → `balancer`. This order is now IN the spec; not deferred. + +- **A.3 multi-seed resource plan** — decision: default to L40S pool. If `N=5, K=6` total budget exceeds 12 GPU-hours (measured on the first multi-seed run), migrate to H100 pool (2-3× faster). The switch is automated via `argo-train.sh` based on measured budget. No human intervention. + +- **A.4 metric bands initial population** — decision: populate from the most recent 3 successful training runs' epoch-by-epoch metric distributions. Band = `[mean − 3σ, mean + 3σ]` per metric. Re-fit after every promotion of a new baseline run. + +All four decisions are concrete enough that implementation agents execute them without further design input. The decision criterion is data-driven (D.8), order-based (C.6), budget-based (A.3), or statistical (A.4) — no discretionary "we'll see" choices. ---