diff --git a/docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md b/docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md index 48b96fe4a..8caadba51 100644 --- a/docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md +++ b/docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md @@ -28,9 +28,9 @@ Each milestone has explicit exit criteria. Implementation does not declare "done ### Tier 1: Convergence discipline (primary, must pass first) -- Multi-seed training (N=5 seeds) produces val-metric distribution with `std / mean < 0.15` on stable epochs (post-warmup). -- Multi-fold walk-forward (K=6 folds) shows no fold-1 grad explosions, no Q saturation at `abs_half`, no cross-fold state-reset bugs. -- Training halts self-terminates on any HEALTH_DIAG metric out-of-band (regression detection, A.4). +- Multi-seed training (N ≥ 5 seeds) produces val-metric distribution with `std / mean < 0.15` on stable epochs. "Stable epochs" = epochs `[warmup_end .. run_end)` where `warmup_end` is determined by the seed-phase decay in B.3 (typically epochs 10–15 for a 60-epoch run) OR explicit flag `--warmup-end N` on the validation harness, whichever is later. +- Multi-fold walk-forward (K ≥ 6 folds) shows no fold-1 grad explosions, no Q saturation at `abs_half`, no cross-fold state-reset bugs. +- Training self-terminates on any HEALTH_DIAG metric out-of-band (regression detection, A.4). - No hot-path DtoH `memcpy` anywhere in the step loop (A.6 audit passes). ### Tier 2: Behavioural parity (val trades escape the Flat-trap) @@ -51,7 +51,7 @@ Each milestone has explicit exit criteria. Implementation does not declare "done ## 3. Cross-cutting invariants (non-negotiable) -Every feature added by this spec obeys all seven invariants. Violations are caught by pre-commit hooks and smoke tests. +Every feature added by this spec obeys all eight invariants. Violations are caught by pre-commit hooks and smoke tests. ### Invariant 1 — ISV-driven adaptive bounds @@ -67,6 +67,8 @@ Enforcement: every new module lands with a row in `docs/dqn-wire-up-audit.md` id ### Invariant 3 — GPU-only hot path +**Definition of "training step loop":** every function called from `fused_training.rs::step_fused` or its descendants that executes once per training step (i.e. 178 times per epoch at current batch size), including kernel invocations inside the captured CUDA graph (`adam_grad_child`, `forward_child`, `aux_grad_child`, etc.) and any host-side orchestration between them. Per-epoch code (update_eval_v_range, update_grad_balance_targets if moved to cold path, etc.) is NOT the hot path. Per-fold and per-checkpoint code is never hot path. + No `cudaMemcpy` / `memcpy_dtoh` / `memcpy_htod` calls inside the training step loop. Zero tolerance. The one allowed cross-boundary mechanism is **pinned + device-mapped memory**: allocated via `cuMemAllocHost + cuMemHostGetDevicePointer`. Kernel writes via the device pointer; host reads via the host pointer to the same physical page. No copy occurs. This is already the ISV pattern; it is promoted to the ONLY allowed per-step cross-boundary mechanism. @@ -100,6 +102,24 @@ Every commit that adds or modifies a component updates: Pre-commit hook rejects component-adding commits that don't touch these docs. +### Invariant 8 — Named dimensions, not indices + +Every dimension, slot, offset, or semantic position in a multi-field buffer has a **named constant** in its definition module. Production code references the name. Raw numeric indices (`[0]`, `[6]`, `[86]`) appear ONLY in the definition site — never in consumer code, kernels, or tests. + +Scope of this invariant: + +- **ISV slots:** names already exist (e.g. `V_CENTER_DIR_INDEX`). This invariant says ALL slot access uses the named constant. Spec's new slots (§6) each get a named constant. +- **Portfolio state:** `ps[0]` through `ps[29]` → named constants in `state_layout.cuh`: `PS_VALUE`, `PS_POSITION`, `PS_CASH`, `PS_ENTRY_PRICE`, …, `PS_PLAN_TARGET_BARS` (23), etc. No kernel reads `ps[23]` directly. +- **Plan ISV dims:** current `plan_isv[0..6)` → named: `PLAN_ISV_PROGRESS` (0), `PLAN_ISV_PNL_VS_TARGET` (1), `PLAN_ISV_PNL_VS_STOP` (2), `PLAN_ISV_ENTRY_CONVICTION` (3), `PLAN_ISV_CONVICTION_DRIFT` (4), `PLAN_ISV_REGIME_SHIFT` (5), plus D.6's new `PLAN_ISV_REMAINING_FRACTION` (6). +- **Plan params:** `plan_params[0..6)` → named: `PLAN_PARAM_TARGET_BARS`, `PLAN_PARAM_PROFIT_TARGET`, `PLAN_PARAM_STOP_LOSS`, `PLAN_PARAM_SCALE_AGGRESSION`, `PLAN_PARAM_CONVICTION`, `PLAN_PARAM_ASYMMETRY`. +- **State vector offsets:** `SL_MARKET_START`, `SL_OFI_START`, `SL_TLOB_START` (new, D.8), `SL_MTF_START`, `SL_PORTFOLIO_START`, `SL_PLAN_ISV_START` — all live in `state_layout.cuh`. +- **Branch indices:** `BRANCH_DIR` (0), `BRANCH_MAG` (1), `BRANCH_ORD` (2), `BRANCH_URG` (3). Already partially named in Rust (via `[f32; 4]` arrays) but kernel code uses raw 0/1/2/3 — migrate. +- **Action sub-indices:** `DIR_SHORT` (0), `DIR_HOLD` (1), `DIR_LONG` (2), `DIR_FLAT` (3); `MAG_QUARTER` (0), `MAG_HALF` (1), `MAG_FULL` (2). + +Rationale: raw indices are the other half of the "hardcoded tuned constant" problem — they encode semantic position as a number. A reader can't tell what `ps[23]` means without grepping. Named constants make the meaning local to the reader. + +Enforcement: audit pass during A.1 (State Reset Registry) and A.2 (ISV Contract) picks up and names every raw index. New constants added to `docs/dqn-named-dims.md` — a simple table of name → index → purpose — and lint via grep: any `*.rs` or `*.cu` file that accesses a slot by raw index outside the definition module flags a pre-commit warning. + --- ## 4. Architecture @@ -139,20 +159,21 @@ Design: - Consumer list (which modules read) - Enforced via module visibility + a single audit table in `docs/isv-slots.md`. - ISV slot `[0]` reserved as schema version (u32 bumped on any slot-layout change). -- Checkpoint load: reads ISV[0] version, migrates or fails explicitly — no silent schema drift. -- Reserve slots [36..48) for second-order ISV signals (Part D, diagnostic-on-ISV, plan temporal dynamics). +- Checkpoint load: reads ISV[0] version. **Initial behaviour is fail-fast** — mismatch triggers explicit error with instructions. Migration helpers (slot-remap functions) added only when a specific migration need arises; not built speculatively. +- ISV slot allocation laid out in full in §6. New ISV slots claimed in this spec fit within the `[37..72)` range; older allocations `[0..37)` unchanged. #### A.3 Multi-Seed / Multi-Fold Validation Harness Current: one seed, one fold triggered by `argo-train.sh`, metrics eyeballed. Signal-to-noise terrible. -Design: `argo-train.sh --multi-seed N --folds K` spawns `N × K` training jobs from a single submission: +Design: `argo-train.sh --multi-seed N --folds K` spawns `N × K` training jobs from a single submission. Defaults: `N=5`, `K=6`. Per Invariant 6 these are minimums; running higher is always allowed. - Per-epoch metric aggregation across seeds: mean ± 2σ, median, worst-case. - HEALTH_DIAG emits cross-seed stats at epoch boundary. - Exit criteria use **pessimistic** estimator (mean − 2σ) not mean. - `norm_stats_foldN.json` merged across seeds to produce `norm_stats_fold_aggregate.json`. - Job orchestration via Argo Workflow DAG (existing pattern in `infra/k8s/argo/train-template.yaml`). +- Resource budget: with N=5, K=6, each job costing ~15-20min L40S time, the multi-seed harness costs ~8-10 GPU-hours per validation pass. Acceptable for pre-merge validation; not for every commit's smoke. #### A.4 Regression Detection @@ -249,7 +270,7 @@ When the policy is trading at task-normal rate, novelty is 0 and the bonus is in Current: PER buffer starts empty, fills via Boltzmann-driven DQN trajectories. At t=0 those trajectories are near-random but magnitude-biased toward the scripted plan MLP output, which produces mostly-losing trades → CQL learns "trading is bad" from a biased sample. -Design: before DQN training starts, seed the PER buffer with ~100K experiences drawn from a **portfolio of scripted policies**: +Design: before DQN training starts, seed the PER buffer with **≥ 100K experiences** (exact count sized to fill at least one PER batch × `REPLAY_SEED_EPOCHS` worth of training steps; typical is 100K-200K) drawn from a **portfolio of scripted policies**: - 40% pure uniform random (all 108 factored actions equiprobable, respecting dir/mag constraints). - 20% momentum-1 / momentum-3 / momentum-5 (simple N-bar momentum entry, fixed magnitude). @@ -258,7 +279,7 @@ Design: before DQN training starts, seed the PER buffer with ~100K experiences d Seed trajectories stored with lower initial PER priority than new DQN trajectories, displaced by DQN experiences over roughly 10–15 epochs. CQL alpha is coupled to the seed-phase decay — see C.5. -Per `feedback_no_feature_flags.md`, this is NOT a feature flag. The seed is always run for a fresh training job. Warm-restart from a checkpoint skips the seed step. +**Not a feature flag** — the seed step is unconditional for a fresh training job. The only exception is warm-restart from a checkpoint, where the replay buffer is restored from the checkpoint and the seed step is implicit in the checkpoint's history. This is a runtime condition (checkpoint present / absent), not a config toggle. Per `feedback_no_feature_flags.md`. #### B.4 Adaptive plan-MLP activation threshold @@ -301,9 +322,9 @@ Enables "did micro_reward actually contribute this epoch or is it silently zero? Current: training and val state distributions can diverge structurally (plan_isv was OOD for months before we caught it). Only detectable post-hoc. -Design: new ISV slot `STATE_KL_TRAIN_VAL_EMA` — running estimate of distributional divergence between train-state and val-state on a subset of state dimensions (portfolio + plan + ISV features). Computed at val backtest time by comparing per-dimension moments (mean, std). +Design: new ISV slot `STATE_KL_TRAIN_VAL_EMA` — running estimate of distributional divergence between train-state and val-state on a subset of state dimensions (portfolio + plan + ISV features). Computed at val backtest time by comparing per-dimension moments (mean, std), aggregated via a KL-like summary over the monitored dimensions. -When `KL > threshold` (also ISV-tracked, adaptive): training-time reward shaping (B.1, B.2) gets amplified automatically. Creates a feedback loop where distribution shift is self-correcting via training emphasis. +**Adaptive threshold:** a second ISV slot `STATE_KL_THRESHOLD_EMA` tracks the EMA of `STATE_KL_TRAIN_VAL_EMA` itself. "KL > threshold" means KL is ≥ 2× its own EMA — i.e. "currently divergent relative to historical baseline", self-calibrating. When the threshold fires, a third ISV slot `STATE_KL_AMPLIFICATION` is bumped from 1.0 toward 2.0 (bounded), which multiplies B.1 and B.2's reward magnitudes. Decays back toward 1.0 as KL normalises. All three slots are scratchpad (slot [58] and [63..65) in §6). #### C.4 Temporal timing bonus @@ -365,7 +386,7 @@ Benefits: - Fire-rate auditing is built-in (addresses `controller_activity` smoke's assertion that load-bearing controllers shouldn't fire in >50% of epochs). - Standardised diagnostics — the HEALTH_DIAG `controllers [anti_lr=true gamma=false clip=true …]` line becomes derived state, not manually-coded. -Biggest-scope piece in the spec. Executed as a series of internal commits (one per existing controller migrated to the protocol) before a final commit that removes the old scaffolding. +Biggest-scope piece in the spec. Executed as a series of internal commits — one per existing controller migrated to the protocol. Old scaffolding (the ad-hoc update functions being replaced) is removed in the SAME commit that migrates its last consumer to the new protocol, not in a deferred cleanup pass. Per `feedback_no_partial_refactor.md` — when a contract changes, every consumer migrates together. ### Part D — Temporal Architecture @@ -375,11 +396,13 @@ Current: Mamba2 + Liquid-TC in trunk; plan_isv state machine; conviction drift; Current: forward works, backward stubbed. -Design: properly wire Mamba2 backward — state-space gradients dx, dA, dB, dC through the recurrence. Substantial CUDA kernel work but load-bearing: every temporal improvement downstream assumes temporal learning actually propagates end-to-end. +Design: properly wire Mamba2 backward for the **DQN path specifically** — state-space gradients dx, dA, dB, dC through the recurrence as consumed by the DQN trunk. Substantial CUDA kernel work but load-bearing: every temporal improvement downstream assumes temporal learning actually propagates end-to-end. -Scope: fused backward kernel covering `mamba2_temporal_kernel.cu`. Validated via grad-norm smoke test (perturbation grad-check against finite difference). +Scope: fused backward kernel covering `mamba2_temporal_kernel.cu` for the DQN-integration call sites only. The same kernel infrastructure can be reused for supervised Mamba2 training, but that is not in this spec's scope. -This closes task #76 for the DQN path; TFT backward (also task #76) covered in Part E if TFT is adopted as auxiliary head. +Validation: grad-norm smoke test (perturbation grad-check against finite difference) on a small fixture; integration test verifying that DQN training with Mamba2 backward produces non-zero gradients propagating into Mamba2 weights over several training steps. + +This closes task #76 for the DQN path; TFT backward (task #76's other half) is addressed in Part E only if/when TFT is adopted as an auxiliary head. #### D.2 Per-branch gamma @@ -429,16 +452,20 @@ Smoother transitions preserve learned temporal signals across folds without comp #### D.6 Plan temporal dynamics enrichment -Current `plan_isv[6]` dimensions: progress, pnl-vs-target, pnl-vs-stop, entry-conviction, conviction-drift, regime-shift. +Current plan_isv has 6 dimensions `[0..6)`: progress, pnl-vs-target, pnl-vs-stop, entry-conviction, conviction-drift, regime-shift. That is `SL_PORTFOLIO_PLAN_DIM = 6` in `state_layout.cuh` and state positions `[86..92)` in the 96-dim state vector. -Design: extend to `plan_isv[7]` — add a new dimension: +Design: extend plan_isv to 7 dimensions `[0..7)` — add a new dimension at index 6: ``` -plan_isv[6] = (plan_target_bars − hold_time) / plan_target_bars - = remaining-fraction, clamped to [0, 1] +plan_isv[6] = clamp((plan_target_bars − hold_time) / plan_target_bars, 0.0, 1.0) + = remaining-fraction ``` -Directly exposes temporal pressure to the policy ("my plan is 80% done, should I take the profit I have?"). Trivial to compute; state-layout version bump. +Directly exposes temporal pressure to the policy ("my plan is 80% done, should I take the profit I have?"). Trivial to compute. Requires coordinated change: + +- `SL_PORTFOLIO_PLAN_DIM = 6 → 7` in `state_layout.cuh`. +- State vector grows from 96 to 97 dimensions (the new plan_isv[6] slots into position [92]; everything downstream shifts by one OR we leave a pad slot and account for it). Decision during impl: prefer padded no-shift to minimise downstream reshuffling. +- Schema version bump (A.2) paired with D.3 and D.8's similar layout impacts; all three land as one coordinated state-layout migration commit rather than three. #### D.7 Liquid Time-constant `liquid_mod` audit @@ -518,18 +545,43 @@ Deliverable: `docs/ml-supervised-to-dqn-concept-audit.md`. ### Landing order (suggested, non-mandatory within spec) -1. Part A.1, A.2, A.6 (state registry + ISV contract + hot-path audit — zero functional change but establishes the substrate). -2. A.5 orphan audit (discovery — zero functional change). -3. C.6 adaptive controller unification — big refactor but non-functional at the surface. -4. C.1, C.2, C.3 (quantile atoms, reward attribution, KL divergence signal — surgical additions). -5. B.1, B.2, B.3, B.4 (reward shaping + replay seed + adaptive plan threshold — behavioural change). -6. D.1, D.2, D.3 (temporal infrastructure — Mamba2 backward, per-branch gamma, horizon-decomposed V). -7. D.4, D.5, D.6, D.7, D.8 (temporal reward, soft transitions, plan dynamics, liquid audit, TLOB wire). -8. E.1–E.8 (supervised concept adoption; each a sub-PR). -9. A.3, A.4 (multi-seed harness + regression detection — validation substrate, no functional change to model). -10. Final validation pass: multi-seed × multi-fold across Tier 1, 2, 3 exit criteria. +Dependencies to respect: -Not all must be sequential — many are independent and can parallelise across worktrees. +- A.2 (ISV contract) before ANY feature claiming new ISV slots. +- A.1 (reset registry) before D.5 (soft fold transitions — extends the registry). +- D.1 (Mamba2 backward) before D.2, D.3, D.4, D.8 (anything that relies on temporal gradients propagating). +- C.6 (adaptive controller unification) before D.2 (per-branch gamma), D.7 (liquid audit) — these use the controller protocol. +- B.3 (replay seed) before C.5 (CQL α-seed coupling — needs the seed-phase signal). +- State-layout changes (D.3 horizon-decomposed V, D.6 plan[6], D.8 TLOB features) land as ONE coordinated commit with a single schema-version bump, not three. + +Suggested order respecting dependencies: + +1. **Substrate (zero functional change):** A.1, A.2, A.5, A.6. +2. **Refactor (non-functional):** C.6 adaptive controller unification. +3. **Surgical atoms:** C.1 quantile atom support. +4. **Core temporal:** D.1 Mamba2 backward — unlocks D.2–D.8. +5. **Temporal features:** D.2 per-branch γ, D.5 soft fold transitions, D.7 liquid audit. +6. **Coordinated state-layout migration:** D.3 + D.6 + D.8 in one commit. +7. **Reward shaping:** B.1, B.2, B.4 + C.4, D.4 (all reward components together since they share ISV slots). +8. **Replay warm-start:** B.3 + C.5 (coupled per C.5 design). +9. **Diagnostics:** C.2 reward attribution, C.3 KL divergence signal. +10. **Supervised concepts:** E.1–E.8 (each a sub-PR, in dependency order determined by E audit). +11. **Validation substrate:** A.3 multi-seed harness, A.4 regression detection, A.4.1 nsys harness. +12. **Final validation pass:** multi-seed × multi-fold across Tier 1, 2, 3 exit criteria. + +Many items in the same step are independent and parallelise across worktrees. The order is suggested; writing-plans may adjust based on concrete work estimation. + +### Spec decomposition for writing-plans + +This spec is large. The writing-plans skill may produce **multiple sequential implementation plans** rather than one monolithic plan — for example: + +- Plan 1: Substrate + refactor (items 1–3 above). +- Plan 2: Temporal core (items 4–6). +- Plan 3: Behavioural (items 7–9). +- Plan 4: Supervised concept audit (item 10) — one mini-plan per E sub-item. +- Plan 5: Validation substrate + final pass (items 11–12). + +Each plan ships independently; all plans are subordinate to this spec's invariants. Invariants enforced across plans via the pre-commit hooks (Invariant 7). --- @@ -570,6 +622,7 @@ New `ISV_TOTAL_DIM` = 72 (from 37). Network-facing `ISV_NETWORK_DIM` remains 23 - `docs/isv-slots.md` — ISV slot allocation registry with type/producer/consumers (A.2, Invariant 7). - `docs/dqn-gpu-hot-path-audit.md` — cross-boundary call classification (Invariant 3, 7). - `docs/ml-supervised-to-dqn-concept-audit.md` — Part E deliverable. +- `docs/dqn-named-dims.md` — named-constant registry (Invariant 8). - `config/metric-bands.toml` — regression-detection bands (A.4, Invariant 7). Pre-commit hook rejects component-adding commits that don't touch the relevant audit docs.