diff --git a/docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md b/docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md new file mode 100644 index 000000000..48b96fe4a --- /dev/null +++ b/docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md @@ -0,0 +1,600 @@ +# DQN v2 — Unified Integrated Policy System + +**Date:** 2026-04-24 +**Status:** Design (awaiting user review before writing-plans) +**Scope:** DQN model only — consolidates every lesson from the val-Flat-collapse investigation, the task #92 gradient-pathology triad, the ISV v-range unification work, the f64→f32 ABI-trap cleanup, and the long-running hard lessons about convergence, fold boundaries, orphan features, and hardcoded tuning. + +--- + +## 1. Purpose + +This spec is not a patch to a specific failure mode. It is a **consolidation of the Foxhunt DQN architecture into one properly integrated system**, synthesising the concepts we already built in supervised, the adaptive controllers we accreted over many sessions, the ISV signal-bus pattern that emerged from the Apr-23 bundle, and the convergence-discipline lessons from multiple fold-1 grad explosions. + +Prior work bolted mechanisms on as problems surfaced. This spec replaces that with a single integrated design that: + +- Obeys a small set of non-negotiable invariants (§3). +- Wires every supervised concept the project has built into its intended DQN consumer path (Parts D, E). +- Unifies the many adaptive controllers under one protocol (Part C.6). +- Makes convergence the primary guardrail, not an after-thought (Part A). +- Eliminates orphan code, hot-path DtoH copies, and hardcoded tuning constants as a design principle (§3). + +The spec lands as one coordinated rollout. Internal commits during implementation are permitted (agents in worktrees, tree clean at every merge to main). Every commit respects the invariants. + +--- + +## 2. Success criteria (tiered milestones) + +Each milestone has explicit exit criteria. Implementation does not declare "done" until ALL tiers pass under the multi-seed / multi-fold harness (§4.A.3). + +### Tier 1: Convergence discipline (primary, must pass first) + +- Multi-seed training (N=5 seeds) produces val-metric distribution with `std / mean < 0.15` on stable epochs (post-warmup). +- Multi-fold walk-forward (K=6 folds) shows no fold-1 grad explosions, no Q saturation at `abs_half`, no cross-fold state-reset bugs. +- Training halts self-terminates on any HEALTH_DIAG metric out-of-band (regression detection, A.4). +- No hot-path DtoH `memcpy` anywhere in the step loop (A.6 audit passes). + +### Tier 2: Behavioural parity (val trades escape the Flat-trap) + +- Val trade count within factor-of-2 of training trade count (approximately >= 1% of val bars positioned, vs. the current 0.01%). +- `plan_isv` `active_frac` in val > 0.2 on average across the window (currently 0.000 — plans structurally fail to activate). +- No systematic bias toward Hold or Flat direction argmax (entropy of argmax direction distribution > 0.8 of uniform). + +### Tier 3: Profitability (val metric positive and stable) + +- Annualised val Sharpe > 1.0 on the validation window. +- Val WinRate ≥ 52% on trades > 500/window. +- Profit Factor ≥ 1.1 stable across the multi-seed runs. + +**No tier is skippable.** Tier 3 without Tier 1 means the result is not reproducible. Tier 2 without Tier 1 means the val-trade-count was a lucky seed. Tier 1 without Tier 2 means convergence is clean but the policy is uselessly conservative. + +--- + +## 3. Cross-cutting invariants (non-negotiable) + +Every feature added by this spec obeys all seven invariants. Violations are caught by pre-commit hooks and smoke tests. + +### Invariant 1 — ISV-driven adaptive bounds + +All adaptive clamps, thresholds, targets, and schedules live in the ISV signal bus and are read by consumers at runtime. No hardcoded multipliers, no tuned `clamp(0.25, 4.0)`, no `4 × median`, no `10 × q_gap`. Authority: `feedback_isv_for_adaptive_bounds.md`. + +The single carve-out is numerical-safety hard limits: values that express dimensional or numerical-stability reality (e.g. `abs_half = 0.5 × (v_max − v_min)`, `1/limit` for a safety floor). These are documented as such at the constant-definition site. + +### Invariant 2 — Wire-It-Up + +Every module, feature, or kernel built is wired into its intended production consumer path in the same commit. Orphan code — compiles but no production path consumes it — is either wired or deleted. Authority: `feedback_wire_everything_up.md`. + +Enforcement: every new module lands with a row in `docs/dqn-wire-up-audit.md` identifying its consumer. Pre-commit hook rejects commits that add a pub module without updating the audit doc. + +### Invariant 3 — GPU-only hot path + +No `cudaMemcpy` / `memcpy_dtoh` / `memcpy_htod` calls inside the training step loop. Zero tolerance. + +The one allowed cross-boundary mechanism is **pinned + device-mapped memory**: allocated via `cuMemAllocHost + cuMemHostGetDevicePointer`. Kernel writes via the device pointer; host reads via the host pointer to the same physical page. No copy occurs. This is already the ISV pattern; it is promoted to the ONLY allowed per-step cross-boundary mechanism. + +`memcpy` DtoH/HtoD is permitted only at cold path: checkpoint save/load, fold transitions, validation-window reshuffle at epoch boundaries. Every such call is annotated with `// cold-path: ok — ` and audited in `docs/dqn-gpu-hot-path-audit.md`. + +Authority: `feedback_gpu_cpu_roundtrip.md` (extended with zero-tolerance language in this spec). + +### Invariant 4 — Pinned-only host memory + +Any host-resident memory that backs a GPU data structure is pinned + device-mapped via `cuMemAllocHost_v2 + cuMemHostGetDevicePointer_v2`. Regular `malloc` for GPU-adjacent buffers is forbidden. UVM (`cudaMallocManaged`) is forbidden (page-fault stalls in the hot path). + +### Invariant 5 — Diagnostic-first + +Every adaptive controller, every reward component, every new state partition, every new ISV slot emits HEALTH_DIAG-compatible telemetry. If it's adaptive or consequential, it's diagnostic. + +Enforcement: `controller_activity` and `reward_component_audit` smoke tests extended with coverage assertions. Missing diagnostics fail smoke. + +### Invariant 6 — Convergence discipline (primary) + +No feature is "done" until it passes multi-seed (N ≥ 5) × multi-fold (K ≥ 6) under the validation harness (§4.A.3). "It worked on one run" is not a proof. + +### Invariant 7 — Wire-up + diagnostic audit per-commit + +Every commit that adds or modifies a component updates: + +- `docs/dqn-wire-up-audit.md` — consumer-path mapping per module. +- `docs/isv-slots.md` — ISV slot allocations with type, producer, consumers. +- `docs/dqn-gpu-hot-path-audit.md` — every cross-boundary call classified. +- `config/metric-bands.toml` — regression-detection bands for new metrics. + +Pre-commit hook rejects component-adding commits that don't touch these docs. + +--- + +## 4. Architecture + +The spec is decomposed into five Parts. All Parts land in one coordinated rollout; internal commits during implementation are permitted (worktree isolation for parallel agents, sequential merges to main). + +Parts A and B are load-bearing for everything else. Part A establishes the convergence substrate. Part B escapes the Flat-trap. Parts C, D, E are refinement, temporal unification, and supervised-concept reuse respectively. + +### Part A — Foundation (convergence substrate) + +#### A.1 State Reset Registry + +Current state: some state resets at fold boundaries (`eval_v_range`, target network hard copy), some persists (learned weights, Adam momentum, ISV history), some is undefined. The Fold-1 grad explosion (task #31) and the fold-boundary gap (task #84) trace to this ambiguity. + +Design: every piece of training-time state is classified into one of four categories via a typed `StateResetRegistry`: + +| Category | Meaning | Example | +|---|---|---| +| `FoldReset` | Cleared at fold boundary | Kelly stats, plan_state, eval_q_*_ema, v-range ISV slots | +| `WindowReset` | Cleared at validation-window boundary | Portfolio state, hold_time, entry_price | +| `SoftReset(decay_bars)` | Anneals toward bootstrap over decay_bars at fold boundary | adaptive_gamma, ISV grad-balance targets | +| `TrainingPersist` | Survives across folds (learned weights) | Network params, target network, Adam momentum | +| `SchemaContract` | Version-stable across runs | ISV slot allocations themselves | + +`reset_fold_state(fold_idx)` iterates the registry. Adding new state requires registering it with a category. Compile-time enforcement where possible via module visibility on a typed registry struct; runtime assertion in constructor. + +See D.5 for the SoftReset motivation (some temporal state benefits from annealing, not hard cuts). + +#### A.2 ISV Slot Contract + Schema Versioning + +Current: 37 slots, each `pub const X_INDEX: usize = N`. No producer/consumer contract, no version, no migration path. + +Design: + +- Introduce `IsvSlot` typed handles with: + - Producer identity (which module writes) + - Consumer list (which modules read) + - Enforced via module visibility + a single audit table in `docs/isv-slots.md`. +- ISV slot `[0]` reserved as schema version (u32 bumped on any slot-layout change). +- Checkpoint load: reads ISV[0] version, migrates or fails explicitly — no silent schema drift. +- Reserve slots [36..48) for second-order ISV signals (Part D, diagnostic-on-ISV, plan temporal dynamics). + +#### A.3 Multi-Seed / Multi-Fold Validation Harness + +Current: one seed, one fold triggered by `argo-train.sh`, metrics eyeballed. Signal-to-noise terrible. + +Design: `argo-train.sh --multi-seed N --folds K` spawns `N × K` training jobs from a single submission: + +- Per-epoch metric aggregation across seeds: mean ± 2σ, median, worst-case. +- HEALTH_DIAG emits cross-seed stats at epoch boundary. +- Exit criteria use **pessimistic** estimator (mean − 2σ) not mean. +- `norm_stats_foldN.json` merged across seeds to produce `norm_stats_fold_aggregate.json`. +- Job orchestration via Argo Workflow DAG (existing pattern in `infra/k8s/argo/train-template.yaml`). + +#### A.4 Regression Detection + +Current: a wildly wrong metric trains to completion; we catch it in logs. + +Design: per-metric historical bands in `config/metric-bands.toml`: + +```toml +[metric_bands.train_sharpe] +warn_low = -5.0 +warn_high = 200.0 +error_low = -50.0 +error_high = 1000.0 +``` + +HEALTH_DIAG emission checks every metric against its band. N consecutive out-of-band epochs → `WARN` log. 2N consecutive → `ERROR` + SIGTERM-self via `std::process::exit(1)`. Training workflow FAIL status; Argo notification fires. + +The bands are populated from the last 3 known-good runs' metric distributions. Updated as part of the promotion workflow when a new baseline run succeeds. + +##### A.4.1 nsys profile harness (sub-task) + +Add `--profile` flag to `argo-train.sh`. When set: + +- ci-builder image includes `nsys` (via apt install or pinned tarball). +- `train-template.yaml` wraps the training binary under `nsys profile --capture-range=cudaProfilerApi --output=profile-${SHORT_SHA}.nsys-rep`. +- .nsys-rep artifact pushed to `foxhunt-training-artifacts` MinIO bucket, path `profiles/${SHORT_SHA}/`. +- HEALTH_DIAG emits bucket path for operator download. + +Baseline profiles captured per release. Regression detection extended: if per-epoch duration grows > 20% over baseline, WARN. + +#### A.5 Orphan Audit (Wire-It-Up enforcement) + +Scan every `pub` module in `crates/ml/src/` and classify: + +| Classification | Action | +|---|---| +| `Wired` (consumed by prod path) | Log in audit doc | +| `Partial` (forward but not backward, or training-only) | Wire missing side via later Parts | +| `Orphan` (no production consumer) | Wire to intended consumer OR delete | +| `Ghost` (consumer path is stubbed) | Unstub + wire | + +Known candidates requiring action: + +- TLOB transformer: orphan to DQN trunk → D.8 wires. +- Mamba2 backward: partial (forward OK, backward stubbed) → D.1 wires. +- Liquid Time-constant `liquid_mod`: unknown effectiveness → D.7 audits. +- TFT backward: partial → D.1 (same infrastructure as Mamba2). +- xLSTM: orphan to DQN → E evaluates. +- KAN: orphan to DQN → E evaluates. + +Deliverable: `docs/dqn-wire-up-audit.md` with one row per pub module. + +#### A.6 Hot-Path Purity Audit + +Scan the training step loop (`fused_training.rs::step_fused`, `gpu_dqn_trainer.rs::*`, all cuda_pipeline kernels) for any `memcpy_dtoh` / `memcpy_htod` / `cudaMemcpy*` calls. Classify per Invariant 3: + +- `OK-pinned` — already using zero-copy pattern. +- `MIGRATE` — uses `memcpy` but should be pinned; fix during spec impl. +- `COLD-PATH` — acceptable placement (checkpoint, fold transition, epoch boundary), annotated. + +Deliverable: `docs/dqn-gpu-hot-path-audit.md`. Pre-commit hook flags any new unaudited cross-boundary call. + +Known suspects to verify: `read_atom_utilization`, `branch_grad_scales()` accessor, PER priority batched updates, `kelly_stats_buf` synchronisation, any fold-boundary readbacks, IQL readiness readback, q-stats readback paths. + +### Part B — Flat-trap Escape + +#### B.1 ISV-driven continuous reward shaping + +Current: Flat opp-cost ≤ 1e-6/bar — invisible. Q(Flat) ≈ 0 always wins argmax over CQL-pessimistic trade-action Q. + +Design: scale the opp-cost by Q-magnitude via ISV[Q_DIR_ABS_REF]: + +``` +reward_flat = -shaping_scale × holding_cost × conviction × vol_proxy × isv[Q_DIR_ABS_REF] +``` + +When Q magnitudes are small (cold start), penalty is small. When Q converges into the ±20..±50 range, penalty scales proportionally so it remains meaningful relative to trade-action Q. Self-adaptive. No tuned scale multiplier; the existing `holding_cost_rate` config scalar is the only external parameter and it is now f32 (post-d64adc14f refactor). + +#### B.2 ISV-driven trade-attempt bonus (novelty-weighted) + +A separate reward term on Flat→Positioned transitions, representing exploration credit: + +``` +bonus = conviction × vol_proxy × novelty +novelty = max(0, 1 − isv[TRADE_ATTEMPT_RATE_EMA] / isv[TRADE_TARGET_RATE]) +``` + +- `TRADE_ATTEMPT_RATE_EMA` (new ISV slot) — per-window Flat→Pos transition rate, updated via adaptive-rate EMA. +- `TRADE_TARGET_RATE` (new ISV slot) — frozen after first K epochs, captures the "normal rate" for this task. Not a tuned constant; measured from the training regime. + +When the policy is trading at task-normal rate, novelty is 0 and the bonus is inert. When the policy under-trades (Flat-trap), novelty → 1 and the bonus fires. Self-disabling once the Flat-trap is escaped. + +#### B.3 Replay Buffer Seeded Warm-Start + +Current: PER buffer starts empty, fills via Boltzmann-driven DQN trajectories. At t=0 those trajectories are near-random but magnitude-biased toward the scripted plan MLP output, which produces mostly-losing trades → CQL learns "trading is bad" from a biased sample. + +Design: before DQN training starts, seed the PER buffer with ~100K experiences drawn from a **portfolio of scripted policies**: + +- 40% pure uniform random (all 108 factored actions equiprobable, respecting dir/mag constraints). +- 20% momentum-1 / momentum-3 / momentum-5 (simple N-bar momentum entry, fixed magnitude). +- 20% mean-reversion (short-horizon contrarian, quantile-based entry). +- 20% VWAP-deviation (volume-weighted trend crossover). + +Seed trajectories stored with lower initial PER priority than new DQN trajectories, displaced by DQN experiences over roughly 10–15 epochs. CQL alpha is coupled to the seed-phase decay — see C.5. + +Per `feedback_no_feature_flags.md`, this is NOT a feature flag. The seed is always run for a fresh training job. Warm-restart from a checkpoint skips the seed step. + +#### B.4 Adaptive plan-MLP activation threshold + +Current: `plan_state[0] > 0.5` hardcoded threshold. In the val-mkgwm run (2026-04-24), plan MLP outputs stayed sub-0.5 for the entire val window → plans never activated → `plan_isv` remained zero despite the val-side machinery being wired. + +Design: new ISV slot `PLAN_PARAMS_0_EMA` tracks the EMA of `plan_params[0]` (target_bars MLP output) across training samples. Activation threshold is: + +``` +activation_threshold = 0.5 × isv[PLAN_PARAMS_0_EMA] +``` + +Self-scaling. If MLP outputs typically sit near 2.0, threshold is 1.0; if typically near 0.3, threshold is 0.15 and plans still activate. No hardcoded 0.5 constant remains. + +Consumed by both training `experience_env_step` and val `backtest_plan_state_isv`. + +### Part C — Refinement + +#### C.1 Quantile-based atom support (replaces task #91) + +Current: `half = max(10 × q_gap, 3 × std_ema, min_floor)`. The `10×` is a tuned constant; saturates `abs_half` when q_gap > 5. + +Design: 8 new ISV slots (2 per branch) `Q_P05_EMA[dir,mag,ord,urg]`, `Q_P95_EMA[dir,mag,ord,urg]`. Updated from per-branch Q statistics via adaptive-rate EMA (matching the existing mean/std EMA pattern). + +``` +half = max(|q_p95 − v_center|, |v_center − q_p5|) +half = half.clamp(min_half_floor, abs_half) +``` + +Covers the observed Q distribution directly. No distributional assumptions. Tail-aware. No tuned multiplier. + +#### C.2 Reward-component attribution + +Current HEALTH_DIAG shows `grad_split_bwd` but no per-sample reward-component breakdown. Debugging which reward term is driving policy behaviour requires log-reading archaeology. + +Design: new ISV slots `REWARD_COMPONENT_EMA_{popart, cf, trail, micro, opp_cost, bonus}` (6 slots) tracking per-component |reward| contribution via adaptive-rate EMA. HEALTH_DIAG emits `reward_split [...]` analog to `grad_split_bwd`. + +Enables "did micro_reward actually contribute this epoch or is it silently zero?" at a glance. + +#### C.3 State-distribution divergence signal + +Current: training and val state distributions can diverge structurally (plan_isv was OOD for months before we caught it). Only detectable post-hoc. + +Design: new ISV slot `STATE_KL_TRAIN_VAL_EMA` — running estimate of distributional divergence between train-state and val-state on a subset of state dimensions (portfolio + plan + ISV features). Computed at val backtest time by comparing per-dimension moments (mean, std). + +When `KL > threshold` (also ISV-tracked, adaptive): training-time reward shaping (B.1, B.2) gets amplified automatically. Creates a feedback loop where distribution shift is self-correcting via training emphasis. + +#### C.4 Temporal timing bonus + +Current: Mamba2 trunk has temporal memory; reward barely reaches 10 bars back. Architecture-reward misalignment. + +Design: on trade exit, compute: + +``` +temporal_timing_bonus = shaping_scale × max(0, bars_early / hold_time) × |final_pnl| × conviction +``` + +`bars_early` = number of bars between trade entry and the peak unrealised P&L during the trade. Causal-safe (computed AT EXIT, uses no future info beyond the hold window we just completed). + +Rewards the policy for "seeing it coming" — predictive timing that Mamba2 is architecturally capable of learning but not currently incentivised to. + +#### C.5 CQL alpha schedule coupled to seed-phase decay + +Current: `cql_alpha = 0.014` fixed. + +In B.3, CQL pessimism learned from seeded experiences would counter the seed's purpose (we want CQL to learn from the seed-diverse Q distribution, not pull it pessimistic). + +Design: + +``` +cql_alpha(t) = final_alpha × (1 − exp(−t / warmup_steps)) +warmup_steps = tracked via ISV, proportional to the seed-phase decay in B.3 +``` + +While seed trajectories dominate the buffer, `cql_alpha` is near zero (no pessimism). As seed trajectories displace, `cql_alpha` ramps to its final value. + +#### C.6 Adaptive Controller Unification (major refactor) + +Current: the DQN has many adaptive controllers (atoms, gamma, grad-balancer, Kelly cap, tau, cql_alpha, epsilon, conviction-floor, plan-threshold, and new ones added by this spec). Each was bolted on independently with its own update rule and ISV interaction. + +Design: unify under `trait AdaptiveController`: + +```rust +trait AdaptiveController { + type Signal; + type Control; + + fn read_signals(&self, isv: &IsvBus) -> Self::Signal; + fn update(&mut self, signals: Self::Signal) -> Self::Control; + fn fire_rate(&self) -> FireRateStats; + fn diagnose(&self) -> DiagSnapshot; +} +``` + +Each existing controller becomes an `impl AdaptiveController`. Shared: + +- ISV-read via typed slot handles (A.2). +- ISV-write of the control value via typed slot handles. +- Fire-rate tracking instrumented by `controller_activity` smoke test. +- HEALTH_DIAG emission via `diagnose()` returning a standardised snapshot. + +Benefits: + +- Adding a new adaptive mechanism follows a protocol, not an ad-hoc accretion. +- Fire-rate auditing is built-in (addresses `controller_activity` smoke's assertion that load-bearing controllers shouldn't fire in >50% of epochs). +- Standardised diagnostics — the HEALTH_DIAG `controllers [anti_lr=true gamma=false clip=true …]` line becomes derived state, not manually-coded. + +Biggest-scope piece in the spec. Executed as a series of internal commits (one per existing controller migrated to the protocol) before a final commit that removes the old scaffolding. + +### Part D — Temporal Architecture + +Current: Mamba2 + Liquid-TC in trunk; plan_isv state machine; conviction drift; per-bar reward with sparse trade-completion. The architecture is deep-temporal; the reward signal is shallow-temporal. Misalignment. + +#### D.1 Complete Mamba2 backward pipeline (task #76 subset) + +Current: forward works, backward stubbed. + +Design: properly wire Mamba2 backward — state-space gradients dx, dA, dB, dC through the recurrence. Substantial CUDA kernel work but load-bearing: every temporal improvement downstream assumes temporal learning actually propagates end-to-end. + +Scope: fused backward kernel covering `mamba2_temporal_kernel.cu`. Validated via grad-norm smoke test (perturbation grad-check against finite difference). + +This closes task #76 for the DQN path; TFT backward (also task #76) covered in Part E if TFT is adopted as auxiliary head. + +#### D.2 Per-branch gamma + +Current: single scalar gamma across all 4 factored branches. + +Design: 4 ISV slots `GAMMA_{DIR, MAG, ORD, URG}`, per-branch gamma updated by an `AdaptiveController` impl. Each branch's C51 projection uses its own gamma. Direction can run gamma=0.99 (trend horizon); urgency can run gamma=0.80 (execution horizon). Driven by per-branch Q-gap spread and atom utilization, not tuned. + +Consumed by: `block_bellman_project_f` in c51_loss_kernel.cu (reads per-branch gamma), `iql_compute_per_sample_support`, etc. + +#### D.3 Horizon-decomposed value function + +Current: IQL `V(s)` is scalar. + +Design: decompose `V(s) = V_short(s) + V_long(s)`: + +- V_short: 10-bar rolling P&L expectation (execution quality). +- V_long: 100-bar P&L expectation (trend capture). +- Two IQL heads, both expectile-regressed. +- Sum is the Q-target; decomposition is diagnostic + allows horizon-weighted action selection. + +Novel: lets the TD target explicitly attribute "this action helps short-term or long-term". Debuggable — we see if policy is over-indexing on one horizon. + +State layout impact: adds one dimension to the IQL value output. Schema version bump (A.2) + state_layout coordinated change. + +#### D.4 Generalised temporal reward coupling + +Extends C.4 with two more components: + +- **Persistence credit:** trade held through noise before taking profit → bonus. Discourages whipsaw exits. +- **Regime-shift penalty:** entered trade assuming regime X, regime flipped mid-trade → penalty proportional to how early the exit should have happened. +- **Temporal conviction consistency:** policy's conviction signal was stable across N bars before entry → bonus for well-reasoned entry. + +All ISV-scaled and conviction-gated. New ISV slots: +`TEMPORAL_REWARD_EMA_{persist, regime_shift, consistency}` — producer: env_step; consumer: HEALTH_DIAG. + +#### D.5 Soft fold-boundary transitions + +Current (per A.1): reset registry with `FoldReset` (hard) and `TrainingPersist` (unchanged). + +Design: registry extended with `SoftReset(decay_bars: u32)` variant. Soft-reset state anneals toward its bootstrap value over the first `decay_bars` of the new fold. Applies to: + +- `adaptive_gamma` — shouldn't jump from 0.95 back to 0.90 at fold boundary. +- Grad-balance ISV targets — re-seed anneals, don't hard-reset. +- Plan-params EMA — partial persist so MLP output distribution is known at fold start. + +Smoother transitions preserve learned temporal signals across folds without compromising fold-independence of weight-related state. + +#### D.6 Plan temporal dynamics enrichment + +Current `plan_isv[6]` dimensions: progress, pnl-vs-target, pnl-vs-stop, entry-conviction, conviction-drift, regime-shift. + +Design: extend to `plan_isv[7]` — add a new dimension: + +``` +plan_isv[6] = (plan_target_bars − hold_time) / plan_target_bars + = remaining-fraction, clamped to [0, 1] +``` + +Directly exposes temporal pressure to the policy ("my plan is 80% done, should I take the profit I have?"). Trivial to compute; state-layout version bump. + +#### D.7 Liquid Time-constant `liquid_mod` audit + +Current: `liquid_mod_buf` appears in HEALTH_DIAG and is passed to c51 kernels. Its measurable effect on Q-update is unclear. + +Design: audit-only subtask — + +1. Confirm `liquid_mod` actually modulates c51 gradients proportionally (trace through `c51_grad_kernel`). +2. Measure its fire rate — does it ever deviate meaningfully from 1.0? +3. Verify its controller is ISV-driven (per Invariant 1). + +Outcome: either wire properly (via C.6's `AdaptiveController` impl) or — if verified as no-op in practice — mark for deletion per `feedback_wire_everything_up.md`. + +#### D.8 TLOB-DQN temporal integration + +Current: TLOB is orphan relative to DQN trunk (A.5 audit). OFI at state[42..62) is classical (VPIN, Kyle's λ, depth imbalance). TLOB's learned attention is not in the DQN forward pass. + +Design: TLOB as **complementary temporal encoder** in the DQN trunk. + +``` +MBP-10 raw → TLOB transformer → tlob_features[D_tlob] (new) +MBP-10 raw → classical OFI → ofi_features[D_ofi] (existing) + +trunk_input = [market ‖ tlob_features ‖ ofi_features ‖ mtf ‖ portfolio ‖ plan_isv] +``` + +Two complementary temporal modalities feeding the DQN: TLOB attention (event-level), Mamba2 state-space (bar-level). + +Wiring: + +- TLOB weights loaded from supervised pretraining checkpoint at DQN init (fast initial value). +- Trainable end-to-end if Mamba2-class backward infrastructure (D.1) is extended to TLOB attention — decided during D.8 impl based on per-step cost. +- If training full end-to-end is too expensive: TLOB frozen, trained on a side stream periodically (supervised on future-return prediction). +- New ISV slot `TLOB_REGIME_FOCUS_EMA` — learned attention weight carrying a regime-focus signal. Feedback into reward shaping (B.1, B.2). + +State-layout impact: adds D_tlob dimensions. Schema version bump (A.2) + state_layout.cuh coordinated change with D.6, D.3. + +### Part E — Supervised→DQN Concept Audit + +Systematic review of ml-supervised concepts the project has already built, with classification for DQN adoption. + +| Concept | Current DQN use | Adoption in spec | +|---|---|---| +| TFT Variable Selection Network (VSN) | Partial (`vsn_mag`, `vsn_dir` masks) | E.1: full TFT VSN over state features | +| Gated Residual Network (GRN) | Audit needed | E.2: if not present, adopt as trunk building block | +| TFT multi-quantile heads | Partial (IQN CVaR) | E.3: full quantile decomposition for risk branch | +| TLOB attention encoder | Orphan to DQN | D.8 (wired) | +| Mamba2 forward | Wired | — | +| Mamba2 backward | Partial | D.1 (wired) | +| Liquid Time-constant | Unclear | D.7 (audited) | +| xLSTM | Orphan | E.4: evaluate for long-horizon memory augmentation | +| KAN | Orphan | E.5: evaluate for function-approximation in small-data regime | +| Encoder-decoder separation | Single trunk | E.6: evaluate separating state-encoder from value-decoder | +| Attention-weight interpretability | Partial | E.7: ISV-expose attention diagnostics from all temporal heads | +| Multi-task auxiliary heads | None | E.8: add Q + next-return-prediction + regime-classification heads | + +Some E items ("evaluate") are investigation sub-tasks producing short design notes that feed follow-up PRs. Others ("wired") are concrete engineering work landing in this spec's rollout. + +Deliverable: `docs/ml-supervised-to-dqn-concept-audit.md`. + +--- + +## 5. Implementation methodology + +### Internal commits + worktrees + +- Spec implementation is expected to span many internal commits. Permitted. +- Agents work in git worktrees (`git worktree add`) per `feedback_no_concurrent_agents_shared_tree.md` when parallelism is needed on non-overlapping files. +- Sequential single-tree work for agents touching shared files. +- Every commit merged to main compiles and passes smoke tests. No "broken for 3 commits while we fix forward". + +### Cleanup contract + +- Every commit landed on main preserves all seven invariants. +- Worktrees are removed after merge. No abandoned worktrees. +- Orphan modules discovered during implementation are resolved in the same commit that discovers them (wired or deleted). + +### Landing order (suggested, non-mandatory within spec) + +1. Part A.1, A.2, A.6 (state registry + ISV contract + hot-path audit — zero functional change but establishes the substrate). +2. A.5 orphan audit (discovery — zero functional change). +3. C.6 adaptive controller unification — big refactor but non-functional at the surface. +4. C.1, C.2, C.3 (quantile atoms, reward attribution, KL divergence signal — surgical additions). +5. B.1, B.2, B.3, B.4 (reward shaping + replay seed + adaptive plan threshold — behavioural change). +6. D.1, D.2, D.3 (temporal infrastructure — Mamba2 backward, per-branch gamma, horizon-decomposed V). +7. D.4, D.5, D.6, D.7, D.8 (temporal reward, soft transitions, plan dynamics, liquid audit, TLOB wire). +8. E.1–E.8 (supervised concept adoption; each a sub-PR). +9. A.3, A.4 (multi-seed harness + regression detection — validation substrate, no functional change to model). +10. Final validation pass: multi-seed × multi-fold across Tier 1, 2, 3 exit criteria. + +Not all must be sequential — many are independent and can parallelise across worktrees. + +--- + +## 6. Appendix: ISV slot allocation (post-spec) + +Current: 37 slots (indices [0..37)). This spec adds: + +| Index range | Purpose | Source | +|---|---|---| +| [0] | Schema version | A.2 | +| [1..12) | (existing) | — | +| [12] | learning_health | (existing) | +| [13..22) | (existing Q-scale, sharpe_ema) | — | +| [22] | SHARPE_EMA | (existing) | +| [23..31) | Per-branch v-range | (existing) | +| [31..35) | Per-branch grad-norm targets | (existing) | +| [35] | Grad scale limit | (existing) | +| [36] | IQL branch_scales floor | (existing) | +| [37..41) | Per-branch gamma (D.2) | D.2 | +| [41..45) | Per-branch Q_P05_EMA (C.1) | C.1 | +| [45..49) | Per-branch Q_P95_EMA (C.1) | C.1 | +| [49] | PLAN_PARAMS_0_EMA (B.4) | B.4 | +| [50] | TRADE_ATTEMPT_RATE_EMA (B.2) | B.2 | +| [51] | TRADE_TARGET_RATE (B.2) | B.2 | +| [52..58) | REWARD_COMPONENT_EMA_* (C.2) | C.2 | +| [58] | STATE_KL_TRAIN_VAL_EMA (C.3) | C.3 | +| [59..62) | TEMPORAL_REWARD_EMA_* (D.4) | D.4 | +| [62] | TLOB_REGIME_FOCUS_EMA (D.8) | D.8 | +| [63..72) | Reserved for Part E additions | — | + +New `ISV_TOTAL_DIM` = 72 (from 37). Network-facing `ISV_NETWORK_DIM` remains 23 — the new slots are scratchpad / adaptive-control state not fed into the attention path. + +--- + +## 7. Appendix: new documents tracked by pre-commit + +- `docs/dqn-wire-up-audit.md` — component → consumer path mapping (Invariant 2, 7). +- `docs/isv-slots.md` — ISV slot allocation registry with type/producer/consumers (A.2, Invariant 7). +- `docs/dqn-gpu-hot-path-audit.md` — cross-boundary call classification (Invariant 3, 7). +- `docs/ml-supervised-to-dqn-concept-audit.md` — Part E deliverable. +- `config/metric-bands.toml` — regression-detection bands (A.4, Invariant 7). + +Pre-commit hook rejects component-adding commits that don't touch the relevant audit docs. + +--- + +## 8. Open questions (to resolve during implementation) + +- D.8: TLOB end-to-end trainable vs frozen-with-side-stream retraining. Decide based on per-step cost benchmark during D.8 impl. +- E.4, E.5, E.6: xLSTM / KAN / encoder-decoder — each produces a short design note; adoption decided per-note, not in this spec. +- C.6: ordering of individual `AdaptiveController` migrations — atoms-first or gamma-first? Decide at the start of C.6's internal commit series based on which has the simplest ISV surface area. +- A.3 multi-seed: N = 5 seeds may be too expensive for L40S budget; may need to run the production-readiness validation on H100 pool instead. Resource plan finalised with A.3 impl. + +--- + +## 9. Non-goals + +Explicit things this spec does NOT address: + +- PPO trainer path (separate model). +- ml-supervised training itself (only its borrowable concepts). +- Deployment / paper trading harness. +- Data-pipeline changes (MBP-10 ingest, fxcache format). +- TFT as a separate predictive model (only TFT concepts borrowed into DQN via E.1, E.2, E.3). + +--- + +**End of spec.**