gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

spec(dqn): DQN v2 unified integrated policy system design

Browse Source 
Design spec consolidating every hard lesson from the val-Flat-collapse
investigation, the task #92 gradient-pathology triad, the ISV v-range
unification work, the f64→f32 ABI cleanup, and many sessions of
fold-1 grad explosions and orphan-feature accumulation.

Seven non-negotiable invariants:
  1. ISV-driven adaptive bounds (no tuned constants)
  2. Wire-It-Up (no orphans)
  3. GPU-only hot path (zero memcpy DtoH; pinned-zero-copy only)
  4. Pinned-only host memory
  5. Diagnostic-first
  6. Convergence discipline (primary guardrail, multi-seed × multi-fold)
  7. Wire-up + diagnostic audit per-commit

Five Parts, landing as one coordinated rollout:
  A. Foundation — reset registry, ISV contract, multi-seed harness,
     regression detection, orphan audit, hot-path purity audit.
  B. Flat-trap escape — ISV-driven reward shaping, trade-attempt bonus,
     replay warm-start via scripted-policy portfolio, adaptive plan
     threshold.
  C. Refinement — quantile atom support, reward attribution, state-dist
     divergence signal, temporal timing bonus, CQL-seed coupling,
     adaptive controller unification refactor.
  D. Temporal architecture — Mamba2 backward completion, per-branch
     gamma, horizon-decomposed value function, generalised temporal
     reward, soft fold transitions, plan temporal dynamics, liquid
     audit, TLOB-DQN integration.
  E. Supervised→DQN concept audit — TFT VSN, GRN, multi-quantile,
     xLSTM, KAN, encoder-decoder, interpretability, auxiliary heads.

Tiered success criteria:
  Tier 1 — convergence discipline (must pass first).
  Tier 2 — behavioural parity (val trades escape Flat-trap).
  Tier 3 — profitability (val Sharpe > 1.0 on window).

ISV slots grow from 37 to 72. Five new audit docs tracked by
pre-commit hook.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-04-24 09:22:43 +02:00
parent d64adc14f5
commit d13b535862
1 changed files with 600 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

600
docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md Normal file
View File

@@ -0,0 +1,600 @@
# DQN v2 — Unified Integrated Policy System
**Date:** 2026-04-24
**Status:** Design (awaiting user review before writing-plans)
**Scope:** DQN model only — consolidates every lesson from the val-Flat-collapse investigation, the task #92 gradient-pathology triad, the ISV v-range unification work, the f64→f32 ABI-trap cleanup, and the long-running hard lessons about convergence, fold boundaries, orphan features, and hardcoded tuning.
---
## 1. Purpose
This spec is not a patch to a specific failure mode. It is a **consolidation of the Foxhunt DQN architecture into one properly integrated system**, synthesising the concepts we already built in supervised, the adaptive controllers we accreted over many sessions, the ISV signal-bus pattern that emerged from the Apr-23 bundle, and the convergence-discipline lessons from multiple fold-1 grad explosions.
Prior work bolted mechanisms on as problems surfaced. This spec replaces that with a single integrated design that:
- Obeys a small set of non-negotiable invariants (§3).
- Wires every supervised concept the project has built into its intended DQN consumer path (Parts D, E).
- Unifies the many adaptive controllers under one protocol (Part C.6).
- Makes convergence the primary guardrail, not an after-thought (Part A).
- Eliminates orphan code, hot-path DtoH copies, and hardcoded tuning constants as a design principle (§3).
The spec lands as one coordinated rollout. Internal commits during implementation are permitted (agents in worktrees, tree clean at every merge to main). Every commit respects the invariants.
---
## 2. Success criteria (tiered milestones)
Each milestone has explicit exit criteria. Implementation does not declare "done" until ALL tiers pass under the multi-seed / multi-fold harness (§4.A.3).
### Tier 1: Convergence discipline (primary, must pass first)
- Multi-seed training (N=5 seeds) produces val-metric distribution with `std / mean < 0.15` on stable epochs (post-warmup).
- Multi-fold walk-forward (K=6 folds) shows no fold-1 grad explosions, no Q saturation at `abs_half`, no cross-fold state-reset bugs.
- Training halts self-terminates on any HEALTH_DIAG metric out-of-band (regression detection, A.4).
- No hot-path DtoH `memcpy` anywhere in the step loop (A.6 audit passes).
### Tier 2: Behavioural parity (val trades escape the Flat-trap)
- Val trade count within factor-of-2 of training trade count (approximately >= 1% of val bars positioned, vs. the current 0.01%).
- `plan_isv` `active_frac` in val > 0.2 on average across the window (currently 0.000 — plans structurally fail to activate).
- No systematic bias toward Hold or Flat direction argmax (entropy of argmax direction distribution > 0.8 of uniform).
### Tier 3: Profitability (val metric positive and stable)
- Annualised val Sharpe > 1.0 on the validation window.
- Val WinRate ≥ 52% on trades > 500/window.
- Profit Factor ≥ 1.1 stable across the multi-seed runs.
**No tier is skippable.** Tier 3 without Tier 1 means the result is not reproducible. Tier 2 without Tier 1 means the val-trade-count was a lucky seed. Tier 1 without Tier 2 means convergence is clean but the policy is uselessly conservative.
---
## 3. Cross-cutting invariants (non-negotiable)
Every feature added by this spec obeys all seven invariants. Violations are caught by pre-commit hooks and smoke tests.
### Invariant 1 — ISV-driven adaptive bounds
All adaptive clamps, thresholds, targets, and schedules live in the ISV signal bus and are read by consumers at runtime. No hardcoded multipliers, no tuned `clamp(0.25, 4.0)`, no `4 × median`, no `10 × q_gap`. Authority: `feedback_isv_for_adaptive_bounds.md`.
The single carve-out is numerical-safety hard limits: values that express dimensional or numerical-stability reality (e.g. `abs_half = 0.5 × (v_max − v_min)`, `1/limit` for a safety floor). These are documented as such at the constant-definition site.
### Invariant 2 — Wire-It-Up
Every module, feature, or kernel built is wired into its intended production consumer path in the same commit. Orphan code — compiles but no production path consumes it — is either wired or deleted. Authority: `feedback_wire_everything_up.md`.
Enforcement: every new module lands with a row in `docs/dqn-wire-up-audit.md` identifying its consumer. Pre-commit hook rejects commits that add a pub module without updating the audit doc.
### Invariant 3 — GPU-only hot path
No `cudaMemcpy` / `memcpy_dtoh` / `memcpy_htod` calls inside the training step loop. Zero tolerance.
The one allowed cross-boundary mechanism is **pinned + device-mapped memory**: allocated via `cuMemAllocHost + cuMemHostGetDevicePointer`. Kernel writes via the device pointer; host reads via the host pointer to the same physical page. No copy occurs. This is already the ISV pattern; it is promoted to the ONLY allowed per-step cross-boundary mechanism.
`memcpy` DtoH/HtoD is permitted only at cold path: checkpoint save/load, fold transitions, validation-window reshuffle at epoch boundaries. Every such call is annotated with `// cold-path: ok — <reason>` and audited in `docs/dqn-gpu-hot-path-audit.md`.
Authority: `feedback_gpu_cpu_roundtrip.md` (extended with zero-tolerance language in this spec).
### Invariant 4 — Pinned-only host memory
Any host-resident memory that backs a GPU data structure is pinned + device-mapped via `cuMemAllocHost_v2 + cuMemHostGetDevicePointer_v2`. Regular `malloc` for GPU-adjacent buffers is forbidden. UVM (`cudaMallocManaged`) is forbidden (page-fault stalls in the hot path).
### Invariant 5 — Diagnostic-first
Every adaptive controller, every reward component, every new state partition, every new ISV slot emits HEALTH_DIAG-compatible telemetry. If it's adaptive or consequential, it's diagnostic.
Enforcement: `controller_activity` and `reward_component_audit` smoke tests extended with coverage assertions. Missing diagnostics fail smoke.
### Invariant 6 — Convergence discipline (primary)
No feature is "done" until it passes multi-seed (N ≥ 5) × multi-fold (K ≥ 6) under the validation harness (§4.A.3). "It worked on one run" is not a proof.
### Invariant 7 — Wire-up + diagnostic audit per-commit
Every commit that adds or modifies a component updates:
- `docs/dqn-wire-up-audit.md` — consumer-path mapping per module.
- `docs/isv-slots.md` — ISV slot allocations with type, producer, consumers.
- `docs/dqn-gpu-hot-path-audit.md` — every cross-boundary call classified.
- `config/metric-bands.toml` — regression-detection bands for new metrics.
Pre-commit hook rejects component-adding commits that don't touch these docs.
---
## 4. Architecture
The spec is decomposed into five Parts. All Parts land in one coordinated rollout; internal commits during implementation are permitted (worktree isolation for parallel agents, sequential merges to main).
Parts A and B are load-bearing for everything else. Part A establishes the convergence substrate. Part B escapes the Flat-trap. Parts C, D, E are refinement, temporal unification, and supervised-concept reuse respectively.
### Part A — Foundation (convergence substrate)
#### A.1 State Reset Registry
Current state: some state resets at fold boundaries (`eval_v_range`, target network hard copy), some persists (learned weights, Adam momentum, ISV history), some is undefined. The Fold-1 grad explosion (task #31) and the fold-boundary gap (task #84) trace to this ambiguity.
Design: every piece of training-time state is classified into one of four categories via a typed `StateResetRegistry`:
| Category | Meaning | Example |
|---|---|---|
| `FoldReset` | Cleared at fold boundary | Kelly stats, plan_state, eval_q_*_ema, v-range ISV slots |
| `WindowReset` | Cleared at validation-window boundary | Portfolio state, hold_time, entry_price |
| `SoftReset(decay_bars)` | Anneals toward bootstrap over decay_bars at fold boundary | adaptive_gamma, ISV grad-balance targets |
| `TrainingPersist` | Survives across folds (learned weights) | Network params, target network, Adam momentum |
| `SchemaContract` | Version-stable across runs | ISV slot allocations themselves |
`reset_fold_state(fold_idx)` iterates the registry. Adding new state requires registering it with a category. Compile-time enforcement where possible via module visibility on a typed registry struct; runtime assertion in constructor.
See D.5 for the SoftReset motivation (some temporal state benefits from annealing, not hard cuts).
#### A.2 ISV Slot Contract + Schema Versioning
Current: 37 slots, each `pub const X_INDEX: usize = N`. No producer/consumer contract, no version, no migration path.
Design:
- Introduce `IsvSlot<T>` typed handles with:
- Producer identity (which module writes)
- Consumer list (which modules read)
- Enforced via module visibility + a single audit table in `docs/isv-slots.md`.
- ISV slot `[0]` reserved as schema version (u32 bumped on any slot-layout change).
- Checkpoint load: reads ISV[0] version, migrates or fails explicitly — no silent schema drift.
- Reserve slots [36..48) for second-order ISV signals (Part D, diagnostic-on-ISV, plan temporal dynamics).
#### A.3 Multi-Seed / Multi-Fold Validation Harness
Current: one seed, one fold triggered by `argo-train.sh`, metrics eyeballed. Signal-to-noise terrible.
Design: `argo-train.sh --multi-seed N --folds K` spawns `N × K` training jobs from a single submission:
- Per-epoch metric aggregation across seeds: mean ± 2σ, median, worst-case.
- HEALTH_DIAG emits cross-seed stats at epoch boundary.
- Exit criteria use **pessimistic** estimator (mean − 2σ) not mean.
- `norm_stats_foldN.json` merged across seeds to produce `norm_stats_fold_aggregate.json`.
- Job orchestration via Argo Workflow DAG (existing pattern in `infra/k8s/argo/train-template.yaml`).
#### A.4 Regression Detection
Current: a wildly wrong metric trains to completion; we catch it in logs.
Design: per-metric historical bands in `config/metric-bands.toml`:
```toml
[metric_bands.train_sharpe]
warn_low = -5.0
warn_high = 200.0
error_low = -50.0
error_high = 1000.0
```
HEALTH_DIAG emission checks every metric against its band. N consecutive out-of-band epochs → `WARN` log. 2N consecutive → `ERROR` + SIGTERM-self via `std::process::exit(1)`. Training workflow FAIL status; Argo notification fires.
The bands are populated from the last 3 known-good runs' metric distributions. Updated as part of the promotion workflow when a new baseline run succeeds.
##### A.4.1 nsys profile harness (sub-task)
Add `--profile` flag to `argo-train.sh`. When set:
- ci-builder image includes `nsys` (via apt install or pinned tarball).
- `train-template.yaml` wraps the training binary under `nsys profile --capture-range=cudaProfilerApi --output=profile-${SHORT_SHA}.nsys-rep`.
- .nsys-rep artifact pushed to `foxhunt-training-artifacts` MinIO bucket, path `profiles/${SHORT_SHA}/`.
- HEALTH_DIAG emits bucket path for operator download.
Baseline profiles captured per release. Regression detection extended: if per-epoch duration grows > 20% over baseline, WARN.
#### A.5 Orphan Audit (Wire-It-Up enforcement)
Scan every `pub` module in `crates/ml/src/` and classify:
| Classification | Action |
|---|---|
| `Wired` (consumed by prod path) | Log in audit doc |
| `Partial` (forward but not backward, or training-only) | Wire missing side via later Parts |
| `Orphan` (no production consumer) | Wire to intended consumer OR delete |
| `Ghost` (consumer path is stubbed) | Unstub + wire |
Known candidates requiring action:
- TLOB transformer: orphan to DQN trunk → D.8 wires.
- Mamba2 backward: partial (forward OK, backward stubbed) → D.1 wires.
- Liquid Time-constant `liquid_mod`: unknown effectiveness → D.7 audits.
- TFT backward: partial → D.1 (same infrastructure as Mamba2).
- xLSTM: orphan to DQN → E evaluates.
- KAN: orphan to DQN → E evaluates.
Deliverable: `docs/dqn-wire-up-audit.md` with one row per pub module.
#### A.6 Hot-Path Purity Audit
Scan the training step loop (`fused_training.rs::step_fused`, `gpu_dqn_trainer.rs::*`, all cuda_pipeline kernels) for any `memcpy_dtoh` / `memcpy_htod` / `cudaMemcpy*` calls. Classify per Invariant 3:
- `OK-pinned` — already using zero-copy pattern.
- `MIGRATE` — uses `memcpy` but should be pinned; fix during spec impl.
- `COLD-PATH` — acceptable placement (checkpoint, fold transition, epoch boundary), annotated.
Deliverable: `docs/dqn-gpu-hot-path-audit.md`. Pre-commit hook flags any new unaudited cross-boundary call.
Known suspects to verify: `read_atom_utilization`, `branch_grad_scales()` accessor, PER priority batched updates, `kelly_stats_buf` synchronisation, any fold-boundary readbacks, IQL readiness readback, q-stats readback paths.
### Part B — Flat-trap Escape
#### B.1 ISV-driven continuous reward shaping
Current: Flat opp-cost ≤ 1e-6/bar — invisible. Q(Flat) ≈ 0 always wins argmax over CQL-pessimistic trade-action Q.
Design: scale the opp-cost by Q-magnitude via ISV[Q_DIR_ABS_REF]:
```
reward_flat = -shaping_scale × holding_cost × conviction × vol_proxy × isv[Q_DIR_ABS_REF]
```
When Q magnitudes are small (cold start), penalty is small. When Q converges into the ±20..±50 range, penalty scales proportionally so it remains meaningful relative to trade-action Q. Self-adaptive. No tuned scale multiplier; the existing `holding_cost_rate` config scalar is the only external parameter and it is now f32 (post-d64adc14f refactor).
#### B.2 ISV-driven trade-attempt bonus (novelty-weighted)
A separate reward term on Flat→Positioned transitions, representing exploration credit:
```
bonus = conviction × vol_proxy × novelty
novelty = max(0, 1 − isv[TRADE_ATTEMPT_RATE_EMA] / isv[TRADE_TARGET_RATE])
```
- `TRADE_ATTEMPT_RATE_EMA` (new ISV slot) — per-window Flat→Pos transition rate, updated via adaptive-rate EMA.
- `TRADE_TARGET_RATE` (new ISV slot) — frozen after first K epochs, captures the "normal rate" for this task. Not a tuned constant; measured from the training regime.
When the policy is trading at task-normal rate, novelty is 0 and the bonus is inert. When the policy under-trades (Flat-trap), novelty → 1 and the bonus fires. Self-disabling once the Flat-trap is escaped.
#### B.3 Replay Buffer Seeded Warm-Start
Current: PER buffer starts empty, fills via Boltzmann-driven DQN trajectories. At t=0 those trajectories are near-random but magnitude-biased toward the scripted plan MLP output, which produces mostly-losing trades → CQL learns "trading is bad" from a biased sample.
Design: before DQN training starts, seed the PER buffer with ~100K experiences drawn from a **portfolio of scripted policies**:
- 40% pure uniform random (all 108 factored actions equiprobable, respecting dir/mag constraints).
- 20% momentum-1 / momentum-3 / momentum-5 (simple N-bar momentum entry, fixed magnitude).
- 20% mean-reversion (short-horizon contrarian, quantile-based entry).
- 20% VWAP-deviation (volume-weighted trend crossover).
Seed trajectories stored with lower initial PER priority than new DQN trajectories, displaced by DQN experiences over roughly 10–15 epochs. CQL alpha is coupled to the seed-phase decay — see C.5.
Per `feedback_no_feature_flags.md`, this is NOT a feature flag. The seed is always run for a fresh training job. Warm-restart from a checkpoint skips the seed step.
#### B.4 Adaptive plan-MLP activation threshold
Current: `plan_state[0] > 0.5` hardcoded threshold. In the val-mkgwm run (2026-04-24), plan MLP outputs stayed sub-0.5 for the entire val window → plans never activated → `plan_isv` remained zero despite the val-side machinery being wired.
Design: new ISV slot `PLAN_PARAMS_0_EMA` tracks the EMA of `plan_params[0]` (target_bars MLP output) across training samples. Activation threshold is:
```
activation_threshold = 0.5 × isv[PLAN_PARAMS_0_EMA]
```
Self-scaling. If MLP outputs typically sit near 2.0, threshold is 1.0; if typically near 0.3, threshold is 0.15 and plans still activate. No hardcoded 0.5 constant remains.
Consumed by both training `experience_env_step` and val `backtest_plan_state_isv`.
### Part C — Refinement
#### C.1 Quantile-based atom support (replaces task #91)
Current: `half = max(10 × q_gap, 3 × std_ema, min_floor)`. The `10×` is a tuned constant; saturates `abs_half` when q_gap > 5.
Design: 8 new ISV slots (2 per branch) `Q_P05_EMA[dir,mag,ord,urg]`, `Q_P95_EMA[dir,mag,ord,urg]`. Updated from per-branch Q statistics via adaptive-rate EMA (matching the existing mean/std EMA pattern).
```
half = max(|q_p95 − v_center|, |v_center − q_p5|)
half = half.clamp(min_half_floor, abs_half)
```
Covers the observed Q distribution directly. No distributional assumptions. Tail-aware. No tuned multiplier.
#### C.2 Reward-component attribution
Current HEALTH_DIAG shows `grad_split_bwd` but no per-sample reward-component breakdown. Debugging which reward term is driving policy behaviour requires log-reading archaeology.
Design: new ISV slots `REWARD_COMPONENT_EMA_{popart, cf, trail, micro, opp_cost, bonus}` (6 slots) tracking per-component |reward| contribution via adaptive-rate EMA. HEALTH_DIAG emits `reward_split [...]` analog to `grad_split_bwd`.
Enables "did micro_reward actually contribute this epoch or is it silently zero?" at a glance.
#### C.3 State-distribution divergence signal
Current: training and val state distributions can diverge structurally (plan_isv was OOD for months before we caught it). Only detectable post-hoc.
Design: new ISV slot `STATE_KL_TRAIN_VAL_EMA` — running estimate of distributional divergence between train-state and val-state on a subset of state dimensions (portfolio + plan + ISV features). Computed at val backtest time by comparing per-dimension moments (mean, std).
When `KL > threshold` (also ISV-tracked, adaptive): training-time reward shaping (B.1, B.2) gets amplified automatically. Creates a feedback loop where distribution shift is self-correcting via training emphasis.
#### C.4 Temporal timing bonus
Current: Mamba2 trunk has temporal memory; reward barely reaches 10 bars back. Architecture-reward misalignment.
Design: on trade exit, compute:
```
temporal_timing_bonus = shaping_scale × max(0, bars_early / hold_time) × |final_pnl| × conviction
```
`bars_early` = number of bars between trade entry and the peak unrealised P&L during the trade. Causal-safe (computed AT EXIT, uses no future info beyond the hold window we just completed).
Rewards the policy for "seeing it coming" — predictive timing that Mamba2 is architecturally capable of learning but not currently incentivised to.
#### C.5 CQL alpha schedule coupled to seed-phase decay
Current: `cql_alpha = 0.014` fixed.
In B.3, CQL pessimism learned from seeded experiences would counter the seed's purpose (we want CQL to learn from the seed-diverse Q distribution, not pull it pessimistic).
Design:
```
cql_alpha(t) = final_alpha × (1 − exp(−t / warmup_steps))
warmup_steps = tracked via ISV, proportional to the seed-phase decay in B.3
```
While seed trajectories dominate the buffer, `cql_alpha` is near zero (no pessimism). As seed trajectories displace, `cql_alpha` ramps to its final value.
#### C.6 Adaptive Controller Unification (major refactor)
Current: the DQN has many adaptive controllers (atoms, gamma, grad-balancer, Kelly cap, tau, cql_alpha, epsilon, conviction-floor, plan-threshold, and new ones added by this spec). Each was bolted on independently with its own update rule and ISV interaction.
Design: unify under `trait AdaptiveController`:
```rust
trait AdaptiveController {
type Signal;
type Control;
fn read_signals(&self, isv: &IsvBus) -> Self::Signal;
fn update(&mut self, signals: Self::Signal) -> Self::Control;
fn fire_rate(&self) -> FireRateStats;
fn diagnose(&self) -> DiagSnapshot;
}
```
Each existing controller becomes an `impl AdaptiveController`. Shared:
- ISV-read via typed slot handles (A.2).
- ISV-write of the control value via typed slot handles.
- Fire-rate tracking instrumented by `controller_activity` smoke test.
- HEALTH_DIAG emission via `diagnose()` returning a standardised snapshot.
Benefits:
- Adding a new adaptive mechanism follows a protocol, not an ad-hoc accretion.
- Fire-rate auditing is built-in (addresses `controller_activity` smoke's assertion that load-bearing controllers shouldn't fire in >50% of epochs).
- Standardised diagnostics — the HEALTH_DIAG `controllers [anti_lr=true gamma=false clip=true …]` line becomes derived state, not manually-coded.
Biggest-scope piece in the spec. Executed as a series of internal commits (one per existing controller migrated to the protocol) before a final commit that removes the old scaffolding.
### Part D — Temporal Architecture
Current: Mamba2 + Liquid-TC in trunk; plan_isv state machine; conviction drift; per-bar reward with sparse trade-completion. The architecture is deep-temporal; the reward signal is shallow-temporal. Misalignment.
#### D.1 Complete Mamba2 backward pipeline (task #76 subset)
Current: forward works, backward stubbed.
Design: properly wire Mamba2 backward — state-space gradients dx, dA, dB, dC through the recurrence. Substantial CUDA kernel work but load-bearing: every temporal improvement downstream assumes temporal learning actually propagates end-to-end.
Scope: fused backward kernel covering `mamba2_temporal_kernel.cu`. Validated via grad-norm smoke test (perturbation grad-check against finite difference).
This closes task #76 for the DQN path; TFT backward (also task #76) covered in Part E if TFT is adopted as auxiliary head.
#### D.2 Per-branch gamma
Current: single scalar gamma across all 4 factored branches.
Design: 4 ISV slots `GAMMA_{DIR, MAG, ORD, URG}`, per-branch gamma updated by an `AdaptiveController` impl. Each branch's C51 projection uses its own gamma. Direction can run gamma=0.99 (trend horizon); urgency can run gamma=0.80 (execution horizon). Driven by per-branch Q-gap spread and atom utilization, not tuned.
Consumed by: `block_bellman_project_f` in c51_loss_kernel.cu (reads per-branch gamma), `iql_compute_per_sample_support`, etc.
#### D.3 Horizon-decomposed value function
Current: IQL `V(s)` is scalar.
Design: decompose `V(s) = V_short(s) + V_long(s)`:
- V_short: 10-bar rolling P&L expectation (execution quality).
- V_long: 100-bar P&L expectation (trend capture).
- Two IQL heads, both expectile-regressed.
- Sum is the Q-target; decomposition is diagnostic + allows horizon-weighted action selection.
Novel: lets the TD target explicitly attribute "this action helps short-term or long-term". Debuggable — we see if policy is over-indexing on one horizon.
State layout impact: adds one dimension to the IQL value output. Schema version bump (A.2) + state_layout coordinated change.
#### D.4 Generalised temporal reward coupling
Extends C.4 with two more components:
- **Persistence credit:** trade held through noise before taking profit → bonus. Discourages whipsaw exits.
- **Regime-shift penalty:** entered trade assuming regime X, regime flipped mid-trade → penalty proportional to how early the exit should have happened.
- **Temporal conviction consistency:** policy's conviction signal was stable across N bars before entry → bonus for well-reasoned entry.
All ISV-scaled and conviction-gated. New ISV slots:
`TEMPORAL_REWARD_EMA_{persist, regime_shift, consistency}` — producer: env_step; consumer: HEALTH_DIAG.
#### D.5 Soft fold-boundary transitions
Current (per A.1): reset registry with `FoldReset` (hard) and `TrainingPersist` (unchanged).
Design: registry extended with `SoftReset(decay_bars: u32)` variant. Soft-reset state anneals toward its bootstrap value over the first `decay_bars` of the new fold. Applies to:
- `adaptive_gamma` — shouldn't jump from 0.95 back to 0.90 at fold boundary.
- Grad-balance ISV targets — re-seed anneals, don't hard-reset.
- Plan-params EMA — partial persist so MLP output distribution is known at fold start.
Smoother transitions preserve learned temporal signals across folds without compromising fold-independence of weight-related state.
#### D.6 Plan temporal dynamics enrichment
Current `plan_isv[6]` dimensions: progress, pnl-vs-target, pnl-vs-stop, entry-conviction, conviction-drift, regime-shift.
Design: extend to `plan_isv[7]` — add a new dimension:
```
plan_isv[6] = (plan_target_bars − hold_time) / plan_target_bars
= remaining-fraction, clamped to [0, 1]
```
Directly exposes temporal pressure to the policy ("my plan is 80% done, should I take the profit I have?"). Trivial to compute; state-layout version bump.
#### D.7 Liquid Time-constant `liquid_mod` audit
Current: `liquid_mod_buf` appears in HEALTH_DIAG and is passed to c51 kernels. Its measurable effect on Q-update is unclear.
Design: audit-only subtask —
1. Confirm `liquid_mod` actually modulates c51 gradients proportionally (trace through `c51_grad_kernel`).
2. Measure its fire rate — does it ever deviate meaningfully from 1.0?
3. Verify its controller is ISV-driven (per Invariant 1).
Outcome: either wire properly (via C.6's `AdaptiveController` impl) or — if verified as no-op in practice — mark for deletion per `feedback_wire_everything_up.md`.
#### D.8 TLOB-DQN temporal integration
Current: TLOB is orphan relative to DQN trunk (A.5 audit). OFI at state[42..62) is classical (VPIN, Kyle's λ, depth imbalance). TLOB's learned attention is not in the DQN forward pass.
Design: TLOB as **complementary temporal encoder** in the DQN trunk.
```
MBP-10 raw → TLOB transformer → tlob_features[D_tlob] (new)
MBP-10 raw → classical OFI → ofi_features[D_ofi] (existing)
trunk_input = [market ‖ tlob_features ‖ ofi_features ‖ mtf ‖ portfolio ‖ plan_isv]
```
Two complementary temporal modalities feeding the DQN: TLOB attention (event-level), Mamba2 state-space (bar-level).
Wiring:
- TLOB weights loaded from supervised pretraining checkpoint at DQN init (fast initial value).
- Trainable end-to-end if Mamba2-class backward infrastructure (D.1) is extended to TLOB attention — decided during D.8 impl based on per-step cost.
- If training full end-to-end is too expensive: TLOB frozen, trained on a side stream periodically (supervised on future-return prediction).
- New ISV slot `TLOB_REGIME_FOCUS_EMA` — learned attention weight carrying a regime-focus signal. Feedback into reward shaping (B.1, B.2).
State-layout impact: adds D_tlob dimensions. Schema version bump (A.2) + state_layout.cuh coordinated change with D.6, D.3.
### Part E — Supervised→DQN Concept Audit
Systematic review of ml-supervised concepts the project has already built, with classification for DQN adoption.
| Concept | Current DQN use | Adoption in spec |
|---|---|---|
| TFT Variable Selection Network (VSN) | Partial (`vsn_mag`, `vsn_dir` masks) | E.1: full TFT VSN over state features |
| Gated Residual Network (GRN) | Audit needed | E.2: if not present, adopt as trunk building block |
| TFT multi-quantile heads | Partial (IQN CVaR) | E.3: full quantile decomposition for risk branch |
| TLOB attention encoder | Orphan to DQN | D.8 (wired) |
| Mamba2 forward | Wired | — |
| Mamba2 backward | Partial | D.1 (wired) |
| Liquid Time-constant | Unclear | D.7 (audited) |
| xLSTM | Orphan | E.4: evaluate for long-horizon memory augmentation |
| KAN | Orphan | E.5: evaluate for function-approximation in small-data regime |
| Encoder-decoder separation | Single trunk | E.6: evaluate separating state-encoder from value-decoder |
| Attention-weight interpretability | Partial | E.7: ISV-expose attention diagnostics from all temporal heads |
| Multi-task auxiliary heads | None | E.8: add Q + next-return-prediction + regime-classification heads |
Some E items ("evaluate") are investigation sub-tasks producing short design notes that feed follow-up PRs. Others ("wired") are concrete engineering work landing in this spec's rollout.
Deliverable: `docs/ml-supervised-to-dqn-concept-audit.md`.
---
## 5. Implementation methodology
### Internal commits + worktrees
- Spec implementation is expected to span many internal commits. Permitted.
- Agents work in git worktrees (`git worktree add`) per `feedback_no_concurrent_agents_shared_tree.md` when parallelism is needed on non-overlapping files.
- Sequential single-tree work for agents touching shared files.
- Every commit merged to main compiles and passes smoke tests. No "broken for 3 commits while we fix forward".
### Cleanup contract
- Every commit landed on main preserves all seven invariants.
- Worktrees are removed after merge. No abandoned worktrees.
- Orphan modules discovered during implementation are resolved in the same commit that discovers them (wired or deleted).
### Landing order (suggested, non-mandatory within spec)
1. Part A.1, A.2, A.6 (state registry + ISV contract + hot-path audit — zero functional change but establishes the substrate).
2. A.5 orphan audit (discovery — zero functional change).
3. C.6 adaptive controller unification — big refactor but non-functional at the surface.
4. C.1, C.2, C.3 (quantile atoms, reward attribution, KL divergence signal — surgical additions).
5. B.1, B.2, B.3, B.4 (reward shaping + replay seed + adaptive plan threshold — behavioural change).
6. D.1, D.2, D.3 (temporal infrastructure — Mamba2 backward, per-branch gamma, horizon-decomposed V).
7. D.4, D.5, D.6, D.7, D.8 (temporal reward, soft transitions, plan dynamics, liquid audit, TLOB wire).
8. E.1–E.8 (supervised concept adoption; each a sub-PR).
9. A.3, A.4 (multi-seed harness + regression detection — validation substrate, no functional change to model).
10. Final validation pass: multi-seed × multi-fold across Tier 1, 2, 3 exit criteria.
Not all must be sequential — many are independent and can parallelise across worktrees.
---
## 6. Appendix: ISV slot allocation (post-spec)
Current: 37 slots (indices [0..37)). This spec adds:
| Index range | Purpose | Source |
|---|---|---|
| [0] | Schema version | A.2 |
| [1..12) | (existing) | — |
| [12] | learning_health | (existing) |
| [13..22) | (existing Q-scale, sharpe_ema) | — |
| [22] | SHARPE_EMA | (existing) |
| [23..31) | Per-branch v-range | (existing) |
| [31..35) | Per-branch grad-norm targets | (existing) |
| [35] | Grad scale limit | (existing) |
| [36] | IQL branch_scales floor | (existing) |
| [37..41) | Per-branch gamma (D.2) | D.2 |
| [41..45) | Per-branch Q_P05_EMA (C.1) | C.1 |
| [45..49) | Per-branch Q_P95_EMA (C.1) | C.1 |
| [49] | PLAN_PARAMS_0_EMA (B.4) | B.4 |
| [50] | TRADE_ATTEMPT_RATE_EMA (B.2) | B.2 |
| [51] | TRADE_TARGET_RATE (B.2) | B.2 |
| [52..58) | REWARD_COMPONENT_EMA_* (C.2) | C.2 |
| [58] | STATE_KL_TRAIN_VAL_EMA (C.3) | C.3 |
| [59..62) | TEMPORAL_REWARD_EMA_* (D.4) | D.4 |
| [62] | TLOB_REGIME_FOCUS_EMA (D.8) | D.8 |
| [63..72) | Reserved for Part E additions | — |
New `ISV_TOTAL_DIM` = 72 (from 37). Network-facing `ISV_NETWORK_DIM` remains 23 — the new slots are scratchpad / adaptive-control state not fed into the attention path.
---
## 7. Appendix: new documents tracked by pre-commit
- `docs/dqn-wire-up-audit.md` — component → consumer path mapping (Invariant 2, 7).
- `docs/isv-slots.md` — ISV slot allocation registry with type/producer/consumers (A.2, Invariant 7).
- `docs/dqn-gpu-hot-path-audit.md` — cross-boundary call classification (Invariant 3, 7).
- `docs/ml-supervised-to-dqn-concept-audit.md` — Part E deliverable.
- `config/metric-bands.toml` — regression-detection bands (A.4, Invariant 7).
Pre-commit hook rejects component-adding commits that don't touch the relevant audit docs.
---
## 8. Open questions (to resolve during implementation)
- D.8: TLOB end-to-end trainable vs frozen-with-side-stream retraining. Decide based on per-step cost benchmark during D.8 impl.
- E.4, E.5, E.6: xLSTM / KAN / encoder-decoder — each produces a short design note; adoption decided per-note, not in this spec.
- C.6: ordering of individual `AdaptiveController` migrations — atoms-first or gamma-first? Decide at the start of C.6's internal commit series based on which has the simplest ISV surface area.
- A.3 multi-seed: N = 5 seeds may be too expensive for L40S budget; may need to run the production-readiness validation on H100 pool instead. Resource plan finalised with A.3 impl.
---
## 9. Non-goals
Explicit things this spec does NOT address:
- PPO trainer path (separate model).
- ml-supervised training itself (only its borrowable concepts).
- Deployment / paper trading harness.
- Data-pipeline changes (MBP-10 ingest, fxcache format).
- TFT as a separate predictive model (only TFT concepts borrowed into DQN via E.1, E.2, E.3).
---
**End of spec.**