diff --git a/docs/superpowers/plans/2026-04-24-dqn-v2-plan-1-substrate.md b/docs/superpowers/plans/2026-04-24-dqn-v2-plan-1-substrate.md new file mode 100644 index 000000000..d7755ac85 --- /dev/null +++ b/docs/superpowers/plans/2026-04-24-dqn-v2-plan-1-substrate.md @@ -0,0 +1,1612 @@ +# DQN v2 Plan 1 — Substrate & Refactor Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Lay the substrate that all subsequent DQN v2 work depends on: state reset registry, ISV slot contract, audit docs, named-dimension constants, and the AdaptiveController unification refactor. Zero behavioural change to the trained policy — this plan only establishes scaffolding and refactors existing ad-hoc machinery into typed protocols. + +**Architecture:** Three audit docs + one config file tracked by pre-commit hook. One `StateResetRegistry` struct categorising every piece of training state. One typed `IsvSlot` contract with a schema-version header. One `AdaptiveController` trait with existing controllers migrated one-at-a-time. Named constants for every semantic index across portfolio state, ISV slots, plan_isv dimensions, plan_params, state-vector offsets, branch indices, and action sub-indices. + +**Tech Stack:** Rust (workspace crates: `ml`, `ml-core`, `ml-dqn`, `ml-supervised`), CUDA C++ via nvcc through `build.rs`, cudarc 0.19, Cargo 2021 edition, SQLx offline mode, sccache with `CARGO_INCREMENTAL=0`. + +**Authority for this plan:** `docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md` — every task in this plan maps to a spec section. Every commit preserves all 9 invariants. + +--- + +## Pre-plan: verify current tree state + +Before starting any task, verify the baseline compiles and no in-flight validation run is active. + +- [ ] **Step 0.1: Verify tree builds** + +Run: +```bash +SQLX_OFFLINE=true CARGO_INCREMENTAL=0 RUSTC_WRAPPER=~/.local/bin/sccache cargo check -p ml +``` +Expected: `Finished 'dev' profile [unoptimized + debuginfo] target(s) in Xs` with no errors. + +- [ ] **Step 0.2: Verify no running training blocks the workflow** + +Run: +```bash +kubectl get wf -n foxhunt --field-selector=status.phase=Running 2>&1 | grep -v ci-pipeline | head -5 +``` +Expected: empty or only `NAME STATUS AGE MESSAGE` header. If any `train-*` workflow is Running, stop it with `argo stop -n foxhunt ` before proceeding. + +- [ ] **Step 0.3: Confirm spec version being implemented** + +Run: +```bash +git log --oneline docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md | head -3 +``` +Expected: top commit `336ee40b9` or later. If the spec has newer commits, re-read before proceeding. + +--- + +## Task 1: Create audit docs + pre-commit hook scaffolding + +**Files:** +- Create: `docs/dqn-wire-up-audit.md` +- Create: `docs/isv-slots.md` +- Create: `docs/dqn-gpu-hot-path-audit.md` +- Create: `docs/dqn-named-dims.md` +- Create: `docs/ml-supervised-to-dqn-concept-audit.md` +- Create: `config/metric-bands.toml` +- Modify: `.claude/helpers/precommit-check.sh` (or wherever the pre-commit hook lives) + +- [ ] **Step 1.1: Locate the existing pre-commit hook** + +Run: +```bash +find .claude .git/hooks scripts -name "*precommit*" -o -name "*pre-commit*" 2>/dev/null | head +cat .git/hooks/pre-commit 2>/dev/null | head -30 +``` + +Expected: locate the hook file that runs `.unwrap()` and stub checks during commits. This is the file we will extend. + +- [ ] **Step 1.2: Write `docs/dqn-wire-up-audit.md` initial skeleton** + +Content: +```markdown +# DQN v2 Wire-Up Audit + +**Status:** Populated during task #95 (A.5 orphan audit). Updated on every commit per Invariant 7. + +**Legend:** +- `Wired` — consumed by production training + val path. +- `Partial` — consumed on one side only (training OR val, or forward OR backward). +- `Orphan` — built but no production consumer. +- `Ghost` — consumer path is stubbed. + +## Components + +| Module / kernel | Consumer path | Classification | Notes | +|---|---|---|---| +| (populated during A.5 audit) | | | | +``` + +- [ ] **Step 1.3: Write `docs/isv-slots.md` initial skeleton** + +Content: +```markdown +# ISV Slot Allocation Registry + +**Source of truth for ISV bus slot allocations.** Every slot has a named constant in `gpu_dqn_trainer.rs`. This doc is the cross-reference. + +**Current `ISV_TOTAL_DIM` as of this plan's start:** 37. Post-full DQN v2 rollout: 72. + +| Index | Name constant | Type | Producer | Consumers | Reset-category | Notes | +|---|---|---|---|---|---|---| +| [0] | `ISV_SCHEMA_VERSION_INDEX` | u32 | construct | load | SchemaContract | Bumped on any slot-layout change | +| [1..12) | (pre-existing) | f32 | varies | varies | varies | Legacy — audit during A.2 | +| [12] | `LEARNING_HEALTH_INDEX` | f32 | isv_signal_update | many | FoldReset | health score ∈ [0, 1] | +| [13..17) | `Q_MAG_MEAN_*_INDEX` | f32 | q_mag_means_reduce | c51 kernels | FoldReset | Quarter/Half/Full mag Q-mean EMAs + |Q| ref | +| [17..22) | `Q_DIR_MEAN_*_INDEX` | f32 | q_dir_means_reduce | c51 kernels | FoldReset | Short/Hold/Long/Flat dir Q-mean EMAs + |Q| ref | +| [22] | `SHARPE_EMA_INDEX` | f32 | training_loop host | isv_signal_update | FoldReset | Training Sharpe EMA | +| [23..31) | `V_{CENTER,HALF}_{DIR,MAG,ORD,URG}_INDEX` | f32 | update_eval_v_range | adaptive_atoms, warm_start | FoldReset | Per-branch Q-support | +| [31..35) | `GRAD_NORM_TARGET_*_INDEX` | f32 | grad_balance_isv_update | branch_grad_rescale | SoftReset(decay_bars=500) | Per-branch grad-norm target | +| [35] | `GRAD_SCALE_LIMIT_INDEX` | f32 | grad_balance_isv_update | branch_grad_rescale | SoftReset(decay_bars=500) | Scale clamp limit | +| [36] | `IQL_BRANCH_SCALE_FLOOR_INDEX` | f32 | construct (static) | iql_per_branch_advantage | SchemaContract | Per-sample branch_scales floor; safety bound | +| [37..72) | (reserved for DQN v2) | | | | | Allocated incrementally by Plans 2-5 | +``` + +- [ ] **Step 1.4: Write `docs/dqn-gpu-hot-path-audit.md` initial skeleton** + +Content: +```markdown +# DQN GPU Hot-Path Audit + +**Invariant 3 enforcement.** Every cross-boundary call (DtoH, HtoD, memcpy between host and device) in the DQN code path is classified below. No hot-path `memcpy` DtoH/HtoD is allowed — zero tolerance. Pinned + device-mapped memory is the ONLY permitted mechanism for per-step host-visible data. + +**Hot path definition:** every function called from `fused_training.rs::step_fused` or its descendants that executes once per training step (178 times per epoch at batch size 8192), including kernel invocations inside the captured CUDA graph (`adam_grad_child`, `forward_child`, `aux_grad_child`) and host-side orchestration between them. + +**Classification:** +- `OK-pinned` — uses the zero-copy pinned + device-mapped pattern (allowed). +- `MIGRATE` — uses `memcpy` but should be pinned. Fix during spec impl. +- `COLD-PATH` — acceptable placement (checkpoint save/load, fold transition, epoch-boundary validation), annotated with reason. + +| File:line | Call signature | Path class | Annotation | Status | +|---|---|---|---|---| +| (populated during A.6 audit) | | | | | +``` + +- [ ] **Step 1.5: Write `docs/dqn-named-dims.md` initial skeleton** + +Content: +```markdown +# DQN Named Dimensions Registry + +**Invariant 8 enforcement.** Every semantic index has a named constant. Raw indices (`[0]`, `[6]`, `[23]`) appear ONLY at definition sites. Consumer code always uses the named constant. + +## State vector offsets + +Defined in `crates/ml/src/cuda_pipeline/state_layout.cuh` and mirrored in Rust via `ml-core::state_layout`. + +| Constant | Value | Meaning | +|---|---|---| +| `SL_STATE_DIM` | 104 | Total state vector length | +| `SL_STATE_DIM_PADDED` | 128 | Padded for cuBLAS K-tile alignment | +| `SL_MARKET_DIM` | 42 | Market-feature block width | +| `SL_OFI_DIM` | 32 | OFI block width | +| `SL_MTF_DIM` | 16 | Multi-timeframe block width | +| `SL_PORTFOLIO_BASE_DIM` | 8 | Portfolio-base block width | +| `SL_PORTFOLIO_PLAN_DIM` | 6 | Plan-ISV block width (extended to 7 in Plan 2) | +| `SL_MARKET_START` | 0 | | +| `SL_OFI_START` | 42 | | +| `SL_MTF_START` | 74 | | +| `SL_PORTFOLIO_START` | 90 | | +| `SL_PLAN_ISV_START` | 98 | | + +## Portfolio state slots (ps[0..30)) + +Defined in `crates/ml/src/cuda_pipeline/experience_kernels.cu` header comments. Migrated to named constants during Task 4 of this plan. + +| Index | Constant | Meaning | +|---|---|---| +| 0 | `PS_VALUE` | Equity | +| 1 | `PS_POSITION` | Current position | +| 2 | `PS_CASH` | Available cash | +| 3 | `PS_ENTRY_PRICE` | Entry price | +| 4 | `PS_MAX_EQUITY` | Peak equity (for drawdown) | +| 5 | `PS_HOLD_TIME` | Bars held | +| 6 | `PS_CUM_RETURN` | Cumulative return | +| 7 | `PS_STEP_COUNT` | Bars in window | +| 8 | `PS_SPREAD_COST` | Current spread cost | +| 9 | `PS_PREV_EQUITY` | Equity at previous step | +| 10 | `PS_HOLD_TIME_TOTAL` | Total hold time | +| 11 | `PS_REALIZED_PNL` | Cumulative realised PnL | +| 12 | `PS_WIN_COUNT_UNUSED` | (slot retained, not currently used) | +| 13 | `PS_LOSS_COUNT_UNUSED` | (slot retained, not currently used) | +| 14 | `PS_KELLY_WIN_COUNT` | Kelly stats: winning trades | +| 15 | `PS_KELLY_LOSS_COUNT` | Kelly stats: losing trades | +| 16 | `PS_KELLY_SUM_WINS` | Kelly stats: sum of winning returns | +| 17 | `PS_KELLY_SUM_LOSSES` | Kelly stats: sum of \|losing returns\| | +| 18 | `PS_RESERVED_18` | reserved | +| 19 | `PS_RESERVED_19` | reserved | +| 20 | `PS_INTRA_TRADE_MAX_DD` | worst intra-trade drawdown (v8 reward) | +| 21 | `PS_INTRA_TRADE_MAX_PNL` | best intra-trade unrealised PnL (hindsight) | +| 22 | `PS_RESERVED_22` | (was interval_sum, slot retained) | +| 23 | `PS_PLAN_TARGET_BARS` | plan: max hold bars (>0.5 = plan active) | +| 24 | `PS_PLAN_PROFIT_TARGET` | plan: raw profit threshold | +| 25 | `PS_PLAN_STOP_LOSS` | plan: raw stop threshold | +| 26 | `PS_PLAN_SCALE_AGGRESSION` | plan: position ramp speed | +| 27 | `PS_PLAN_CONVICTION` | plan: conviction at entry ∈ [0,1] | +| 28 | `PS_PLAN_ASYMMETRY` | plan: profit/stop asymmetry | +| 29 | `PS_PLAN_ENTRY_REGIME` | plan: regime_stability at entry (for P12) | + +## Plan_isv dimensions (plan_isv[0..6)) + +Will extend to 7 in Plan 2 (D.6 remaining_fraction). These constants live alongside `SL_PLAN_ISV_START` in `state_layout.cuh`. + +| Index | Constant | Meaning | +|---|---|---| +| 0 | `PLAN_ISV_PROGRESS` | hold_time / target_bars (clamped 2.0) | +| 1 | `PLAN_ISV_PNL_VS_TARGET` | unrealized / (target × equity) | +| 2 | `PLAN_ISV_PNL_VS_STOP` | -unrealized / (stop × equity) | +| 3 | `PLAN_ISV_ENTRY_CONVICTION` | conviction at entry | +| 4 | `PLAN_ISV_CONVICTION_DRIFT` | current conviction − entry conviction | +| 5 | `PLAN_ISV_REGIME_SHIFT` | \|isv[11] − entry_stability\| | + +## Plan_params output (plan_params[0..6)) + +From the trade_plan MLP output. + +| Index | Constant | Meaning | +|---|---|---| +| 0 | `PLAN_PARAM_TARGET_BARS` | | +| 1 | `PLAN_PARAM_PROFIT_TARGET` | | +| 2 | `PLAN_PARAM_STOP_LOSS` | | +| 3 | `PLAN_PARAM_SCALE_AGGRESSION` | | +| 4 | `PLAN_PARAM_CONVICTION` | | +| 5 | `PLAN_PARAM_ASYMMETRY` | | + +## Branch indices (4-branch factored action) + +| Index | Constant | Meaning | +|---|---|---| +| 0 | `BRANCH_DIR` | Direction branch (4 actions) | +| 1 | `BRANCH_MAG` | Magnitude branch (3 actions) | +| 2 | `BRANCH_ORD` | Order-type branch (3 actions) | +| 3 | `BRANCH_URG` | Urgency branch (3 actions) | + +## Direction action sub-indices + +| Index | Constant | Meaning | +|---|---|---| +| 0 | `DIR_SHORT` | Open short (or close long + open short) | +| 1 | `DIR_HOLD` | Keep current position (no-op on position) | +| 2 | `DIR_LONG` | Open long (or close short + open long) | +| 3 | `DIR_FLAT` | Close any position to zero | + +## Magnitude action sub-indices + +| Index | Constant | Meaning | +|---|---|---| +| 0 | `MAG_QUARTER` | 0.25× max_position | +| 1 | `MAG_HALF` | 0.50× max_position | +| 2 | `MAG_FULL` | 1.00× max_position | +``` + +- [ ] **Step 1.6: Write `docs/ml-supervised-to-dqn-concept-audit.md` initial skeleton** + +Content: +```markdown +# Supervised → DQN Concept Audit (Part E deliverable) + +Authoritative decisions on which ml-supervised concepts the DQN adopts. Per Invariant 9 (no deferred work), every entry is either `IN`, `OUT`, or `AUDIT-THEN-DECIDE` with the audit producing one of the two terminal classifications during this spec's implementation. + +| Concept | Supervised home | Classification | Action plan | Status | +|---|---|---|---|---| +| TFT Variable Selection Network | ml-supervised::tft | IN | E.1: extend vsn_mag/vsn_dir to full VSN over state feature groups | pending | +| Gated Residual Network | ml-supervised::tft | IN-IF-ABSENT | E.2: if A.5 audit finds absent, adopt; else mark existing as canonical | pending | +| TFT multi-quantile heads | ml-supervised::tft | IN | E.3: extend IQN to 5/25/50/75/95 quantile decomposition | pending | +| TLOB attention encoder | ml-supervised::tlob | IN | D.8 (Plan 2): wire to DQN trunk | pending | +| Mamba2 forward | ml-supervised::mamba2 | WIRED | already in DQN trunk | — | +| Mamba2 backward | ml-supervised::mamba2 | IN | D.1 (Plan 2): complete | pending | +| Liquid Time-constant | ml-supervised::liquid | AUDIT-THEN-WIRE-OR-DELETE | D.7 (Plan 2): audit decides | pending | +| xLSTM | ml-supervised::xlstm | OUT | Redundant with Mamba2 + TLOB; no DQN wiring | — | +| KAN | ml-supervised::kan | OUT | Function-approx not a bottleneck; no DQN wiring | — | +| Encoder-decoder separation | ml-supervised::tft | IN | E.4: extend D.3 horizon-decomposed V to full trunk/head separation | pending | +| Attention-weight interpretability | various | IN | E.5: ISV-expose attention weights from all temporal heads | pending | +| Multi-task auxiliary heads | ml-supervised | IN | E.6: next-return + regime-classification auxiliary heads | pending | +``` + +- [ ] **Step 1.7: Write `config/metric-bands.toml` initial content** + +Content: +```toml +# Regression-detection bands for DQN training metrics. +# Invariant 7. Per A.4, bands populated from last 3 known-good runs. +# Band = [mean − 3σ, mean + 3σ] per metric; recomputed after baseline promotion. +# N consecutive out-of-band epochs → WARN; 2N consecutive → ERROR + SIGTERM-self. + +[settings] +consecutive_epochs_for_warn = 3 +consecutive_epochs_for_error = 6 + +# Placeholder bands — populated by A.4.1 nsys baseline run + populate_metric_bands.sh +# (populated during Plan 5 execution; this file lands with empty metric entries for Plan 1). + +[bands] +``` + +- [ ] **Step 1.8: Extend pre-commit hook to reject commits missing audit-doc updates** + +Modify the pre-commit hook (path found in Step 1.1; typical location `.claude/helpers/precommit-check.sh` or `.git/hooks/pre-commit`). Add this function block: + +```bash +# DQN v2 Invariant 7 enforcement: component-adding commits must update audit docs. +check_audit_doc_updates() { + local staged=$(git diff --cached --name-only) + # Detect "component-adding" commits: new .rs or .cu files, or modifications to + # files that define ISV slots (gpu_dqn_trainer.rs), kernels (cuda_pipeline/*.cu), + # or state layout (state_layout.cuh). + local component_changes=$(echo "$staged" | grep -E '(^crates/ml/src/(cuda_pipeline|trainers/dqn)/|^crates/ml/src/cuda_pipeline/state_layout\.cuh$)') + if [ -z "$component_changes" ]; then return 0; fi + + local audit_docs_touched=$(echo "$staged" | grep -E '^docs/(dqn-wire-up-audit|isv-slots|dqn-gpu-hot-path-audit|dqn-named-dims|ml-supervised-to-dqn-concept-audit)\.md$') + if [ -z "$audit_docs_touched" ]; then + echo "❌ Invariant 7 violation: component changes without audit-doc update" + echo " Changed components:" + echo "$component_changes" | sed 's/^/ /' + echo " You must update one of:" + echo " docs/dqn-wire-up-audit.md" + echo " docs/isv-slots.md" + echo " docs/dqn-gpu-hot-path-audit.md" + echo " docs/dqn-named-dims.md" + echo " docs/ml-supervised-to-dqn-concept-audit.md" + return 1 + fi +} + +# DQN v2 Invariant 9 enforcement: reject TODO/FIXME/XXX/HACK/TBD markers in added code. +check_no_todo_fixme() { + local added_lines=$(git diff --cached --diff-filter=AM -U0 -- '*.rs' '*.cu' '*.cuh' \ + | grep -E '^\+' | grep -vE '^\+\+\+') + local bad=$(echo "$added_lines" | grep -E '(TODO|FIXME|XXX|HACK|TBD|unimplemented!|todo!\()') + if [ -n "$bad" ]; then + echo "❌ Invariant 9 violation: deferred-work markers found in staged code" + echo "$bad" | sed 's/^/ /' + return 1 + fi +} + +check_audit_doc_updates || exit 1 +check_no_todo_fixme || exit 1 +``` + +Insert these checks just before the hook's existing final `exit 0`. + +- [ ] **Step 1.9: Test the pre-commit hook rejects a stub-marker commit** + +Run: +```bash +cd /tmp && mkdir -p dqn-v2-hook-test && cd dqn-v2-hook-test +git init -q +# Copy the modified hook into a scratch repo +# (or test directly in the main repo after step 1.8 is in place) + +cd ~/Work/foxhunt +echo "// TODO: implement me" > crates/ml/src/trainers/dqn/test_hook_rejection.rs +git add crates/ml/src/trainers/dqn/test_hook_rejection.rs +git commit -m "test hook" 2>&1 +``` + +Expected: commit is rejected with `❌ Invariant 9 violation: deferred-work markers found in staged code`. + +Clean up: +```bash +rm crates/ml/src/trainers/dqn/test_hook_rejection.rs +git reset HEAD -- crates/ml/src/trainers/dqn/test_hook_rejection.rs 2>/dev/null +``` + +- [ ] **Step 1.10: Commit the audit scaffolding** + +```bash +git add docs/dqn-wire-up-audit.md docs/isv-slots.md docs/dqn-gpu-hot-path-audit.md \ + docs/dqn-named-dims.md docs/ml-supervised-to-dqn-concept-audit.md \ + config/metric-bands.toml +# add the pre-commit hook path that was modified in Step 1.8 +git add +git commit -m "$(cat <<'EOF' +infra(dqn-v2): audit doc scaffolding + pre-commit enforcement + +Plan 1 Task 1. Creates the five audit docs plus config/metric-bands.toml +that track Invariants 2, 7, 8 per the DQN v2 spec, and extends the +pre-commit hook with two checks: + + - component-adding commits must touch an audit doc (Invariant 7) + - added code may not contain TODO/FIXME/XXX/HACK/TBD/unimplemented!/ + todo! markers (Invariant 9) + +Tests: manually verified by staging a TODO-marked file; commit +rejected with the correct error message. + +Co-Authored-By: Claude Opus 4.7 (1M context) +EOF +)" +``` + +Expected: commit succeeds. + +--- + +## Task 2: Define the StateResetRegistry (A.1) + +**Files:** +- Create: `crates/ml/src/trainers/dqn/state_reset_registry.rs` +- Modify: `crates/ml/src/trainers/dqn/mod.rs` (export the new module) +- Test: `crates/ml/src/trainers/dqn/state_reset_registry.rs` (inline `#[cfg(test)]` block) + +- [ ] **Step 2.1: Write the failing test for StateResetRegistry classification** + +Create `crates/ml/src/trainers/dqn/state_reset_registry.rs` with only the test module first: + +```rust +//! Formal classification of every piece of training-time state by reset lifecycle. +//! See `docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md` §4.A.1. + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn registry_classifies_known_state() { + let r = StateResetRegistry::new(); + assert_eq!(r.category("kelly_stats"), Some(ResetCategory::FoldReset)); + assert_eq!(r.category("adaptive_gamma"), Some(ResetCategory::SoftReset { decay_bars: 500 })); + assert_eq!(r.category("network_params"), Some(ResetCategory::TrainingPersist)); + assert_eq!(r.category("ISV_SCHEMA_VERSION"), Some(ResetCategory::SchemaContract)); + } + + #[test] + fn registry_fold_reset_iterates_only_fold_reset_entries() { + let r = StateResetRegistry::new(); + let names: Vec<&str> = r.fold_reset_entries().map(|e| e.name).collect(); + assert!(names.contains(&"kelly_stats")); + assert!(!names.contains(&"network_params")); + } + + #[test] + fn registry_unknown_name_returns_none() { + let r = StateResetRegistry::new(); + assert_eq!(r.category("nonexistent_state"), None); + } +} +``` + +- [ ] **Step 2.2: Run test to verify it fails** + +Run: +```bash +SQLX_OFFLINE=true CARGO_INCREMENTAL=0 cargo test -p ml state_reset_registry 2>&1 | tail -20 +``` + +Expected: compile errors `cannot find type StateResetRegistry` / `cannot find type ResetCategory`. + +- [ ] **Step 2.3: Implement the StateResetRegistry** + +Add to the top of `crates/ml/src/trainers/dqn/state_reset_registry.rs`, above the test module: + +```rust +/// Reset lifecycle category for a piece of training-time state. +/// +/// See `docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md` §4.A.1. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum ResetCategory { + /// Cleared at fold boundary. + FoldReset, + /// Cleared at validation-window boundary. + WindowReset, + /// Anneals toward bootstrap value over `decay_bars` bars at fold boundary. + SoftReset { decay_bars: u32 }, + /// Survives across folds (learned weights). + TrainingPersist, + /// Version-stable across runs; touched only by explicit migration. + SchemaContract, +} + +/// One entry in the reset registry. +#[derive(Debug, Clone, Copy)] +pub struct RegistryEntry { + pub name: &'static str, + pub category: ResetCategory, + /// Short description — what this state represents. Used in diagnostic dumps. + pub description: &'static str, +} + +/// Typed registry classifying every piece of training-time state. +/// +/// Populated at construction; read-only thereafter. Adding new state requires +/// adding a new entry here in the same commit that introduces the state +/// (per Invariant 2 Wire-It-Up). +pub struct StateResetRegistry { + entries: Vec, +} + +impl StateResetRegistry { + /// Build the registry with all currently-known training state. + pub fn new() -> Self { + let entries = vec![ + // ───── Fold-reset state ───────────────────────────────────── + RegistryEntry { + name: "kelly_stats", + category: ResetCategory::FoldReset, + description: "Per-window Kelly win/loss counters (ps[14..18])", + }, + RegistryEntry { + name: "plan_state", + category: ResetCategory::FoldReset, + description: "Val plan_state_buf [N, 7] — plan entry snapshots", + }, + RegistryEntry { + name: "eval_q_mean_ema", + category: ResetCategory::FoldReset, + description: "Per-branch Q-mean EMAs in update_eval_v_range", + }, + RegistryEntry { + name: "eval_q_std_ema", + category: ResetCategory::FoldReset, + description: "Per-branch Q-std EMAs in update_eval_v_range", + }, + RegistryEntry { + name: "isv_v_range_slots", + category: ResetCategory::FoldReset, + description: "ISV[23..31) — v_center/v_half per branch", + }, + RegistryEntry { + name: "isv_learning_health", + category: ResetCategory::FoldReset, + description: "ISV[12] — learning_health EMA", + }, + RegistryEntry { + name: "isv_sharpe_ema", + category: ResetCategory::FoldReset, + description: "ISV[22] — training Sharpe EMA", + }, + RegistryEntry { + name: "isv_q_means", + category: ResetCategory::FoldReset, + description: "ISV[13..22) — per-direction/magnitude Q-mean EMAs + |Q| refs", + }, + // ───── Window-reset state ─────────────────────────────────── + RegistryEntry { + name: "portfolio_state", + category: ResetCategory::WindowReset, + description: "Per-window portfolio (ps[0..8]) — reset at window start", + }, + RegistryEntry { + name: "hold_time", + category: ResetCategory::WindowReset, + description: "ps[5] / ps[10] — consecutive bars in position", + }, + RegistryEntry { + name: "entry_price", + category: ResetCategory::WindowReset, + description: "ps[3] — entry price of current trade", + }, + // ───── Soft-reset state (anneals across folds) ────────────── + RegistryEntry { + name: "adaptive_gamma", + category: ResetCategory::SoftReset { decay_bars: 500 }, + description: "training_loop.rs adaptive_gamma — anneals to 0.905 bootstrap", + }, + RegistryEntry { + name: "isv_grad_balance_targets", + category: ResetCategory::SoftReset { decay_bars: 500 }, + description: "ISV[31..35) — per-branch grad-norm targets", + }, + RegistryEntry { + name: "isv_grad_scale_limit", + category: ResetCategory::SoftReset { decay_bars: 500 }, + description: "ISV[35] — scale-clamp limit EMA", + }, + // ───── Training-persist state ─────────────────────────────── + RegistryEntry { + name: "network_params", + category: ResetCategory::TrainingPersist, + description: "Online network weights (all tensors)", + }, + RegistryEntry { + name: "target_network", + category: ResetCategory::TrainingPersist, + description: "Target network weights — Polyak EMA of online", + }, + RegistryEntry { + name: "adam_momentum", + category: ResetCategory::TrainingPersist, + description: "Adam optimizer first moment (m)", + }, + RegistryEntry { + name: "adam_variance", + category: ResetCategory::TrainingPersist, + description: "Adam optimizer second moment (v)", + }, + RegistryEntry { + name: "adam_step", + category: ResetCategory::TrainingPersist, + description: "Adam step counter t", + }, + RegistryEntry { + name: "replay_buffer", + category: ResetCategory::TrainingPersist, + description: "PER experience buffer — persists across folds", + }, + // ───── Schema-contract state ──────────────────────────────── + RegistryEntry { + name: "ISV_SCHEMA_VERSION", + category: ResetCategory::SchemaContract, + description: "ISV[0] — slot-layout schema version (migrated only)", + }, + RegistryEntry { + name: "isv_iql_branch_scale_floor", + category: ResetCategory::SchemaContract, + description: "ISV[36] — IQL branch_scales floor (safety bound, bootstrap-only)", + }, + ]; + Self { entries } + } + + /// Look up a state's category by name. Returns `None` for unknown names + /// (caller should assert Some before use to catch typos). + pub fn category(&self, name: &str) -> Option { + self.entries.iter().find(|e| e.name == name).map(|e| e.category) + } + + /// Iterator over all entries with `FoldReset` category. + /// Used by the fold-boundary reset logic. + pub fn fold_reset_entries(&self) -> impl Iterator { + self.entries.iter().filter(|e| e.category == ResetCategory::FoldReset) + } + + /// Iterator over all entries with `WindowReset` category. + pub fn window_reset_entries(&self) -> impl Iterator { + self.entries.iter().filter(|e| e.category == ResetCategory::WindowReset) + } + + /// Iterator over all entries with `SoftReset` category. + pub fn soft_reset_entries(&self) -> impl Iterator { + self.entries.iter().filter(|e| matches!(e.category, ResetCategory::SoftReset { .. })) + } + + /// All entries (debug/diagnostic dump). + pub fn all(&self) -> &[RegistryEntry] { + &self.entries + } +} + +impl Default for StateResetRegistry { + fn default() -> Self { + Self::new() + } +} +``` + +- [ ] **Step 2.4: Export from the dqn module** + +Modify `crates/ml/src/trainers/dqn/mod.rs`. Find the section with other `pub mod` declarations. Add: + +```rust +pub mod state_reset_registry; +pub use state_reset_registry::{ResetCategory, RegistryEntry, StateResetRegistry}; +``` + +- [ ] **Step 2.5: Run test to verify it passes** + +Run: +```bash +SQLX_OFFLINE=true CARGO_INCREMENTAL=0 cargo test -p ml state_reset_registry 2>&1 | tail -10 +``` + +Expected: `test result: ok. 3 passed; 0 failed`. + +- [ ] **Step 2.6: Add an audit-doc entry for state_reset_registry** + +Append to `docs/dqn-wire-up-audit.md` below the legend: + +```markdown +| `crates/ml/src/trainers/dqn/state_reset_registry.rs` | consumers in Plan 1 Task 3+ (reset_fold_state call) | Wired (consumer added in same plan) | A.1 primary implementation | +``` + +- [ ] **Step 2.7: Commit** + +```bash +git add crates/ml/src/trainers/dqn/state_reset_registry.rs \ + crates/ml/src/trainers/dqn/mod.rs \ + docs/dqn-wire-up-audit.md +git commit -m "$(cat <<'EOF' +feat(dqn-v2): A.1 state reset registry + +Typed classification of every piece of training state by reset +lifecycle: FoldReset, WindowReset, SoftReset(decay_bars), +TrainingPersist, SchemaContract. + +Replaces the previous vibes-driven fold-boundary reset logic with an +explicit registry. Adding new state requires adding an entry in the +same commit (Invariant 2 Wire-It-Up). + +Consumer wiring (reset_fold_state, reset_window_state helpers that +iterate the registry) follows in Task 3 of Plan 1. + +Tests: 3 unit tests covering category lookup, fold-reset filtering, +unknown-name handling. + +Authority: docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md §4.A.1 + +Co-Authored-By: Claude Opus 4.7 (1M context) +EOF +)" +``` + +Expected: commit succeeds. + +--- + +## Task 3: Wire StateResetRegistry into the fold-boundary reset path + +**Files:** +- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs` — find the fold-boundary reset block +- Modify: `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` — `reset_eval_v_range_state` and related reset helpers + +- [ ] **Step 3.1: Locate the existing fold-boundary reset call site** + +Run: +```bash +grep -n "reset_eval_v_range_state\|fold_start\|fold boundary\|on_fold_start" \ + crates/ml/src/trainers/dqn/trainer/training_loop.rs | head -10 +``` + +Expected: find the function / block that runs at the start of a new fold. Note the line number for Step 3.2. + +- [ ] **Step 3.2: Add `reset_fold_state` using the registry** + +At the top of `crates/ml/src/trainers/dqn/trainer/training_loop.rs`, add the import: +```rust +use crate::trainers::dqn::{StateResetRegistry, ResetCategory}; +``` + +Find the fold-boundary block (Step 3.1) and replace the ad-hoc reset calls with a single iteration over the registry. Before: +```rust +// OLD (vibes-driven): +fused.reset_eval_v_range_state()?; +// other scattered reset calls... +``` + +After: +```rust +// NEW: registry-driven reset +let registry = StateResetRegistry::new(); +for entry in registry.fold_reset_entries() { + self.reset_named_state(entry.name, fold_idx)?; +} +// SoftReset handled separately (D.5 Plan 2); for this plan we keep the +// pre-existing ad-hoc handling and mark SoftReset entries as noop here. +``` + +- [ ] **Step 3.3: Implement `reset_named_state` dispatch** + +In the same file, add a new method on the training_loop's struct: + +```rust +/// Dispatch a named-state reset to its owning subsystem. +/// +/// Invariant: every entry in the StateResetRegistry's FoldReset category +/// has a match arm here. Unknown names are a bug (the registry entry +/// should have been added in the same commit as the state it describes). +fn reset_named_state(&mut self, name: &str, fold_idx: u32) -> Result<(), MLError> { + match name { + "kelly_stats" => { + // Kelly stats are held in portfolio state ps[14..18] — cleared + // per-window by the env_step kernel's fold_start detection. + // No-op here; the per-window reset logic handles it. + } + "plan_state" => { + if let Some(ref mut fused) = self.fused_ctx { + fused.zero_plan_state_buf()?; + } + } + "eval_q_mean_ema" | "eval_q_std_ema" | "isv_v_range_slots" => { + if let Some(ref mut fused) = self.fused_ctx { + fused.reset_eval_v_range_state()?; + } + } + "isv_learning_health" => { + if let Some(ref mut fused) = self.fused_ctx { + fused.write_isv_signal_at(crate::cuda_pipeline::gpu_dqn_trainer::LEARNING_HEALTH_INDEX, 0.5); + } + } + "isv_sharpe_ema" => { + if let Some(ref mut fused) = self.fused_ctx { + fused.write_isv_signal_at(crate::cuda_pipeline::gpu_dqn_trainer::SHARPE_EMA_INDEX, 0.0); + } + } + "isv_q_means" => { + if let Some(ref mut fused) = self.fused_ctx { + // Zero the 9 Q-mean EMA slots [13..22). + for idx in 13..22 { + fused.write_isv_signal_at(idx, 0.0); + } + } + } + _ => { + return Err(MLError::ModelError(format!( + "StateResetRegistry fold-reset dispatch: unknown name '{}'. \ + Every FoldReset entry must have a dispatch arm in reset_named_state.", + name + ))); + } + } + tracing::debug!( + target: "state_reset_registry", + fold = fold_idx, + name = name, + "fold-reset applied" + ); + Ok(()) +} +``` + +- [ ] **Step 3.4: Add `zero_plan_state_buf` accessor on the fused trainer** + +Find `crates/ml/src/trainers/dqn/fused_training.rs` or `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` depending on where the backtest evaluator holds `plan_state_buf`. If in the evaluator only (which is val-side only), Task 3 does not need to clear it on fold boundary — skip this step and remove the `plan_state` arm from `reset_named_state` (also remove the registry entry). Note the discovery in the audit doc and the commit message. + +Otherwise, add: + +```rust +pub fn zero_plan_state_buf(&mut self) -> Result<(), MLError> { + self.stream.memset_zeros(&mut self.plan_state_buf) + .map_err(|e| MLError::ModelError(format!("zero plan_state_buf: {e}"))) +} +``` + +- [ ] **Step 3.5: Run the DQN smoke tests to verify fold-reset still works** + +Run: +```bash +FOXHUNT_TEST_DATA=test_data/futures-baseline SQLX_OFFLINE=true \ + cargo test -p ml --lib -- multi_fold_convergence --ignored --nocapture 2>&1 | tail -30 +``` + +Expected: smoke test passes. If it fails, the failure must be in a registry dispatch arm (compile-clean but behaviourally wrong) — fix in this step, not deferred. + +- [ ] **Step 3.6: Compile-check** + +```bash +SQLX_OFFLINE=true CARGO_INCREMENTAL=0 RUSTC_WRAPPER=~/.local/bin/sccache cargo check -p ml 2>&1 | tail -5 +``` + +Expected: `Finished 'dev' profile [unoptimized + debuginfo] target(s) in Xs` — no new warnings beyond the pre-existing 8. + +- [ ] **Step 3.7: Update audit docs** + +Append to `docs/dqn-wire-up-audit.md`: +```markdown +| `training_loop.rs::reset_named_state` | consumer of StateResetRegistry::fold_reset_entries | Wired | A.1 Task 3 dispatch | +``` + +- [ ] **Step 3.8: Commit** + +```bash +git add crates/ml/src/trainers/dqn/trainer/training_loop.rs \ + crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs \ + crates/ml/src/trainers/dqn/fused_training.rs \ + docs/dqn-wire-up-audit.md +git commit -m "$(cat <<'EOF' +feat(dqn-v2): A.1 wire StateResetRegistry into fold-boundary reset + +Replaces the ad-hoc scattered fold-boundary reset calls with a single +registry-driven iteration. Adding new fold-reset state now requires +adding a registry entry AND a dispatch arm in the same commit. + +Tests: multi_fold_convergence smoke passes; no behavioural change to +the reset semantics — only a refactor of the reset-dispatch layer. + +Authority: spec §4.A.1. Plan 1 Task 3. + +Co-Authored-By: Claude Opus 4.7 (1M context) +EOF +)" +``` + +--- + +## Task 4: Named-dimension refactor (Invariant 8) + +This is the largest task in Plan 1. Break into sub-tasks 4A–4F for portfolio state, plan_isv, plan_params, branch indices, direction sub-indices, magnitude sub-indices. + +**Files:** +- Modify: `crates/ml/src/cuda_pipeline/state_layout.cuh` — add all named constants +- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu` — replace raw `ps[N]` with named macros +- Modify: `crates/ml/src/cuda_pipeline/backtest_env_kernel.cu` — same +- Modify: `crates/ml/src/cuda_pipeline/backtest_plan_kernel.cu` — same +- Modify: `crates/ml/src/cuda_pipeline/trade_physics.cuh` — same +- Modify: `crates/ml-core/src/state_layout.rs` — Rust mirror +- Create: `docs/dqn-named-dims.md` updates (populated per sub-task) + +### Task 4A: Portfolio state slot constants (ps[0..30)) + +- [ ] **Step 4A.1: Add named `#define` constants in state_layout.cuh** + +Modify `crates/ml/src/cuda_pipeline/state_layout.cuh`. Add a new section after the existing `SL_*` block: + +```c +// ──────────────────────────────────────────────────────────────────────────── +// Portfolio state slot indices (ps[0..30)). +// Invariant 8: every slot has a named constant. Raw ps[N] access outside this +// header or within a kernel that imports this header is a lint violation. +// ──────────────────────────────────────────────────────────────────────────── +#define PS_VALUE 0 // equity (mark-to-market) +#define PS_POSITION 1 +#define PS_CASH 2 +#define PS_ENTRY_PRICE 3 +#define PS_MAX_EQUITY 4 // for drawdown tracking +#define PS_HOLD_TIME 5 // consecutive bars in position +#define PS_CUM_RETURN 6 +#define PS_STEP_COUNT 7 +#define PS_SPREAD_COST 8 +#define PS_PREV_EQUITY 9 +#define PS_HOLD_TIME_TOTAL 10 +#define PS_REALIZED_PNL 11 +#define PS_RESERVED_12 12 // (slot retained) +#define PS_RESERVED_13 13 // (slot retained) +#define PS_KELLY_WIN_COUNT 14 +#define PS_KELLY_LOSS_COUNT 15 +#define PS_KELLY_SUM_WINS 16 +#define PS_KELLY_SUM_LOSSES 17 +#define PS_RESERVED_18 18 +#define PS_RESERVED_19 19 +#define PS_INTRA_TRADE_MAX_DD 20 // worst intra-trade drawdown (v8 reward) +#define PS_INTRA_TRADE_MAX_PNL 21 // best intra-trade unrealized PnL +#define PS_RESERVED_22 22 // (was interval_sum, slot retained) +#define PS_PLAN_TARGET_BARS 23 // > 0.5 == plan active +#define PS_PLAN_PROFIT_TARGET 24 +#define PS_PLAN_STOP_LOSS 25 +#define PS_PLAN_SCALE_AGGRESSION 26 +#define PS_PLAN_CONVICTION 27 // conviction at entry [0, 1] +#define PS_PLAN_ASYMMETRY 28 +#define PS_PLAN_ENTRY_REGIME 29 // regime_stability at entry (P12) +#define PS_DIM 30 // sentinel: portfolio state size +``` + +- [ ] **Step 4A.2: Audit raw `ps[N]` access sites in kernels** + +Run: +```bash +grep -rn 'ps\[[0-9]' crates/ml/src/cuda_pipeline/*.cu crates/ml/src/cuda_pipeline/*.cuh \ + | grep -vE 'state_layout\.cuh' | wc -l +``` + +Record the count. This is the migration scope. + +- [ ] **Step 4A.3: Replace `ps[23]` with `PS_PLAN_TARGET_BARS` across all `.cu` / `.cuh` files** + +Run (safe because `#include "state_layout.cuh"` is already standard): +```bash +for f in crates/ml/src/cuda_pipeline/*.cu crates/ml/src/cuda_pipeline/*.cuh; do + sed -i 's/\bps\[23\]/ps[PS_PLAN_TARGET_BARS]/g' "$f" +done +``` + +- [ ] **Step 4A.4: Repeat for every ps[N] index** + +Script a batched replacement (keep the exact mapping from Step 4A.1): +```bash +cd crates/ml/src/cuda_pipeline +declare -A MAP=( + [0]=PS_VALUE + [1]=PS_POSITION + [2]=PS_CASH + [3]=PS_ENTRY_PRICE + [4]=PS_MAX_EQUITY + [5]=PS_HOLD_TIME + [6]=PS_CUM_RETURN + [7]=PS_STEP_COUNT + [8]=PS_SPREAD_COST + [9]=PS_PREV_EQUITY + [10]=PS_HOLD_TIME_TOTAL + [11]=PS_REALIZED_PNL + [12]=PS_RESERVED_12 + [13]=PS_RESERVED_13 + [14]=PS_KELLY_WIN_COUNT + [15]=PS_KELLY_LOSS_COUNT + [16]=PS_KELLY_SUM_WINS + [17]=PS_KELLY_SUM_LOSSES + [18]=PS_RESERVED_18 + [19]=PS_RESERVED_19 + [20]=PS_INTRA_TRADE_MAX_DD + [21]=PS_INTRA_TRADE_MAX_PNL + [22]=PS_RESERVED_22 + [23]=PS_PLAN_TARGET_BARS + [24]=PS_PLAN_PROFIT_TARGET + [25]=PS_PLAN_STOP_LOSS + [26]=PS_PLAN_SCALE_AGGRESSION + [27]=PS_PLAN_CONVICTION + [28]=PS_PLAN_ASYMMETRY + [29]=PS_PLAN_ENTRY_REGIME +) +for k in "${!MAP[@]}"; do + name="${MAP[$k]}" + for f in *.cu *.cuh; do + sed -i "s/\bps\[$k\]/ps[$name]/g" "$f" + done +done +cd - +``` + +- [ ] **Step 4A.5: Verify no stray `ps[digit]` remains** + +Run: +```bash +grep -rn 'ps\[[0-9]' crates/ml/src/cuda_pipeline/*.cu crates/ml/src/cuda_pipeline/*.cuh \ + | grep -vE 'state_layout\.cuh|PS_DIM' +``` + +Expected: empty output. Any remaining matches indicate a missed case — fix inline. + +- [ ] **Step 4A.6: Compile-check** + +```bash +SQLX_OFFLINE=true CARGO_INCREMENTAL=0 RUSTC_WRAPPER=~/.local/bin/sccache cargo check -p ml 2>&1 | tail -5 +``` + +Expected: Finished with no new warnings/errors. + +- [ ] **Step 4A.7: Commit** + +```bash +git add crates/ml/src/cuda_pipeline/state_layout.cuh \ + crates/ml/src/cuda_pipeline/*.cu \ + crates/ml/src/cuda_pipeline/*.cuh \ + docs/dqn-named-dims.md +git commit -m "$(cat <<'EOF' +refactor(dqn-v2): Invariant 8 — named constants for portfolio state ps[0..30) + +Replace raw ps[N] access across all CUDA kernels with PS_* named +constants defined in state_layout.cuh. 30 constants total; enforces +Invariant 8 for the portfolio-state block. + +Raw ps[digit] access outside state_layout.cuh is now a lint violation +(grep -rn 'ps\[[0-9]' should return zero outside the header). + +No behavioural change; pure refactor. + +Plan 1 Task 4A. Spec §3 Invariant 8. + +Co-Authored-By: Claude Opus 4.7 (1M context) +EOF +)" +``` + +### Tasks 4B–4F: Remaining named-dimension sub-tasks + +Follow the same pattern for: +- **4B: plan_isv dimensions (plan_isv[0..6))** — add `PLAN_ISV_*` constants; replace raw indices. +- **4C: plan_params (plan_params[0..6))** — add `PLAN_PARAM_*` constants. +- **4D: branch indices (0..4)** — add `BRANCH_DIR`, `BRANCH_MAG`, `BRANCH_ORD`, `BRANCH_URG`; replace raw 0/1/2/3 in branch-indexing code (carefully: not every 0..3 literal is a branch index). +- **4E: direction sub-indices** — add `DIR_SHORT`, `DIR_HOLD`, `DIR_LONG`, `DIR_FLAT`. +- **4F: magnitude sub-indices** — add `MAG_QUARTER`, `MAG_HALF`, `MAG_FULL`. + +Each sub-task follows the Task 4A template: add constants → audit raw-index access → sed replacement → verify empty grep → compile-check → commit with identical message structure. + +For 4D (branch indices), use more targeted sed patterns because raw `0..4` literals may appear in unrelated contexts. Replace only where context makes it a branch index: +```bash +# Example: in gpu_dqn_trainer.rs, &0_i32 vs &1_i32 in branch-arg positions +# Better: read the file and apply edits with explicit line numbers via Edit tool. +``` + +Record the final per-sub-task commit SHA in `docs/dqn-named-dims.md`. + +--- + +## Task 5: A.2 ISV Slot Contract + Schema Version + +**Files:** +- Modify: `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` — add `ISV_SCHEMA_VERSION_INDEX`, schema check in constructor +- Create: `crates/ml/src/cuda_pipeline/isv_bus.rs` — `IsvSlot` typed handle (optional, see Step 5.3 — may defer to later plans if adds too much scope) + +- [ ] **Step 5.1: Add `ISV_SCHEMA_VERSION_INDEX` constant** + +Modify `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` near the top where ISV slot constants are declared. Change `ISV_TOTAL_DIM` from 37 → 38 AND shift existing slot indices up by 1. Actually: + +**Decision (per spec §4.A.2):** The schema version is ISV[0] — requires shifting every existing ISV slot by 1. This is a large-but-tractable migration. + +Alternative decision: put the schema version at the END (ISV[37]) and reserve it there. Less disruption. + +**Per Invariant 9** (no deferred work) and spec-text "ISV slot [0] reserved as schema version": we do the shift. All call sites audit. + +Because this is substantial, split into sub-steps: + +- [ ] **Step 5.2: Decision gate — shift or append?** + +Re-read spec §4.A.2 verbatim. The spec says "ISV slot [0] reserved as schema version". Implementation choice per Invariant 9: shift. No append shortcut. + +- [ ] **Step 5.3: Verify scope of ISV-slot shift** + +Run: +```bash +grep -rn '_INDEX\s*:\s*usize\s*=\s*[0-9]' crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs | head -20 +grep -rn 'isv_signals\[[0-9]\]' crates/ml/src/cuda_pipeline/*.cu | head -20 +``` + +Note the count of ISV slot usages. All must be updated. + +- [ ] **Step 5.4: Shift every ISV slot index up by 1** + +Edit `gpu_dqn_trainer.rs` constants: every `pub const X_INDEX: usize = N` becomes `pub const X_INDEX: usize = N + 1`. Add new `pub const ISV_SCHEMA_VERSION_INDEX: usize = 0` at the top of the ISV constant block. Update `ISV_TOTAL_DIM` from 37 → 38. + +In kernel .cu files, replace every `isv_signals[N]` with `isv_signals[N+1]` — **but prefer** replacing with the named constant equivalent (since Task 4B–4F defined them). + +This is a substantial mechanical refactor. Use the Edit tool at each site with the full context around it; do not use sed for cu files because `isv_signals[13]` might be coincidentally part of unrelated expressions. + +- [ ] **Step 5.5: Set ISV[0] = current schema version in constructor** + +In the ISV bus construction block (`gpu_dqn_trainer.rs` around line 8069), after the `std::ptr::write_bytes(host_ptr as *mut u8, 0, num_bytes);` line and before the v-range bootstrap, add: + +```rust +// ISV schema version. Bumped on any slot-layout change. Checkpoint load +// reads this to detect schema drift and fail explicitly. +*sig_ptr.add(ISV_SCHEMA_VERSION_INDEX) = ISV_SCHEMA_VERSION_CURRENT as f32; +``` + +Where `ISV_SCHEMA_VERSION_CURRENT` is a new module-level constant: +```rust +/// Current ISV schema version. Bumped any time slot layout changes. +/// Plan 1 establishes version 1 (the initial versioned layout). +pub const ISV_SCHEMA_VERSION_CURRENT: u32 = 1; +``` + +- [ ] **Step 5.6: Add schema-check at checkpoint load** + +Locate the checkpoint load path (`crates/ml/src/trainers/dqn/trainer/constructor.rs` or similar, depending on where DQN checkpoint restore lives). Add: + +```rust +// ISV schema version check. Fail-fast on mismatch; no silent migration. +let checkpoint_version = read_isv_slot_from_checkpoint(&checkpoint, ISV_SCHEMA_VERSION_INDEX) as u32; +if checkpoint_version != ISV_SCHEMA_VERSION_CURRENT { + return Err(MLError::ModelError(format!( + "ISV schema version mismatch: checkpoint has {}, current code requires {}. \ + Re-train from scratch or add an explicit migration function. \ + See docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md §4.A.2.", + checkpoint_version, ISV_SCHEMA_VERSION_CURRENT + ))); +} +``` + +If `read_isv_slot_from_checkpoint` doesn't exist, add it in the same commit — it's a simple byte-offset read from the safetensors block that contains the ISV buffer. + +- [ ] **Step 5.7: Run DQN smoke tests to verify no regression** + +```bash +FOXHUNT_TEST_DATA=test_data/futures-baseline SQLX_OFFLINE=true \ + cargo test -p ml --lib -- smoke_tests --ignored --nocapture 2>&1 | tail -40 +``` + +Expected: all smoke tests pass. Any that reference ISV slot indices directly need updating (should have been caught in Step 5.4 grep). + +- [ ] **Step 5.8: Update docs/isv-slots.md with the shifted layout** + +Open `docs/isv-slots.md` and rewrite the table: ISV[0] is now `ISV_SCHEMA_VERSION_INDEX`, every other slot's index is +1. + +- [ ] **Step 5.9: Commit** + +```bash +git add crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs \ + crates/ml/src/cuda_pipeline/*.cu \ + crates/ml/src/trainers/dqn/trainer/*.rs \ + docs/isv-slots.md +git commit -m "$(cat <<'EOF' +feat(dqn-v2): A.2 ISV slot schema version at ISV[0] + +ISV_SCHEMA_VERSION_INDEX = 0, ISV_TOTAL_DIM = 38 (was 37). Every +existing ISV slot shifted up by 1. Constructor writes ISV[0] = +ISV_SCHEMA_VERSION_CURRENT (=1). Checkpoint load fails-fast on +version mismatch; no silent migration. + +Consumer migration: every isv_signals[N] reference updated. Uses +named constants (PS_*, BRANCH_*, etc.) where possible, per Invariant 8. + +Tests: all smoke tests pass (magnitude_distribution, +multi_fold_convergence, controller_activity, +reward_component_audit, exploration_coverage, +surrogate_noise_check). + +Plan 1 Task 5. Spec §4.A.2. + +Co-Authored-By: Claude Opus 4.7 (1M context) +EOF +)" +``` + +--- + +## Task 6: A.5 Orphan Audit — populate the wire-up doc + +**Files:** +- Modify: `docs/dqn-wire-up-audit.md` — populate from grep + +- [ ] **Step 6.1: Enumerate every pub module and kernel in the DQN path** + +Run: +```bash +{ + find crates/ml/src -name '*.rs' -exec grep -l '^pub mod\|^pub fn\|^pub struct' {} \; + find crates/ml/src/cuda_pipeline -name '*.cu' +} | sort -u +``` + +Pipe to a scratch file `/tmp/dqn-modules.txt`. + +- [ ] **Step 6.2: For each module, determine consumer** + +For each entry, grep for consumers: +```bash +while read -r file; do + if [[ "$file" == *.rs ]]; then + module_name=$(basename "$file" .rs) + consumer_count=$(grep -rl "use.*${module_name}\|${module_name}::" crates/ml/src --include='*.rs' | grep -v "$file" | wc -l) + echo "$file CONSUMERS=$consumer_count" + else + kernel_name=$(basename "$file" .cu) + consumer_count=$(grep -rl "$kernel_name" crates/ml/src --include='*.rs' | wc -l) + echo "$file CONSUMERS=$consumer_count" + fi +done < /tmp/dqn-modules.txt > /tmp/dqn-modules-classified.txt +head -50 /tmp/dqn-modules-classified.txt +``` + +- [ ] **Step 6.3: Fill in the audit doc** + +Open `docs/dqn-wire-up-audit.md` and, using `/tmp/dqn-modules-classified.txt` as input, populate the table with one row per module. Classification rules: +- `CONSUMERS=0` → `Orphan` +- `CONSUMERS≥1` → `Wired` if consumer is production path; `Partial` if only consumed by tests/benchmarks +- TFT-related, xLSTM, KAN, TLOB — copy decisions from `docs/ml-supervised-to-dqn-concept-audit.md` + +Take as long as needed; this is the one-time audit that Invariant 2 builds upon. + +- [ ] **Step 6.4: Resolve every `Orphan` discovered** + +For each module classified `Orphan`: +- If in scope for this plan (Plan 1) — wire it in the same commit as Task 6 completion. +- If in scope for Plan 2 (temporal) — note explicitly which Task in Plan 2 wires it. +- If in scope for Plan 4 (E.1–E.6) — note which E sub-task wires it. +- If genuinely unused — delete it in the same commit (per `feedback_wire_everything_up.md`). + +No orphan is left as "will decide later". Every one has a concrete action. + +- [ ] **Step 6.5: Commit** + +```bash +git add docs/dqn-wire-up-audit.md +# plus any deletion / wiring commits for orphans resolved inline +git commit -m "$(cat <<'EOF' +audit(dqn-v2): A.5 orphan audit complete + +Populated docs/dqn-wire-up-audit.md with every pub module and kernel +in the DQN path. Each entry classified as Wired / Partial / Orphan / +Ghost with the action plan linking to the plan+task that resolves +any non-Wired status. + +No Orphan left unclassified. Resolved orphans either wired in this +commit or explicitly linked to a later Plan's task. + +Plan 1 Task 6. Spec §4.A.5, Invariant 2. + +Co-Authored-By: Claude Opus 4.7 (1M context) +EOF +)" +``` + +--- + +## Task 7: A.6 Hot-Path Purity Audit + +**Files:** +- Modify: `docs/dqn-gpu-hot-path-audit.md` — populate + +- [ ] **Step 7.1: Grep every cross-boundary call** + +Run: +```bash +{ + grep -rn 'memcpy_dtoh\|memcpy_htod\|cudaMemcpy\|cuMemcpy' \ + crates/ml/src --include='*.rs' --include='*.cu' --include='*.cuh' 2>&1 + grep -rn '\.synchronize()\|stream_sync' \ + crates/ml/src --include='*.rs' 2>&1 +} | tee /tmp/dqn-xboundary.txt | head -50 +``` + +- [ ] **Step 7.2: Classify each call** + +For each line in `/tmp/dqn-xboundary.txt`: +1. Read the surrounding 20 lines of the file to determine context. +2. Classify: + - `OK-pinned` — the buffer is allocated with `cuMemAllocHost_v2 + cuMemHostGetDevicePointer_v2`. Check the allocation site. + - `COLD-PATH` — the call runs per-epoch (in update_eval_v_range, at fold boundary), per-checkpoint (save/load), or per-window boundary. + - `MIGRATE` — the call runs per-step and uses `memcpy` rather than pinned — this is an Invariant 3 violation. +3. Fill in `docs/dqn-gpu-hot-path-audit.md`. + +- [ ] **Step 7.3: Fix every `MIGRATE` classification** + +For each `MIGRATE` call, replace with pinned + device-mapped allocation in the same commit. This is per Invariant 9 — no deferred work. + +Typical fix pattern: +```rust +// BEFORE: +self.stream.memcpy_dtoh(&self.some_dev_buf, &mut host_scratch)?; + +// AFTER: allocate pinned + device-mapped ONCE in the constructor +// (host_scratch_pinned: *mut f32, host_scratch_dev_ptr: u64) +// Then in the hot path, just read from host_scratch_pinned — no call. +``` + +- [ ] **Step 7.4: Run smoke tests to verify no regression** + +```bash +FOXHUNT_TEST_DATA=test_data/futures-baseline SQLX_OFFLINE=true \ + cargo test -p ml --lib -- smoke_tests --ignored --nocapture 2>&1 | tail -40 +``` + +- [ ] **Step 7.5: Commit** + +```bash +git add crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs \ + crates/ml/src/trainers/dqn/fused_training.rs \ + docs/dqn-gpu-hot-path-audit.md +git commit -m "$(cat <<'EOF' +audit(dqn-v2): A.6 hot-path purity — zero memcpy DtoH in step loop + +Populated docs/dqn-gpu-hot-path-audit.md with every cross-boundary +call. Migrated N MIGRATE-classified calls to pinned + device-mapped +memory. Remaining calls are COLD-PATH (checkpoint/fold/epoch) and +annotated with their reason. + +Invariant 3 now enforced across the DQN path. + +Plan 1 Task 7. Spec §4.A.6, Invariant 3. + +Co-Authored-By: Claude Opus 4.7 (1M context) +EOF +)" +``` + +--- + +## Task 8: C.6 AdaptiveController trait + migrate first controller (atoms) + +**Files:** +- Create: `crates/ml/src/trainers/dqn/adaptive_controller.rs` +- Modify: `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` — use the trait for atom-support controller + +- [ ] **Step 8.1: Write failing test** + +Create `crates/ml/src/trainers/dqn/adaptive_controller.rs`: + +```rust +//! Unified protocol for adaptive controllers in the DQN trainer. +//! Spec §4.C.6. + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn controller_fire_rate_stats_compute() { + let mut stats = FireRateStats::default(); + stats.record_fire(true); + stats.record_fire(false); + stats.record_fire(true); + assert_eq!(stats.fire_count, 2); + assert_eq!(stats.observation_count, 3); + assert!((stats.fire_rate() - 2.0 / 3.0).abs() < 1e-9); + } + + #[test] + fn diag_snapshot_empty_by_default() { + let snap = DiagSnapshot::default(); + assert!(snap.fields.is_empty()); + } +} +``` + +- [ ] **Step 8.2: Run test to verify it fails** + +Run: +```bash +SQLX_OFFLINE=true CARGO_INCREMENTAL=0 cargo test -p ml adaptive_controller 2>&1 | tail -10 +``` + +Expected: compile errors `cannot find type FireRateStats`. + +- [ ] **Step 8.3: Implement the trait** + +Prepend to `adaptive_controller.rs` (above the test module): + +```rust +use std::collections::HashMap; + +/// Per-controller fire-rate statistics. A "fire" is any observation where +/// the controller emitted a control value different from its previous emission. +/// Used by the `controller_activity` smoke test to verify load-bearing +/// controllers don't fire in > 50% of epochs. +#[derive(Debug, Clone, Default)] +pub struct FireRateStats { + pub fire_count: u64, + pub observation_count: u64, +} + +impl FireRateStats { + pub fn record_fire(&mut self, fired: bool) { + self.observation_count += 1; + if fired { + self.fire_count += 1; + } + } + + pub fn fire_rate(&self) -> f64 { + if self.observation_count == 0 { + 0.0 + } else { + self.fire_count as f64 / self.observation_count as f64 + } + } + + pub fn reset(&mut self) { + *self = Self::default(); + } +} + +/// Standardised diagnostic emission from an adaptive controller. +/// Emitted at epoch boundary into HEALTH_DIAG. +#[derive(Debug, Clone, Default)] +pub struct DiagSnapshot { + /// Named fields → value. Example: `{"gamma": 0.95, "fire_rate": 0.12}`. + pub fields: HashMap, +} + +impl DiagSnapshot { + pub fn with(mut self, name: impl Into, value: f64) -> Self { + self.fields.insert(name.into(), value); + self + } +} + +/// Unified protocol for adaptive controllers in the DQN trainer. +/// +/// Each controller reads signals from the ISV bus, computes a control value +/// (gamma, atom positions, Kelly floor, CQL alpha, etc.), writes its output +/// back to ISV, and emits diagnostics. +pub trait AdaptiveController { + /// Opaque signal vector the controller reads. + type Signal; + /// Opaque control value the controller emits. + type Control; + + /// Read inputs from the ISV bus. + fn read_signals(&self, isv: &IsvBus) -> Self::Signal; + + /// Compute the new control value. May update internal state. + fn update(&mut self, signals: Self::Signal) -> Self::Control; + + /// Write the control value back to ISV / consumer buffers. + fn write_output(&self, ctrl: &Self::Control, isv: &mut IsvBus); + + /// Per-controller fire-rate statistics. + fn fire_rate(&self) -> &FireRateStats; + + /// Emit a diagnostic snapshot at epoch boundary. + fn diagnose(&self) -> DiagSnapshot; + + /// Controller name, used in HEALTH_DIAG field labels. + fn name(&self) -> &'static str; +} + +/// Abstraction over the ISV bus for controllers. +/// In production, this wraps `isv_signals_pinned: *mut f32`. +/// In tests, a plain `Vec` is used for isolation. +pub struct IsvBus<'a> { + pub(crate) slots: &'a mut [f32], +} + +impl<'a> IsvBus<'a> { + pub fn new(slots: &'a mut [f32]) -> Self { + Self { slots } + } + pub fn read(&self, idx: usize) -> f32 { self.slots[idx] } + pub fn write(&mut self, idx: usize, v: f32) { self.slots[idx] = v; } +} +``` + +- [ ] **Step 8.4: Run test to verify it passes** + +```bash +SQLX_OFFLINE=true CARGO_INCREMENTAL=0 cargo test -p ml adaptive_controller 2>&1 | tail -5 +``` + +Expected: `test result: ok. 2 passed; 0 failed`. + +- [ ] **Step 8.5: Commit the trait and harness** + +```bash +git add crates/ml/src/trainers/dqn/adaptive_controller.rs \ + crates/ml/src/trainers/dqn/mod.rs \ + docs/dqn-wire-up-audit.md +git commit -m "$(cat <<'EOF' +feat(dqn-v2): C.6 AdaptiveController trait + test harness + +Unified protocol for every adaptive controller in the DQN trainer. +FireRateStats for controller_activity smoke. DiagSnapshot for +standardised HEALTH_DIAG emission. IsvBus abstraction over pinned +device-mapped memory. + +No concrete controller migration yet — Task 9 migrates atoms (first +in the sequence per spec §8 C.6 decision: atoms → gamma → Kelly → +cql_alpha → tau → epsilon → conviction_floor → plan_threshold → +balancer). + +Tests: 2 unit tests on FireRateStats and DiagSnapshot. + +Plan 1 Task 8. Spec §4.C.6. + +Co-Authored-By: Claude Opus 4.7 (1M context) +EOF +)" +``` + +### Tasks 9 through 16: migrate each adaptive controller in spec-specified order + +For each controller, follow this pattern: + +1. **Write failing test** for the controller's `impl AdaptiveController`. +2. **Run test to fail** (trait impl doesn't exist yet). +3. **Implement the trait** for the controller; delete the old ad-hoc function. +4. **Wire consumers** to call through the trait (may be via a central dispatcher that holds all `Box`). +5. **Run test to pass.** +6. **Run smoke tests** (`controller_activity` is the key one — it asserts fire rates). +7. **Update docs/dqn-wire-up-audit.md and docs/isv-slots.md**. +8. **Commit** with spec-referenced message. + +Concrete controllers in spec order: +- Task 9: atoms controller (`recompute_atom_positions` + `warm_start_atom_positions`) +- Task 10: gamma controller (the `adaptive_gamma` in `training_loop.rs`) +- Task 11: Kelly cap controller (`kelly_position_cap`'s `conviction` + `safety_multiplier` computation) +- Task 12: cql_alpha controller (currently static; prepares for the seed-phase coupling in Plan 3) +- Task 13: tau controller (stochastic SARSA temperature in `c51_loss_kernel`) +- Task 14: epsilon controller (cosine-annealed with floor) +- Task 15: conviction-floor controller (IQL branch_scales floor, currently static at ISV[36]) +- Task 16: plan-threshold controller (currently hardcoded 0.5, prepares for B.4 in Plan 3) +- Task 17: grad-balancer controller (the `grad_balance_isv_update` kernel — wraps it under the trait) + +After the last migration (Task 17) is complete, the old ad-hoc update functions are all removed in Task 17's commit — per spec §4.C.6 "Old scaffolding is removed in the SAME commit that migrates its last consumer". + +--- + +## Task 18: Plan 1 validation run + +**Files:** +- None (runtime validation) + +- [ ] **Step 18.1: Compile-check** + +```bash +SQLX_OFFLINE=true CARGO_INCREMENTAL=0 RUSTC_WRAPPER=~/.local/bin/sccache cargo check -p ml 2>&1 | tail -5 +``` + +Expected: `Finished ... target(s) in Xs` — no new warnings beyond baseline. + +- [ ] **Step 18.2: Full smoke test sweep** + +```bash +FOXHUNT_TEST_DATA=test_data/futures-baseline SQLX_OFFLINE=true \ + cargo test -p ml --lib -- smoke_tests --ignored --nocapture 2>&1 | tail -60 +``` + +Expected: every smoke test passes (magnitude_distribution, multi_fold_convergence, controller_activity, reward_component_audit, exploration_coverage, surrogate_noise_check). + +- [ ] **Step 18.3: Push all Plan 1 commits to origin** + +```bash +git push origin main 2>&1 | tail -3 +``` + +Expected: push succeeds. Plan 1 landing complete. + +- [ ] **Step 18.4: Launch a short 3-epoch L40S training run to smoke-test the full pipeline** + +```bash +./scripts/argo-train.sh dqn --gpu-pool ci-training-l40s --baseline --epochs 3 2>&1 | grep "^Name:" +``` + +Record the workflow name. Monitor for: +- No ERROR-level logs. +- HEALTH_DIAG emissions include new adaptive-controller diag fields. +- Per-epoch timing within 10% of the baseline (from train-bnz2r earlier this session). + +- [ ] **Step 18.5: Commit "Plan 1 complete" note** + +```bash +echo "Plan 1 complete: " >> docs/superpowers/plans/2026-04-24-dqn-v2-plan-1-substrate.md +git add docs/superpowers/plans/2026-04-24-dqn-v2-plan-1-substrate.md +git commit -m "plan(dqn-v2): Plan 1 substrate + refactor complete" +git push origin main +``` + +--- + +## Plan 1 exit criteria + +Plan 1 is "done" when: + +1. All 9 invariants hold across every commit (spot-checked via the pre-commit hook firing correctly). +2. `docs/dqn-wire-up-audit.md`, `docs/isv-slots.md`, `docs/dqn-gpu-hot-path-audit.md`, `docs/dqn-named-dims.md`, `docs/ml-supervised-to-dqn-concept-audit.md`, `config/metric-bands.toml` all exist and are populated. +3. Every raw `ps[N]`, `plan_isv[N]`, `plan_params[N]` access has been replaced with a named constant. +4. `StateResetRegistry` is the sole fold-reset dispatch path. +5. `AdaptiveController` trait has 9 concrete impls covering every previous ad-hoc controller. +6. ISV schema version at ISV[0] is populated at construct and checked at load. +7. Hot-path purity audit has zero `MIGRATE` entries remaining. +8. All smoke tests pass. +9. A short 3-epoch L40S run produces the expected HEALTH_DIAG emissions and timing. + +Plan 2 (Temporal core) begins only after Plan 1 passes exit criteria. + +--- + +**End of Plan 1.**