diff --git a/docs/superpowers/plans/2026-04-24-dqn-v2-plan-1-substrate.md b/docs/superpowers/plans/2026-04-24-dqn-v2-plan-1-substrate.md index 1050a7496..ed6b01b7d 100644 --- a/docs/superpowers/plans/2026-04-24-dqn-v2-plan-1-substrate.md +++ b/docs/superpowers/plans/2026-04-24-dqn-v2-plan-1-substrate.md @@ -2,9 +2,9 @@ > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. -**Goal:** Lay the substrate that all subsequent DQN v2 work depends on: state reset registry, ISV slot contract, audit docs, named-dimension constants, and the AdaptiveController unification refactor. Zero behavioural change to the trained policy — this plan only establishes scaffolding and refactors existing ad-hoc machinery into typed protocols. +**Goal:** Lay the substrate that all subsequent DQN v2 work depends on: state reset registry, ISV slot contract, audit docs, named-dimension constants, and the AdaptiveMonitor unification refactor (GPU kernels compute; CPU observes, per spec §4.C.6 revision). Zero behavioural change to the trained policy — this plan only establishes scaffolding and refactors existing ad-hoc machinery into typed protocols. -**Architecture:** Three audit docs + one config file tracked by pre-commit hook. One `StateResetRegistry` struct categorising every piece of training state. One typed `IsvSlot` contract with a compile-time **layout fingerprint** (structural hash of the slot list) at ISV[0] — fail-fast only, no migration path ever exists. One `AdaptiveController` trait with existing controllers migrated one-at-a-time. Named constants for every semantic index across portfolio state, ISV slots, plan_isv dimensions, plan_params, state-vector offsets, branch indices, and action sub-indices. +**Architecture:** Three audit docs + one config file tracked by pre-commit hook. One `StateResetRegistry` struct categorising every piece of training state. One typed `IsvSlot` contract with a compile-time **layout fingerprint** (structural hash of the slot list) at ISV[0..2) — fail-fast only, no migration path ever exists. One read-only `AdaptiveMonitor` trait with 6 GPU-kernel-plus-CPU-monitor pairs (atoms, gamma, kelly_cap, tau, epsilon, grad_balancer) and 3 static-config ISV writes (cql_alpha, conviction_floor, plan_threshold). **GPU drives, CPU reads** — CPU-side code never computes an adaptive value; all computation is in GPU kernels reading from and writing to ISV. Named constants for every semantic index across portfolio state, ISV slots, plan_isv dimensions, plan_params, state-vector offsets, branch indices, and action sub-indices. **Tech Stack:** Rust (workspace crates: `ml`, `ml-core`, `ml-dqn`, `ml-supervised`), CUDA C++ via nvcc through `build.rs`, cudarc 0.19, Cargo 2021 edition, SQLx offline mode, sccache with `CARGO_INCREMENTAL=0`. @@ -1470,215 +1470,293 @@ EOF --- -## Task 8: C.6 AdaptiveController trait + migrate first controller (atoms) +## Task 8: C.6 AdaptiveMonitor trait + test harness (GPU-drives architecture) + +**DESIGN NOTE:** Task 8 replaces the earlier `AdaptiveController` trait with a read-only `AdaptiveMonitor` trait. See spec §4.C.6 revision (2026-04-24). The reason: adaptive values are computed by GPU kernels from ISV signals; CPU-side code is pure observation. The old `update()` / `write_output()` trait methods are removed — CPU never computes an adaptive value. **Files:** -- Create: `crates/ml/src/trainers/dqn/adaptive_controller.rs` -- Modify: `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` — use the trait for atom-support controller +- Create: `crates/ml/src/trainers/dqn/adaptive_monitor.rs` (renamed from `adaptive_controller.rs`) -- [ ] **Step 8.1: Write failing test** +### Step 8.1: Write the failing test -Create `crates/ml/src/trainers/dqn/adaptive_controller.rs`: +Create `crates/ml/src/trainers/dqn/adaptive_monitor.rs` with ONLY the test module first: ```rust -//! Unified protocol for adaptive controllers in the DQN trainer. -//! Spec §4.C.6. +//! Read-only observer for GPU-computed adaptive values in the DQN trainer. +//! Spec §4.C.6 (2026-04-24 revision: GPU computes, CPU reads). #[cfg(test)] mod tests { use super::*; #[test] - fn controller_fire_rate_stats_compute() { + fn fire_rate_stats_compute() { let mut stats = FireRateStats::default(); stats.record_fire(true); stats.record_fire(false); stats.record_fire(true); - assert_eq!(stats.fire_count, 2); - assert_eq!(stats.observation_count, 3); - assert!((stats.fire_rate() - 2.0 / 3.0).abs() < 1e-9); + assert!((stats.fire_rate() - 2.0f32 / 3.0f32).abs() < 1e-6); } #[test] - fn diag_snapshot_empty_by_default() { - let snap = DiagSnapshot::default(); - assert!(snap.fields.is_empty()); + fn diag_snapshot_builder() { + let snap = DiagSnapshot::new().with("v", 0.5).with("t", 1.0); + assert_eq!(snap.fields.len(), 2); + assert_eq!(snap.fields.get("v"), Some(&0.5f32)); } } ``` -- [ ] **Step 8.2: Run test to verify it fails** +### Step 8.2: Run — expect FAIL -Run: ```bash -SQLX_OFFLINE=true CARGO_INCREMENTAL=0 cargo test -p ml adaptive_controller 2>&1 | tail -10 +SQLX_OFFLINE=true CARGO_INCREMENTAL=0 cargo test -p ml adaptive_monitor 2>&1 | tail -10 ``` Expected: compile errors `cannot find type FireRateStats`. -- [ ] **Step 8.3: Implement the trait** +### Step 8.3: Implement the trait -Prepend to `adaptive_controller.rs` (above the test module): +Prepend to `adaptive_monitor.rs` above the test module: ```rust use std::collections::HashMap; -/// Per-controller fire-rate statistics. A "fire" is any observation where -/// the controller emitted a control value different from its previous emission. -/// Used by the `controller_activity` smoke test to verify load-bearing -/// controllers don't fire in > 50% of epochs. -#[derive(Debug, Clone, Default)] -pub struct FireRateStats { - pub fire_count: u64, - pub observation_count: u64, +/// Thin read/write view over the ISV pinned-memory bus. +/// +/// Monitors use `read()` exclusively. The constructor / schedule-input path +/// uses `write()` for CPU-born inputs (epoch_idx, static configs). GPU kernels +/// write via their device pointer — not via `IsvBus`. +pub(crate) struct IsvBus<'a> { + signals: &'a mut [f32], +} + +impl<'a> IsvBus<'a> { + pub(crate) fn new(signals: &'a mut [f32]) -> Self { + Self { signals } + } + pub(crate) fn read(&self, slot: usize) -> f32 { self.signals[slot] } + pub(crate) fn write(&mut self, slot: usize, value: f32) { self.signals[slot] = value; } + pub(crate) fn len(&self) -> usize { self.signals.len() } +} + +/// Fire-rate statistics tracked by a monitor across observations. +/// +/// A "fire" is a monitor observation where the value materially changed from +/// the previous observation. Used by `controller_activity` smoke to verify +/// load-bearing mechanisms do not fire in > 50% of epochs. +#[derive(Debug, Default, Clone)] +pub(crate) struct FireRateStats { + fires: u32, + epochs: u32, } impl FireRateStats { - pub fn record_fire(&mut self, fired: bool) { - self.observation_count += 1; - if fired { - self.fire_count += 1; - } + pub(crate) fn record_fire(&mut self, fired: bool) { + self.epochs += 1; + if fired { self.fires += 1; } } - - pub fn fire_rate(&self) -> f64 { - if self.observation_count == 0 { - 0.0 - } else { - self.fire_count as f64 / self.observation_count as f64 - } + pub(crate) fn fire_rate(&self) -> f32 { + if self.epochs == 0 { 0.0 } else { self.fires as f32 / self.epochs as f32 } } - - pub fn reset(&mut self) { - *self = Self::default(); + pub(crate) fn reset(&mut self) { + self.fires = 0; + self.epochs = 0; } } -/// Standardised diagnostic emission from an adaptive controller. -/// Emitted at epoch boundary into HEALTH_DIAG. -#[derive(Debug, Clone, Default)] -pub struct DiagSnapshot { - /// Named fields → value. Example: `{"gamma": 0.95, "fire_rate": 0.12}`. - pub fields: HashMap, +/// Diagnostic snapshot emitted by a monitor each epoch. +/// +/// Uses `&'static str` keys (no allocation per key). Built via `.with()`. +#[derive(Debug, Default, Clone)] +pub(crate) struct DiagSnapshot { + pub fields: HashMap<&'static str, f32>, } impl DiagSnapshot { - pub fn with(mut self, name: impl Into, value: f64) -> Self { - self.fields.insert(name.into(), value); + pub(crate) fn new() -> Self { Self { fields: HashMap::new() } } + pub(crate) fn with(mut self, key: &'static str, value: f32) -> Self { + self.fields.insert(key, value); self } } -/// Unified protocol for adaptive controllers in the DQN trainer. +/// Read-only observer for a GPU-computed adaptive value. /// -/// Each controller reads signals from the ISV bus, computes a control value -/// (gamma, atom positions, Kelly floor, CQL alpha, etc.), writes its output -/// back to ISV, and emits diagnostics. -pub trait AdaptiveController { - /// Opaque signal vector the controller reads. - type Signal; - /// Opaque control value the controller emits. - type Control; +/// All adaptive values are computed by GPU kernels from ISV signals. CPU-side +/// code does NOT compute adaptive values; it only reads and reports. See +/// spec §4.C.6 (2026-04-24 revision). +pub(crate) trait AdaptiveMonitor { + /// Read the GPU-computed current value from ISV. + fn read(&self, isv: &IsvBus<'_>) -> f32; - /// Read inputs from the ISV bus. - fn read_signals(&self, isv: &IsvBus) -> Self::Signal; + /// Emit diagnostic snapshot for HEALTH_DIAG. + fn diagnose(&self, isv: &IsvBus<'_>) -> DiagSnapshot; - /// Compute the new control value. May update internal state. - fn update(&mut self, signals: Self::Signal) -> Self::Control; + /// Record one observation; drives fire-rate tracking at epoch boundary. + fn observe(&mut self, current: f32); - /// Write the control value back to ISV / consumer buffers. - fn write_output(&self, ctrl: &Self::Control, isv: &mut IsvBus); - - /// Per-controller fire-rate statistics. + /// Fire-rate statistics across prior observations. fn fire_rate(&self) -> &FireRateStats; - /// Emit a diagnostic snapshot at epoch boundary. - fn diagnose(&self) -> DiagSnapshot; - - /// Controller name, used in HEALTH_DIAG field labels. + /// Short name used in diagnostic output (e.g. "gamma", "kelly_cap"). fn name(&self) -> &'static str; } - -/// Abstraction over the ISV bus for controllers. -/// In production, this wraps `isv_signals_pinned: *mut f32`. -/// In tests, a plain `Vec` is used for isolation. -pub struct IsvBus<'a> { - pub(crate) slots: &'a mut [f32], -} - -impl<'a> IsvBus<'a> { - pub fn new(slots: &'a mut [f32]) -> Self { - Self { slots } - } - pub fn read(&self, idx: usize) -> f32 { self.slots[idx] } - pub fn write(&mut self, idx: usize, v: f32) { self.slots[idx] = v; } -} ``` -- [ ] **Step 8.4: Run test to verify it passes** +### Step 8.4: Run test — expect PASS ```bash -SQLX_OFFLINE=true CARGO_INCREMENTAL=0 cargo test -p ml adaptive_controller 2>&1 | tail -5 +SQLX_OFFLINE=true CARGO_INCREMENTAL=0 cargo test -p ml adaptive_monitor 2>&1 | tail -5 ``` -Expected: `test result: ok. 2 passed; 0 failed`. +Expected: 2 passed. -- [ ] **Step 8.5: Commit the trait and harness** +### Step 8.5: Export from mod.rs + +In `crates/ml/src/trainers/dqn/mod.rs`: +```rust +pub(crate) mod adaptive_monitor; +``` + +### Step 8.6: Compile-check ```bash -git add crates/ml/src/trainers/dqn/adaptive_controller.rs \ +SQLX_OFFLINE=true CARGO_INCREMENTAL=0 cargo check -p ml 2>&1 | tail -3 +``` + +Expected: `Finished` with the 8-warning baseline. + +### Step 8.7: Audit doc + +Append to `docs/dqn-wire-up-audit.md`: +```markdown +| `trainers/dqn/adaptive_monitor.rs` | Read-only observer trait + harness (FireRateStats, DiagSnapshot, IsvBus<'a>); consumers added in Plan 1 Tasks 9-17 (atoms/gamma/kelly_cap/tau/epsilon/grad_balancer monitors) | Wired (consumers added in same plan) | C.6 GPU-drives-CPU-reads | — | +``` + +### Step 8.8: Commit + +```bash +git add crates/ml/src/trainers/dqn/adaptive_monitor.rs \ crates/ml/src/trainers/dqn/mod.rs \ docs/dqn-wire-up-audit.md git commit -m "$(cat <<'EOF' -feat(dqn-v2): C.6 AdaptiveController trait + test harness +feat(dqn-v2): C.6 AdaptiveMonitor trait (GPU-drives, CPU-reads) -Unified protocol for every adaptive controller in the DQN trainer. -FireRateStats for controller_activity smoke. DiagSnapshot for -standardised HEALTH_DIAG emission. IsvBus abstraction over pinned -device-mapped memory. +Read-only observer trait for GPU-computed adaptive values. Replaces +the earlier AdaptiveController trait which had CPU-side update() and +write_output() — those violated the newly-codified principle that +GPU kernels compute all adaptive decisions and CPU-side code only +reads from ISV. -No concrete controller migration yet — Task 9 migrates atoms (first -in the sequence per spec §8 C.6 decision: atoms → gamma → Kelly → -cql_alpha → tau → epsilon → conviction_floor → plan_threshold → -balancer). +FireRateStats for controller_activity smoke. DiagSnapshot built via +.with() chain. IsvBus read/write (write for CPU-born inputs only — +epoch_idx, static configs; NEVER for adaptive values). Tests: 2 unit tests on FireRateStats and DiagSnapshot. -Plan 1 Task 8. Spec §4.C.6. +Plan 1 Task 8. Spec §4.C.6 (2026-04-24 revision). Co-Authored-By: Claude Opus 4.7 (1M context) EOF )" ``` -### Tasks 9 through 16: migrate each adaptive controller in spec-specified order - -For each controller, follow this pattern: - -1. **Write failing test** for the controller's `impl AdaptiveController`. -2. **Run test to fail** (trait impl doesn't exist yet). -3. **Implement the trait** for the controller; delete the old ad-hoc function. -4. **Wire consumers** to call through the trait (may be via a central dispatcher that holds all `Box`). -5. **Run test to pass.** -6. **Run smoke tests** (`controller_activity` is the key one — it asserts fire rates). -7. **Update docs/dqn-wire-up-audit.md and docs/isv-slots.md**. -8. **Commit** with spec-referenced message. - -Concrete controllers in spec order: -- Task 9: atoms controller (`recompute_atom_positions` + `warm_start_atom_positions`) -- Task 10: gamma controller (the `adaptive_gamma` in `training_loop.rs`) -- Task 11: Kelly cap controller (`kelly_position_cap`'s `conviction` + `safety_multiplier` computation) -- Task 12: cql_alpha controller (currently static; prepares for the seed-phase coupling in Plan 3) -- Task 13: tau controller (stochastic SARSA temperature in `c51_loss_kernel`) -- Task 14: epsilon controller (cosine-annealed with floor) -- Task 15: conviction-floor controller (IQL branch_scales floor, currently static at ISV[36]) -- Task 16: plan-threshold controller (currently hardcoded 0.5, prepares for B.4 in Plan 3) -- Task 17: grad-balancer controller (the `grad_balance_isv_update` kernel — wraps it under the trait) - -After the last migration (Task 17) is complete, the old ad-hoc update functions are all removed in Task 17's commit — per spec §4.C.6 "Old scaffolding is removed in the SAME commit that migrates its last consumer". - --- +## Tasks 9-17: Reactive GPU kernels + CPU monitors (new structure) + +Each of Tasks 9-11, 13-14, 17 lands a GPU kernel + CPU `AdaptiveMonitor` pair. Tasks 12, 15, 16 are static-config ISV constructor writes (no monitor needed). All old CPU-compute scaffolding is removed in the same commit as the replacement lands (`feedback_no_partial_refactor.md`). + +### Common pattern for GPU-kernel + monitor pairs + +1. **Write failing CPU-monitor unit test** — monitor reads a fixture ISV slice, asserts value + diagnose fields. +2. **Write failing GPU-kernel smoke test** — launch kernel with known ISV inputs, read back, assert expected output. +3. **Implement kernel** in `crates/ml/src/cuda_pipeline/_update_kernel.cu`. Reads ISV slots, writes to designated output slot. +4. **Wire kernel launch** in `fused_training.rs::process_epoch_boundary` (cold path, per-epoch). +5. **Implement monitor** in `crates/ml/src/trainers/dqn/monitors/_monitor.rs`. Read-only. +6. **Delete old CPU-compute code** (the pre-revert leftover, if any). +7. **Run tests** — expect PASS. +8. **Audit doc + isv-slots.md** — add rows for new slots + new monitor. +9. **Commit.** + +### Task 9: atoms kernel + monitor + +- New kernel: `cuda_pipeline/atoms_update_kernel.cu`. Reads ISV v-range slots (`V_CENTER_*`, `V_HALF_*`); writes atom positions to device buffer (existing) + optional summary slot. +- New monitor: `trainers/dqn/monitors/atoms_monitor.rs`. Reads atom-summary ISV slots; emits diagnostic. +- Old CPU code deleted: `fused.recompute_atom_positions()` CPU path and `warm_start_atom_positions()` CPU path. + +### Task 10: gamma kernel + monitor + +- New ISV slot: `GAMMA_EFF_INDEX` (GPU-written). +- New kernel: `cuda_pipeline/gamma_update_kernel.cu`. Reads ISV Q-stats util signals; writes to `GAMMA_EFF_INDEX`. +- New monitor: `trainers/dqn/monitors/gamma_monitor.rs`. +- Old CPU code deleted: the `adaptive_gamma` update block in `training_loop.rs`. + +### Task 11: kelly_cap kernel + monitor + +- New ISV slot: `KELLY_CAP_EFF_INDEX`. +- New kernel: `cuda_pipeline/kelly_cap_update_kernel.cu`. Reads Kelly stats from ISV (via the already-ISV-exposed Kelly slot summary); writes to `KELLY_CAP_EFF_INDEX`. +- New monitor: `trainers/dqn/monitors/kelly_cap_monitor.rs`. +- Old CPU code deleted: Kelly f_mean / avg_win_ratio computation in `training_loop.rs`. + +### Task 12: cql_alpha static ISV write (NO monitor, NO kernel) + +- New ISV slot: `CQL_ALPHA_INDEX`. +- Constructor writes `hyperparams.cql_alpha` to `CQL_ALPHA_INDEX` once. +- CQL kernel launch reads from ISV slot instead of a config field. +- No monitor — value is static, identity read. +- Plan 3 B.3 will rewrite this into a kernel that updates `CQL_ALPHA_INDEX` from replay-seed-phase signals; at that point Task 12's "no kernel" becomes "GPU kernel". This plan stays static. + +### Task 13: tau (Polyak EMA) kernel + monitor + +- New ISV slots: `TAU_EFF_INDEX` (GPU-written), `EPOCH_IDX_INDEX` (CPU-written at epoch boundary), `TOTAL_EPOCHS_INDEX` (CPU-written at constructor). +- New kernel: `cuda_pipeline/tau_update_kernel.cu`. Reads `EPOCH_IDX_INDEX`, `TOTAL_EPOCHS_INDEX`, `LEARNING_HEALTH_INDEX`; computes cosine schedule + health floor; writes to `TAU_EFF_INDEX`. +- New monitor: `trainers/dqn/monitors/tau_monitor.rs`. +- `iqn.set_tau_host()` rewritten to read from `TAU_EFF_INDEX` (via pinned pointer — zero-copy). +- Old CPU code deleted: the cosine-schedule + `apply_health_coupled_tau_floor()` computation in `fused_training.rs`. + +### Task 14: epsilon kernel + monitor + +- New ISV slot: `EPSILON_EFF_INDEX` (GPU-written). Reuses `EPOCH_IDX_INDEX` / `TOTAL_EPOCHS_INDEX` from Task 13. +- New kernel: `cuda_pipeline/epsilon_update_kernel.cu`. Reads `EPOCH_IDX_INDEX`, `TOTAL_EPOCHS_INDEX`, and any volatility-reactive signal; writes to `EPSILON_EFF_INDEX`. +- New monitor: `trainers/dqn/monitors/epsilon_monitor.rs`. +- Old CPU code deleted: `base_floor * (0.5 + volatility)` block in `initialize_epoch_state`. + +### Task 15: conviction_floor static ISV write (NO monitor, NO kernel) + +- New ISV slot: `CONVICTION_FLOOR_INDEX`. Constructor writes the configured floor (typically 0.1). IQL kernel reads from `CONVICTION_FLOOR_INDEX`. +- No monitor. + +### Task 16: plan_threshold static ISV write (NO monitor, NO kernel) + +- New ISV slot: `PLAN_THRESHOLD_INDEX`. Constructor writes `0.5`. `backtest_plan_kernel.cu` and `experience_kernels.cu` read from `PLAN_THRESHOLD_INDEX`. +- No monitor. Plan 3 B.4 rewrites these kernels to read from `PLAN_PARAMS_0_EMA_INDEX` — transforming this slot to dynamic. + +### Task 17: grad_balancer monitor only (kernel already exists) + +- Kernel `grad_balance_isv_update` already GPU-driven — no kernel work. +- New monitor: `trainers/dqn/monitors/grad_balancer_monitor.rs`. Reads `GRAD_NORM_TARGET_*` and `GRAD_SCALE_LIMIT_INDEX`. + +### Per-task-batch smoke validation + +After each task's commit: + +```bash +SQLX_OFFLINE=true CARGO_INCREMENTAL=0 cargo check -p ml 2>&1 | tail -3 +SQLX_OFFLINE=true CARGO_INCREMENTAL=0 cargo test -p ml --lib -- monitors:: adaptive_monitor 2>&1 | tail -15 +``` + +### End-of-block scaffolding removal + +Task 17's commit also removes any shared CPU-side controller-dispatcher scaffolding, per §4.C.6's "Old scaffolding is removed in the SAME commit that migrates its last consumer". If the pre-revert `controllers/` module directory contains stale files, they are removed here. The final layout is: + +- `crates/ml/src/trainers/dqn/adaptive_monitor.rs` — trait + harness. +- `crates/ml/src/trainers/dqn/monitors/` — 6 monitor impls (atoms, gamma, kelly_cap, tau, epsilon, grad_balancer). +- `crates/ml/src/cuda_pipeline/` — 5 new `*_update_kernel.cu` files (atoms, gamma, kelly_cap, tau, epsilon). `grad_balance_isv_update` kernel stays. +- ISV slots for 3 static configs (cql_alpha, conviction_floor, plan_threshold) written at constructor. + + ## Task 18: Plan 1 validation run **Files:** @@ -1739,7 +1817,7 @@ Plan 1 is "done" when: 2. `docs/dqn-wire-up-audit.md`, `docs/isv-slots.md`, `docs/dqn-gpu-hot-path-audit.md`, `docs/dqn-named-dims.md`, `docs/ml-supervised-to-dqn-concept-audit.md`, `config/metric-bands.toml` all exist and are populated. 3. Every raw `ps[N]`, `plan_isv[N]`, `plan_params[N]` access has been replaced with a named constant. 4. `StateResetRegistry` is the sole fold-reset dispatch path. -5. `AdaptiveController` trait has 9 concrete impls covering every previous ad-hoc controller. +5. `AdaptiveMonitor` trait has 6 concrete impls (atoms, gamma, kelly_cap, tau, epsilon, grad_balancer — all read-only observers of GPU-computed values). 3 static configs (cql_alpha, conviction_floor, plan_threshold) written to ISV at constructor. All adaptive computation happens in GPU kernels; CPU-side code never computes adaptive values (spec §4.C.6, 2026-04-24 revision). 6. ISV layout fingerprint at ISV[0] is populated at construct (via compile-time structural hash) and fail-fast-checked at checkpoint load. No `migrate_*` functions exist. No integer version sequence. 7. Hot-path purity audit has zero `MIGRATE` entries remaining. 8. All smoke tests pass. diff --git a/docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md b/docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md index 9b48e00c3..5521facea 100644 --- a/docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md +++ b/docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md @@ -373,38 +373,77 @@ warmup_steps = tracked via ISV, proportional to the seed-phase decay in B.3 While seed trajectories dominate the buffer, `cql_alpha` is near zero (no pessimism). As seed trajectories displace, `cql_alpha` ramps to its final value. -#### C.6 Adaptive Controller Unification (major refactor) +#### C.6 Adaptive Mechanism Unification (major refactor) -Current: the DQN has many adaptive controllers (atoms, gamma, grad-balancer, Kelly cap, tau, cql_alpha, epsilon, conviction-floor, plan-threshold, and new ones added by this spec). Each was bolted on independently with its own update rule and ISV interaction. +Current: the DQN has many adaptive mechanisms (atoms, gamma, grad-balancer, Kelly cap, tau, cql_alpha, epsilon, conviction-floor, plan-threshold, and new ones added by this spec). Each was bolted on independently with its own update rule and ISV interaction. Most were CPU-computed and pushed to GPU consumers. -Design: unify under `trait AdaptiveController`: +**Design — GPU computes, CPU reads (non-negotiable):** + +All adaptive values and all schedules are computed by GPU kernels from ISV-visible inputs and written to ISV slots. CPU-side code has two roles only: + +1. **CPU-born inputs to ISV** — values that fundamentally originate on CPU (epoch_idx, total_epochs, static config constants) are written to dedicated ISV slots once at the appropriate cadence (epoch boundary for epoch_idx, constructor for static configs). This is a pinned-memory zero-copy write. +2. **Read-only observation** — CPU reads ISV slots to emit HEALTH_DIAG diagnostics and track fire-rate statistics for the `controller_activity` smoke. + +CPU never computes an adaptive value or schedule output. If the value is f(ISV_signals), a GPU kernel computes it. + +**Rationale:** + +- Unified adaptive machinery: every adaptive mechanism follows the same on-GPU pattern. No CPU-orchestrated special cases. +- Per-sample granularity available — adaptive mechanisms can be per-sample if useful (e.g., Expected SARSA τ already does this inline in c51_loss_kernel), whereas CPU controllers are bounded to per-epoch. +- Zero CPU→GPU config transfer in the hot path. CPU writes to ISV via the pinned pointer (no copy); GPU reads from ISV directly. +- ISV is the single source of truth for every adaptive value. CPU-side code is pure observability, enforcing Invariant 3's spirit beyond the hot path into epoch-boundary code as well. +- No conflation of "schedule" (epoch-deterministic) and "adaptive" (reactive to observed signals): both compute on GPU. The input signals differ; the compute location does not. + +**The observer trait:** ```rust -trait AdaptiveController { - type Signal; - type Control; - - fn read_signals(&self, isv: &IsvBus) -> Self::Signal; - fn update(&mut self, signals: Self::Signal) -> Self::Control; - fn fire_rate(&self) -> FireRateStats; - fn diagnose(&self) -> DiagSnapshot; +pub(crate) trait AdaptiveMonitor { + /// Read the GPU-computed current value from ISV. + fn read(&self, isv: &IsvBus<'_>) -> f32; + /// Emit diagnostic snapshot for HEALTH_DIAG. + fn diagnose(&self, isv: &IsvBus<'_>) -> DiagSnapshot; + /// Record one observation; drives fire-rate tracking across epoch boundaries. + fn observe(&mut self, current: f32); + fn fire_rate(&self) -> &FireRateStats; + fn name(&self) -> &'static str; } ``` -Each existing controller becomes an `impl AdaptiveController`. Shared: +No `update()`, no `write_output()`, no `Signal`/`Control` associated types. Pure observation. -- ISV-read via typed slot handles (A.2). -- ISV-write of the control value via typed slot handles. -- Fire-rate tracking instrumented by `controller_activity` smoke test. -- HEALTH_DIAG emission via `diagnose()` returning a standardised snapshot. +**Mechanism → GPU kernel mapping:** -Benefits: +| Mechanism | Nature | Kernel behaviour | CPU monitor? | +|---|---|---|---| +| atoms | Reactive to Q-distribution | reads ISV v-range; writes atom positions to device buffer + summary to ISV | yes | +| gamma | Reactive to Q-stats util | reads ISV Q-stats util; writes γ to ISV | yes | +| kelly_cap | Reactive to trade P&L distribution | reads ISV Kelly stats; writes cap to ISV | yes | +| tau (Polyak EMA) | Schedule + health-reactive | reads ISV epoch_idx + total_epochs + health; writes τ_eff to ISV | yes | +| epsilon | Schedule + volatility-reactive | reads ISV epoch_idx + total_epochs + volatility; writes ε to ISV | yes | +| grad_balancer | Reactive to per-branch grad norms | existing `grad_balance_isv_update` kernel (already GPU-driven) | yes (new) | +| cql_alpha | Static config | CPU writes to ISV at constructor; no kernel | no | +| conviction_floor | Static config | CPU writes to ISV at constructor; no kernel | no | +| plan_threshold | Static config | CPU writes to ISV at constructor; no kernel (Plan 3 §B.4 makes reactive) | no | -- Adding a new adaptive mechanism follows a protocol, not an ad-hoc accretion. -- Fire-rate auditing is built-in (addresses `controller_activity` smoke's assertion that load-bearing controllers shouldn't fire in >50% of epochs). -- Standardised diagnostics — the HEALTH_DIAG `controllers [anti_lr=true gamma=false clip=true …]` line becomes derived state, not manually-coded. +Expected SARSA τ in `c51_loss_kernel` is already per-sample GPU-computed from ISV — no change needed; diagnostic surface already wired via `last_sarsa_tau_factor()`. Exemplar of the pattern. -Biggest-scope piece in the spec. Executed as a series of internal commits — one per existing controller migrated to the protocol. Old scaffolding (the ad-hoc update functions being replaced) is removed in the SAME commit that migrates its last consumer to the new protocol, not in a deferred cleanup pass. Per `feedback_no_partial_refactor.md` — when a contract changes, every consumer migrates together. +**New ISV slots for computed outputs + CPU-born inputs:** + +Allocated in dedicated ranges (§6 appendix). The fingerprint seed updates automatically as slots are added; fingerprint position shifts to the new tail. + +- `EPOCH_IDX_INDEX`, `TOTAL_EPOCHS_INDEX` — CPU-written at epoch boundary and constructor respectively. +- `EPSILON_EFF_INDEX`, `TAU_EFF_INDEX`, `GAMMA_EFF_INDEX`, `KELLY_CAP_EFF_INDEX` — GPU-written by the respective kernels. +- `CQL_ALPHA_INDEX`, `CONVICTION_FLOOR_INDEX`, `PLAN_THRESHOLD_INDEX` — CPU-written at constructor for the three static configs. + +Rationale for CPU→ISV static writes (rather than Rust constants): consumers read via ISV for consistency with the adaptive mechanisms. A reader cannot tell whether a value is static or GPU-updated; both use the same slot-read pattern. This eliminates a class of "wrong lookup" bugs and gives static values a uniform diagnostic surface. + +**Benefits preserved from the original C.6 design:** + +- Adding a new adaptive mechanism follows a protocol — write a GPU kernel + a CPU monitor. +- Fire-rate auditing built in via `controller_activity` smoke. +- Standardised diagnostics — HEALTH_DIAG's `controllers [...]` line derived from monitor state. + +**Execution:** a series of internal commits — spec+plan revision first, then revert any prior CPU-compute implementation, then per-kernel implementation + CPU monitor pairs. Old CPU-side scaffolding is removed in the SAME commit that lands its GPU-kernel replacement, per `feedback_no_partial_refactor.md`. ### Part D — Temporal Architecture