spec+plan(dqn-v2): GPU drives, CPU reads (§4.C.6 revision)

User raised that the CPU-controller pattern (where CPU computes adaptive
values from ISV and writes back to ISV) violates the architectural
principle: GPU kernels should compute adaptive decisions; CPU-side code
is pure observation.

Replaces the AdaptiveController trait with read-only AdaptiveMonitor:
- No update() — CPU doesn't compute adaptive values.
- No write_output() — CPU doesn't write adaptive values to ISV.
- read() returns the GPU-computed value from ISV.
- diagnose() emits HEALTH_DIAG snapshot.
- observe() + fire_rate() track how often the GPU-computed value changes.

Reclassifies the 9 adaptive mechanisms:
- 6 reactive get GPU kernel + CPU monitor: atoms, gamma, kelly_cap, tau
  (Polyak EMA), epsilon, grad_balancer (last is already GPU-driven).
- 3 static get ISV constructor-write, no monitor: cql_alpha,
  conviction_floor, plan_threshold.

New ISV slots for GPU-written adaptive outputs (EPSILON_EFF, TAU_EFF,
GAMMA_EFF, KELLY_CAP_EFF) and CPU-born inputs (EPOCH_IDX, TOTAL_EPOCHS),
plus slots for the 3 static configs.

Rationale:
- Unified adaptive machinery, no CPU-side special cases.
- Per-sample granularity available (Expected SARSA τ in c51_loss_kernel
  is already the exemplar — reads ISV q_gap + health per sample).
- Zero CPU→GPU config transfer in any path.
- ISV is single source of truth for every adaptive value.

Plan 1 Tasks 8-17 restructured:
- Task 8 creates AdaptiveMonitor trait (not AdaptiveController).
- Tasks 9, 10, 11, 13, 14, 17: GPU kernel + monitor pairs.
- Tasks 12, 15, 16: static ISV writes only.

Follow-up commits will revert Batch A (d76849f31) and Batch B (4189da563)
which implemented the old CPU-compute pattern, then re-implement under
this design.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-04-24 16:20:24 +02:00
parent 4189da563f
commit cf36091eeb
2 changed files with 268 additions and 151 deletions

View File

@@ -2,9 +2,9 @@
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Lay the substrate that all subsequent DQN v2 work depends on: state reset registry, ISV slot contract, audit docs, named-dimension constants, and the AdaptiveController unification refactor. Zero behavioural change to the trained policy — this plan only establishes scaffolding and refactors existing ad-hoc machinery into typed protocols.
**Goal:** Lay the substrate that all subsequent DQN v2 work depends on: state reset registry, ISV slot contract, audit docs, named-dimension constants, and the AdaptiveMonitor unification refactor (GPU kernels compute; CPU observes, per spec §4.C.6 revision). Zero behavioural change to the trained policy — this plan only establishes scaffolding and refactors existing ad-hoc machinery into typed protocols.
**Architecture:** Three audit docs + one config file tracked by pre-commit hook. One `StateResetRegistry` struct categorising every piece of training state. One typed `IsvSlot<T>` contract with a compile-time **layout fingerprint** (structural hash of the slot list) at ISV[0] — fail-fast only, no migration path ever exists. One `AdaptiveController` trait with existing controllers migrated one-at-a-time. Named constants for every semantic index across portfolio state, ISV slots, plan_isv dimensions, plan_params, state-vector offsets, branch indices, and action sub-indices.
**Architecture:** Three audit docs + one config file tracked by pre-commit hook. One `StateResetRegistry` struct categorising every piece of training state. One typed `IsvSlot<T>` contract with a compile-time **layout fingerprint** (structural hash of the slot list) at ISV[0..2) — fail-fast only, no migration path ever exists. One read-only `AdaptiveMonitor` trait with 6 GPU-kernel-plus-CPU-monitor pairs (atoms, gamma, kelly_cap, tau, epsilon, grad_balancer) and 3 static-config ISV writes (cql_alpha, conviction_floor, plan_threshold). **GPU drives, CPU reads** — CPU-side code never computes an adaptive value; all computation is in GPU kernels reading from and writing to ISV. Named constants for every semantic index across portfolio state, ISV slots, plan_isv dimensions, plan_params, state-vector offsets, branch indices, and action sub-indices.
**Tech Stack:** Rust (workspace crates: `ml`, `ml-core`, `ml-dqn`, `ml-supervised`), CUDA C++ via nvcc through `build.rs`, cudarc 0.19, Cargo 2021 edition, SQLx offline mode, sccache with `CARGO_INCREMENTAL=0`.
@@ -1470,215 +1470,293 @@ EOF
---
## Task 8: C.6 AdaptiveController trait + migrate first controller (atoms)
## Task 8: C.6 AdaptiveMonitor trait + test harness (GPU-drives architecture)
**DESIGN NOTE:** Task 8 replaces the earlier `AdaptiveController` trait with a read-only `AdaptiveMonitor` trait. See spec §4.C.6 revision (2026-04-24). The reason: adaptive values are computed by GPU kernels from ISV signals; CPU-side code is pure observation. The old `update()` / `write_output()` trait methods are removed — CPU never computes an adaptive value.
**Files:**
- Create: `crates/ml/src/trainers/dqn/adaptive_controller.rs`
- Modify: `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` — use the trait for atom-support controller
- Create: `crates/ml/src/trainers/dqn/adaptive_monitor.rs` (renamed from `adaptive_controller.rs`)
- [ ] **Step 8.1: Write failing test**
### Step 8.1: Write the failing test
Create `crates/ml/src/trainers/dqn/adaptive_controller.rs`:
Create `crates/ml/src/trainers/dqn/adaptive_monitor.rs` with ONLY the test module first:
```rust
//! Unified protocol for adaptive controllers in the DQN trainer.
//! Spec §4.C.6.
//! Read-only observer for GPU-computed adaptive values in the DQN trainer.
//! Spec §4.C.6 (2026-04-24 revision: GPU computes, CPU reads).
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn controller_fire_rate_stats_compute() {
fn fire_rate_stats_compute() {
let mut stats = FireRateStats::default();
stats.record_fire(true);
stats.record_fire(false);
stats.record_fire(true);
assert_eq!(stats.fire_count, 2);
assert_eq!(stats.observation_count, 3);
assert!((stats.fire_rate() - 2.0 / 3.0).abs() < 1e-9);
assert!((stats.fire_rate() - 2.0f32 / 3.0f32).abs() < 1e-6);
}
#[test]
fn diag_snapshot_empty_by_default() {
let snap = DiagSnapshot::default();
assert!(snap.fields.is_empty());
fn diag_snapshot_builder() {
let snap = DiagSnapshot::new().with("v", 0.5).with("t", 1.0);
assert_eq!(snap.fields.len(), 2);
assert_eq!(snap.fields.get("v"), Some(&0.5f32));
}
}
```
- [ ] **Step 8.2: Run test to verify it fails**
### Step 8.2: Run — expect FAIL
Run:
```bash
SQLX_OFFLINE=true CARGO_INCREMENTAL=0 cargo test -p ml adaptive_controller 2>&1 | tail -10
SQLX_OFFLINE=true CARGO_INCREMENTAL=0 cargo test -p ml adaptive_monitor 2>&1 | tail -10
```
Expected: compile errors `cannot find type FireRateStats`.
- [ ] **Step 8.3: Implement the trait**
### Step 8.3: Implement the trait
Prepend to `adaptive_controller.rs` (above the test module):
Prepend to `adaptive_monitor.rs` above the test module:
```rust
use std::collections::HashMap;
/// Per-controller fire-rate statistics. A "fire" is any observation where
/// the controller emitted a control value different from its previous emission.
/// Used by the `controller_activity` smoke test to verify load-bearing
/// controllers don't fire in > 50% of epochs.
#[derive(Debug, Clone, Default)]
pub struct FireRateStats {
pub fire_count: u64,
pub observation_count: u64,
/// Thin read/write view over the ISV pinned-memory bus.
///
/// Monitors use `read()` exclusively. The constructor / schedule-input path
/// uses `write()` for CPU-born inputs (epoch_idx, static configs). GPU kernels
/// write via their device pointer — not via `IsvBus`.
pub(crate) struct IsvBus<'a> {
signals: &'a mut [f32],
}
impl<'a> IsvBus<'a> {
pub(crate) fn new(signals: &'a mut [f32]) -> Self {
Self { signals }
}
pub(crate) fn read(&self, slot: usize) -> f32 { self.signals[slot] }
pub(crate) fn write(&mut self, slot: usize, value: f32) { self.signals[slot] = value; }
pub(crate) fn len(&self) -> usize { self.signals.len() }
}
/// Fire-rate statistics tracked by a monitor across observations.
///
/// A "fire" is a monitor observation where the value materially changed from
/// the previous observation. Used by `controller_activity` smoke to verify
/// load-bearing mechanisms do not fire in > 50% of epochs.
#[derive(Debug, Default, Clone)]
pub(crate) struct FireRateStats {
fires: u32,
epochs: u32,
}
impl FireRateStats {
pub fn record_fire(&mut self, fired: bool) {
self.observation_count += 1;
if fired {
self.fire_count += 1;
}
pub(crate) fn record_fire(&mut self, fired: bool) {
self.epochs += 1;
if fired { self.fires += 1; }
}
pub fn fire_rate(&self) -> f64 {
if self.observation_count == 0 {
0.0
} else {
self.fire_count as f64 / self.observation_count as f64
}
pub(crate) fn fire_rate(&self) -> f32 {
if self.epochs == 0 { 0.0 } else { self.fires as f32 / self.epochs as f32 }
}
pub fn reset(&mut self) {
*self = Self::default();
pub(crate) fn reset(&mut self) {
self.fires = 0;
self.epochs = 0;
}
}
/// Standardised diagnostic emission from an adaptive controller.
/// Emitted at epoch boundary into HEALTH_DIAG.
#[derive(Debug, Clone, Default)]
pub struct DiagSnapshot {
/// Named fields → value. Example: `{"gamma": 0.95, "fire_rate": 0.12}`.
pub fields: HashMap<String, f64>,
/// Diagnostic snapshot emitted by a monitor each epoch.
///
/// Uses `&'static str` keys (no allocation per key). Built via `.with()`.
#[derive(Debug, Default, Clone)]
pub(crate) struct DiagSnapshot {
pub fields: HashMap<&'static str, f32>,
}
impl DiagSnapshot {
pub fn with(mut self, name: impl Into<String>, value: f64) -> Self {
self.fields.insert(name.into(), value);
pub(crate) fn new() -> Self { Self { fields: HashMap::new() } }
pub(crate) fn with(mut self, key: &'static str, value: f32) -> Self {
self.fields.insert(key, value);
self
}
}
/// Unified protocol for adaptive controllers in the DQN trainer.
/// Read-only observer for a GPU-computed adaptive value.
///
/// Each controller reads signals from the ISV bus, computes a control value
/// (gamma, atom positions, Kelly floor, CQL alpha, etc.), writes its output
/// back to ISV, and emits diagnostics.
pub trait AdaptiveController {
/// Opaque signal vector the controller reads.
type Signal;
/// Opaque control value the controller emits.
type Control;
/// All adaptive values are computed by GPU kernels from ISV signals. CPU-side
/// code does NOT compute adaptive values; it only reads and reports. See
/// spec §4.C.6 (2026-04-24 revision).
pub(crate) trait AdaptiveMonitor {
/// Read the GPU-computed current value from ISV.
fn read(&self, isv: &IsvBus<'_>) -> f32;
/// Read inputs from the ISV bus.
fn read_signals(&self, isv: &IsvBus) -> Self::Signal;
/// Emit diagnostic snapshot for HEALTH_DIAG.
fn diagnose(&self, isv: &IsvBus<'_>) -> DiagSnapshot;
/// Compute the new control value. May update internal state.
fn update(&mut self, signals: Self::Signal) -> Self::Control;
/// Record one observation; drives fire-rate tracking at epoch boundary.
fn observe(&mut self, current: f32);
/// Write the control value back to ISV / consumer buffers.
fn write_output(&self, ctrl: &Self::Control, isv: &mut IsvBus);
/// Per-controller fire-rate statistics.
/// Fire-rate statistics across prior observations.
fn fire_rate(&self) -> &FireRateStats;
/// Emit a diagnostic snapshot at epoch boundary.
fn diagnose(&self) -> DiagSnapshot;
/// Controller name, used in HEALTH_DIAG field labels.
/// Short name used in diagnostic output (e.g. "gamma", "kelly_cap").
fn name(&self) -> &'static str;
}
/// Abstraction over the ISV bus for controllers.
/// In production, this wraps `isv_signals_pinned: *mut f32`.
/// In tests, a plain `Vec<f32>` is used for isolation.
pub struct IsvBus<'a> {
pub(crate) slots: &'a mut [f32],
}
impl<'a> IsvBus<'a> {
pub fn new(slots: &'a mut [f32]) -> Self {
Self { slots }
}
pub fn read(&self, idx: usize) -> f32 { self.slots[idx] }
pub fn write(&mut self, idx: usize, v: f32) { self.slots[idx] = v; }
}
```
- [ ] **Step 8.4: Run test to verify it passes**
### Step 8.4: Run test — expect PASS
```bash
SQLX_OFFLINE=true CARGO_INCREMENTAL=0 cargo test -p ml adaptive_controller 2>&1 | tail -5
SQLX_OFFLINE=true CARGO_INCREMENTAL=0 cargo test -p ml adaptive_monitor 2>&1 | tail -5
```
Expected: `test result: ok. 2 passed; 0 failed`.
Expected: 2 passed.
- [ ] **Step 8.5: Commit the trait and harness**
### Step 8.5: Export from mod.rs
In `crates/ml/src/trainers/dqn/mod.rs`:
```rust
pub(crate) mod adaptive_monitor;
```
### Step 8.6: Compile-check
```bash
git add crates/ml/src/trainers/dqn/adaptive_controller.rs \
SQLX_OFFLINE=true CARGO_INCREMENTAL=0 cargo check -p ml 2>&1 | tail -3
```
Expected: `Finished` with the 8-warning baseline.
### Step 8.7: Audit doc
Append to `docs/dqn-wire-up-audit.md`:
```markdown
| `trainers/dqn/adaptive_monitor.rs` | Read-only observer trait + harness (FireRateStats, DiagSnapshot, IsvBus<'a>); consumers added in Plan 1 Tasks 9-17 (atoms/gamma/kelly_cap/tau/epsilon/grad_balancer monitors) | Wired (consumers added in same plan) | C.6 GPU-drives-CPU-reads | — |
```
### Step 8.8: Commit
```bash
git add crates/ml/src/trainers/dqn/adaptive_monitor.rs \
crates/ml/src/trainers/dqn/mod.rs \
docs/dqn-wire-up-audit.md
git commit -m "$(cat <<'EOF'
feat(dqn-v2): C.6 AdaptiveController trait + test harness
feat(dqn-v2): C.6 AdaptiveMonitor trait (GPU-drives, CPU-reads)
Unified protocol for every adaptive controller in the DQN trainer.
FireRateStats for controller_activity smoke. DiagSnapshot for
standardised HEALTH_DIAG emission. IsvBus abstraction over pinned
device-mapped memory.
Read-only observer trait for GPU-computed adaptive values. Replaces
the earlier AdaptiveController trait which had CPU-side update() and
write_output() — those violated the newly-codified principle that
GPU kernels compute all adaptive decisions and CPU-side code only
reads from ISV.
No concrete controller migration yet — Task 9 migrates atoms (first
in the sequence per spec §8 C.6 decision: atoms → gamma → Kelly
cql_alpha → tau → epsilon → conviction_floor → plan_threshold →
balancer).
FireRateStats for controller_activity smoke. DiagSnapshot built via
.with() chain. IsvBus read/write (write for CPU-born inputs only
epoch_idx, static configs; NEVER for adaptive values).
Tests: 2 unit tests on FireRateStats and DiagSnapshot.
Plan 1 Task 8. Spec §4.C.6.
Plan 1 Task 8. Spec §4.C.6 (2026-04-24 revision).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
)"
```
### Tasks 9 through 16: migrate each adaptive controller in spec-specified order
For each controller, follow this pattern:
1. **Write failing test** for the controller's `impl AdaptiveController`.
2. **Run test to fail** (trait impl doesn't exist yet).
3. **Implement the trait** for the controller; delete the old ad-hoc function.
4. **Wire consumers** to call through the trait (may be via a central dispatcher that holds all `Box<dyn AdaptiveController>`).
5. **Run test to pass.**
6. **Run smoke tests** (`controller_activity` is the key one — it asserts fire rates).
7. **Update docs/dqn-wire-up-audit.md and docs/isv-slots.md**.
8. **Commit** with spec-referenced message.
Concrete controllers in spec order:
- Task 9: atoms controller (`recompute_atom_positions` + `warm_start_atom_positions`)
- Task 10: gamma controller (the `adaptive_gamma` in `training_loop.rs`)
- Task 11: Kelly cap controller (`kelly_position_cap`'s `conviction` + `safety_multiplier` computation)
- Task 12: cql_alpha controller (currently static; prepares for the seed-phase coupling in Plan 3)
- Task 13: tau controller (stochastic SARSA temperature in `c51_loss_kernel`)
- Task 14: epsilon controller (cosine-annealed with floor)
- Task 15: conviction-floor controller (IQL branch_scales floor, currently static at ISV[36])
- Task 16: plan-threshold controller (currently hardcoded 0.5, prepares for B.4 in Plan 3)
- Task 17: grad-balancer controller (the `grad_balance_isv_update` kernel — wraps it under the trait)
After the last migration (Task 17) is complete, the old ad-hoc update functions are all removed in Task 17's commit — per spec §4.C.6 "Old scaffolding is removed in the SAME commit that migrates its last consumer".
---
## Tasks 9-17: Reactive GPU kernels + CPU monitors (new structure)
Each of Tasks 9-11, 13-14, 17 lands a GPU kernel + CPU `AdaptiveMonitor` pair. Tasks 12, 15, 16 are static-config ISV constructor writes (no monitor needed). All old CPU-compute scaffolding is removed in the same commit as the replacement lands (`feedback_no_partial_refactor.md`).
### Common pattern for GPU-kernel + monitor pairs
1. **Write failing CPU-monitor unit test** — monitor reads a fixture ISV slice, asserts value + diagnose fields.
2. **Write failing GPU-kernel smoke test** — launch kernel with known ISV inputs, read back, assert expected output.
3. **Implement kernel** in `crates/ml/src/cuda_pipeline/<name>_update_kernel.cu`. Reads ISV slots, writes to designated output slot.
4. **Wire kernel launch** in `fused_training.rs::process_epoch_boundary` (cold path, per-epoch).
5. **Implement monitor** in `crates/ml/src/trainers/dqn/monitors/<name>_monitor.rs`. Read-only.
6. **Delete old CPU-compute code** (the pre-revert leftover, if any).
7. **Run tests** — expect PASS.
8. **Audit doc + isv-slots.md** — add rows for new slots + new monitor.
9. **Commit.**
### Task 9: atoms kernel + monitor
- New kernel: `cuda_pipeline/atoms_update_kernel.cu`. Reads ISV v-range slots (`V_CENTER_*`, `V_HALF_*`); writes atom positions to device buffer (existing) + optional summary slot.
- New monitor: `trainers/dqn/monitors/atoms_monitor.rs`. Reads atom-summary ISV slots; emits diagnostic.
- Old CPU code deleted: `fused.recompute_atom_positions()` CPU path and `warm_start_atom_positions()` CPU path.
### Task 10: gamma kernel + monitor
- New ISV slot: `GAMMA_EFF_INDEX` (GPU-written).
- New kernel: `cuda_pipeline/gamma_update_kernel.cu`. Reads ISV Q-stats util signals; writes to `GAMMA_EFF_INDEX`.
- New monitor: `trainers/dqn/monitors/gamma_monitor.rs`.
- Old CPU code deleted: the `adaptive_gamma` update block in `training_loop.rs`.
### Task 11: kelly_cap kernel + monitor
- New ISV slot: `KELLY_CAP_EFF_INDEX`.
- New kernel: `cuda_pipeline/kelly_cap_update_kernel.cu`. Reads Kelly stats from ISV (via the already-ISV-exposed Kelly slot summary); writes to `KELLY_CAP_EFF_INDEX`.
- New monitor: `trainers/dqn/monitors/kelly_cap_monitor.rs`.
- Old CPU code deleted: Kelly f_mean / avg_win_ratio computation in `training_loop.rs`.
### Task 12: cql_alpha static ISV write (NO monitor, NO kernel)
- New ISV slot: `CQL_ALPHA_INDEX`.
- Constructor writes `hyperparams.cql_alpha` to `CQL_ALPHA_INDEX` once.
- CQL kernel launch reads from ISV slot instead of a config field.
- No monitor — value is static, identity read.
- Plan 3 B.3 will rewrite this into a kernel that updates `CQL_ALPHA_INDEX` from replay-seed-phase signals; at that point Task 12's "no kernel" becomes "GPU kernel". This plan stays static.
### Task 13: tau (Polyak EMA) kernel + monitor
- New ISV slots: `TAU_EFF_INDEX` (GPU-written), `EPOCH_IDX_INDEX` (CPU-written at epoch boundary), `TOTAL_EPOCHS_INDEX` (CPU-written at constructor).
- New kernel: `cuda_pipeline/tau_update_kernel.cu`. Reads `EPOCH_IDX_INDEX`, `TOTAL_EPOCHS_INDEX`, `LEARNING_HEALTH_INDEX`; computes cosine schedule + health floor; writes to `TAU_EFF_INDEX`.
- New monitor: `trainers/dqn/monitors/tau_monitor.rs`.
- `iqn.set_tau_host()` rewritten to read from `TAU_EFF_INDEX` (via pinned pointer — zero-copy).
- Old CPU code deleted: the cosine-schedule + `apply_health_coupled_tau_floor()` computation in `fused_training.rs`.
### Task 14: epsilon kernel + monitor
- New ISV slot: `EPSILON_EFF_INDEX` (GPU-written). Reuses `EPOCH_IDX_INDEX` / `TOTAL_EPOCHS_INDEX` from Task 13.
- New kernel: `cuda_pipeline/epsilon_update_kernel.cu`. Reads `EPOCH_IDX_INDEX`, `TOTAL_EPOCHS_INDEX`, and any volatility-reactive signal; writes to `EPSILON_EFF_INDEX`.
- New monitor: `trainers/dqn/monitors/epsilon_monitor.rs`.
- Old CPU code deleted: `base_floor * (0.5 + volatility)` block in `initialize_epoch_state`.
### Task 15: conviction_floor static ISV write (NO monitor, NO kernel)
- New ISV slot: `CONVICTION_FLOOR_INDEX`. Constructor writes the configured floor (typically 0.1). IQL kernel reads from `CONVICTION_FLOOR_INDEX`.
- No monitor.
### Task 16: plan_threshold static ISV write (NO monitor, NO kernel)
- New ISV slot: `PLAN_THRESHOLD_INDEX`. Constructor writes `0.5`. `backtest_plan_kernel.cu` and `experience_kernels.cu` read from `PLAN_THRESHOLD_INDEX`.
- No monitor. Plan 3 B.4 rewrites these kernels to read from `PLAN_PARAMS_0_EMA_INDEX` — transforming this slot to dynamic.
### Task 17: grad_balancer monitor only (kernel already exists)
- Kernel `grad_balance_isv_update` already GPU-driven — no kernel work.
- New monitor: `trainers/dqn/monitors/grad_balancer_monitor.rs`. Reads `GRAD_NORM_TARGET_*` and `GRAD_SCALE_LIMIT_INDEX`.
### Per-task-batch smoke validation
After each task's commit:
```bash
SQLX_OFFLINE=true CARGO_INCREMENTAL=0 cargo check -p ml 2>&1 | tail -3
SQLX_OFFLINE=true CARGO_INCREMENTAL=0 cargo test -p ml --lib -- monitors:: adaptive_monitor 2>&1 | tail -15
```
### End-of-block scaffolding removal
Task 17's commit also removes any shared CPU-side controller-dispatcher scaffolding, per §4.C.6's "Old scaffolding is removed in the SAME commit that migrates its last consumer". If the pre-revert `controllers/` module directory contains stale files, they are removed here. The final layout is:
- `crates/ml/src/trainers/dqn/adaptive_monitor.rs` — trait + harness.
- `crates/ml/src/trainers/dqn/monitors/` — 6 monitor impls (atoms, gamma, kelly_cap, tau, epsilon, grad_balancer).
- `crates/ml/src/cuda_pipeline/` — 5 new `*_update_kernel.cu` files (atoms, gamma, kelly_cap, tau, epsilon). `grad_balance_isv_update` kernel stays.
- ISV slots for 3 static configs (cql_alpha, conviction_floor, plan_threshold) written at constructor.
## Task 18: Plan 1 validation run
**Files:**
@@ -1739,7 +1817,7 @@ Plan 1 is "done" when:
2. `docs/dqn-wire-up-audit.md`, `docs/isv-slots.md`, `docs/dqn-gpu-hot-path-audit.md`, `docs/dqn-named-dims.md`, `docs/ml-supervised-to-dqn-concept-audit.md`, `config/metric-bands.toml` all exist and are populated.
3. Every raw `ps[N]`, `plan_isv[N]`, `plan_params[N]` access has been replaced with a named constant.
4. `StateResetRegistry` is the sole fold-reset dispatch path.
5. `AdaptiveController` trait has 9 concrete impls covering every previous ad-hoc controller.
5. `AdaptiveMonitor` trait has 6 concrete impls (atoms, gamma, kelly_cap, tau, epsilon, grad_balancer — all read-only observers of GPU-computed values). 3 static configs (cql_alpha, conviction_floor, plan_threshold) written to ISV at constructor. All adaptive computation happens in GPU kernels; CPU-side code never computes adaptive values (spec §4.C.6, 2026-04-24 revision).
6. ISV layout fingerprint at ISV[0] is populated at construct (via compile-time structural hash) and fail-fast-checked at checkpoint load. No `migrate_*` functions exist. No integer version sequence.
7. Hot-path purity audit has zero `MIGRATE` entries remaining.
8. All smoke tests pass.

View File

@@ -373,38 +373,77 @@ warmup_steps = tracked via ISV, proportional to the seed-phase decay in B.3
While seed trajectories dominate the buffer, `cql_alpha` is near zero (no pessimism). As seed trajectories displace, `cql_alpha` ramps to its final value.
#### C.6 Adaptive Controller Unification (major refactor)
#### C.6 Adaptive Mechanism Unification (major refactor)
Current: the DQN has many adaptive controllers (atoms, gamma, grad-balancer, Kelly cap, tau, cql_alpha, epsilon, conviction-floor, plan-threshold, and new ones added by this spec). Each was bolted on independently with its own update rule and ISV interaction.
Current: the DQN has many adaptive mechanisms (atoms, gamma, grad-balancer, Kelly cap, tau, cql_alpha, epsilon, conviction-floor, plan-threshold, and new ones added by this spec). Each was bolted on independently with its own update rule and ISV interaction. Most were CPU-computed and pushed to GPU consumers.
Design: unify under `trait AdaptiveController`:
**Design — GPU computes, CPU reads (non-negotiable):**
All adaptive values and all schedules are computed by GPU kernels from ISV-visible inputs and written to ISV slots. CPU-side code has two roles only:
1. **CPU-born inputs to ISV** — values that fundamentally originate on CPU (epoch_idx, total_epochs, static config constants) are written to dedicated ISV slots once at the appropriate cadence (epoch boundary for epoch_idx, constructor for static configs). This is a pinned-memory zero-copy write.
2. **Read-only observation** — CPU reads ISV slots to emit HEALTH_DIAG diagnostics and track fire-rate statistics for the `controller_activity` smoke.
CPU never computes an adaptive value or schedule output. If the value is f(ISV_signals), a GPU kernel computes it.
**Rationale:**
- Unified adaptive machinery: every adaptive mechanism follows the same on-GPU pattern. No CPU-orchestrated special cases.
- Per-sample granularity available — adaptive mechanisms can be per-sample if useful (e.g., Expected SARSA τ already does this inline in c51_loss_kernel), whereas CPU controllers are bounded to per-epoch.
- Zero CPU→GPU config transfer in the hot path. CPU writes to ISV via the pinned pointer (no copy); GPU reads from ISV directly.
- ISV is the single source of truth for every adaptive value. CPU-side code is pure observability, enforcing Invariant 3's spirit beyond the hot path into epoch-boundary code as well.
- No conflation of "schedule" (epoch-deterministic) and "adaptive" (reactive to observed signals): both compute on GPU. The input signals differ; the compute location does not.
**The observer trait:**
```rust
trait AdaptiveController {
type Signal;
type Control;
fn read_signals(&self, isv: &IsvBus) -> Self::Signal;
fn update(&mut self, signals: Self::Signal) -> Self::Control;
fn fire_rate(&self) -> FireRateStats;
fn diagnose(&self) -> DiagSnapshot;
pub(crate) trait AdaptiveMonitor {
/// Read the GPU-computed current value from ISV.
fn read(&self, isv: &IsvBus<'_>) -> f32;
/// Emit diagnostic snapshot for HEALTH_DIAG.
fn diagnose(&self, isv: &IsvBus<'_>) -> DiagSnapshot;
/// Record one observation; drives fire-rate tracking across epoch boundaries.
fn observe(&mut self, current: f32);
fn fire_rate(&self) -> &FireRateStats;
fn name(&self) -> &'static str;
}
```
Each existing controller becomes an `impl AdaptiveController`. Shared:
No `update()`, no `write_output()`, no `Signal`/`Control` associated types. Pure observation.
- ISV-read via typed slot handles (A.2).
- ISV-write of the control value via typed slot handles.
- Fire-rate tracking instrumented by `controller_activity` smoke test.
- HEALTH_DIAG emission via `diagnose()` returning a standardised snapshot.
**Mechanism → GPU kernel mapping:**
Benefits:
| Mechanism | Nature | Kernel behaviour | CPU monitor? |
|---|---|---|---|
| atoms | Reactive to Q-distribution | reads ISV v-range; writes atom positions to device buffer + summary to ISV | yes |
| gamma | Reactive to Q-stats util | reads ISV Q-stats util; writes γ to ISV | yes |
| kelly_cap | Reactive to trade P&L distribution | reads ISV Kelly stats; writes cap to ISV | yes |
| tau (Polyak EMA) | Schedule + health-reactive | reads ISV epoch_idx + total_epochs + health; writes τ_eff to ISV | yes |
| epsilon | Schedule + volatility-reactive | reads ISV epoch_idx + total_epochs + volatility; writes ε to ISV | yes |
| grad_balancer | Reactive to per-branch grad norms | existing `grad_balance_isv_update` kernel (already GPU-driven) | yes (new) |
| cql_alpha | Static config | CPU writes to ISV at constructor; no kernel | no |
| conviction_floor | Static config | CPU writes to ISV at constructor; no kernel | no |
| plan_threshold | Static config | CPU writes to ISV at constructor; no kernel (Plan 3 §B.4 makes reactive) | no |
- Adding a new adaptive mechanism follows a protocol, not an ad-hoc accretion.
- Fire-rate auditing is built-in (addresses `controller_activity` smoke's assertion that load-bearing controllers shouldn't fire in >50% of epochs).
- Standardised diagnostics — the HEALTH_DIAG `controllers [anti_lr=true gamma=false clip=true …]` line becomes derived state, not manually-coded.
Expected SARSA τ in `c51_loss_kernel` is already per-sample GPU-computed from ISV — no change needed; diagnostic surface already wired via `last_sarsa_tau_factor()`. Exemplar of the pattern.
Biggest-scope piece in the spec. Executed as a series of internal commits — one per existing controller migrated to the protocol. Old scaffolding (the ad-hoc update functions being replaced) is removed in the SAME commit that migrates its last consumer to the new protocol, not in a deferred cleanup pass. Per `feedback_no_partial_refactor.md` — when a contract changes, every consumer migrates together.
**New ISV slots for computed outputs + CPU-born inputs:**
Allocated in dedicated ranges (§6 appendix). The fingerprint seed updates automatically as slots are added; fingerprint position shifts to the new tail.
- `EPOCH_IDX_INDEX`, `TOTAL_EPOCHS_INDEX` — CPU-written at epoch boundary and constructor respectively.
- `EPSILON_EFF_INDEX`, `TAU_EFF_INDEX`, `GAMMA_EFF_INDEX`, `KELLY_CAP_EFF_INDEX` — GPU-written by the respective kernels.
- `CQL_ALPHA_INDEX`, `CONVICTION_FLOOR_INDEX`, `PLAN_THRESHOLD_INDEX` — CPU-written at constructor for the three static configs.
Rationale for CPU→ISV static writes (rather than Rust constants): consumers read via ISV for consistency with the adaptive mechanisms. A reader cannot tell whether a value is static or GPU-updated; both use the same slot-read pattern. This eliminates a class of "wrong lookup" bugs and gives static values a uniform diagnostic surface.
**Benefits preserved from the original C.6 design:**
- Adding a new adaptive mechanism follows a protocol — write a GPU kernel + a CPU monitor.
- Fire-rate auditing built in via `controller_activity` smoke.
- Standardised diagnostics — HEALTH_DIAG's `controllers [...]` line derived from monitor state.
**Execution:** a series of internal commits — spec+plan revision first, then revert any prior CPU-compute implementation, then per-kernel implementation + CPU monitor pairs. Old CPU-side scaffolding is removed in the SAME commit that lands its GPU-kernel replacement, per `feedback_no_partial_refactor.md`.
### Part D — Temporal Architecture