spec+plan(dqn-v2): GPU drives, CPU reads (§4.C.6 revision)
User raised that the CPU-controller pattern (where CPU computes adaptive values from ISV and writes back to ISV) violates the architectural principle: GPU kernels should compute adaptive decisions; CPU-side code is pure observation. Replaces the AdaptiveController trait with read-only AdaptiveMonitor: - No update() — CPU doesn't compute adaptive values. - No write_output() — CPU doesn't write adaptive values to ISV. - read() returns the GPU-computed value from ISV. - diagnose() emits HEALTH_DIAG snapshot. - observe() + fire_rate() track how often the GPU-computed value changes. Reclassifies the 9 adaptive mechanisms: - 6 reactive get GPU kernel + CPU monitor: atoms, gamma, kelly_cap, tau (Polyak EMA), epsilon, grad_balancer (last is already GPU-driven). - 3 static get ISV constructor-write, no monitor: cql_alpha, conviction_floor, plan_threshold. New ISV slots for GPU-written adaptive outputs (EPSILON_EFF, TAU_EFF, GAMMA_EFF, KELLY_CAP_EFF) and CPU-born inputs (EPOCH_IDX, TOTAL_EPOCHS), plus slots for the 3 static configs. Rationale: - Unified adaptive machinery, no CPU-side special cases. - Per-sample granularity available (Expected SARSA τ in c51_loss_kernel is already the exemplar — reads ISV q_gap + health per sample). - Zero CPU→GPU config transfer in any path. - ISV is single source of truth for every adaptive value. Plan 1 Tasks 8-17 restructured: - Task 8 creates AdaptiveMonitor trait (not AdaptiveController). - Tasks 9, 10, 11, 13, 14, 17: GPU kernel + monitor pairs. - Tasks 12, 15, 16: static ISV writes only. Follow-up commits will revert Batch A (d76849f31) and Batch B (4189da563) which implemented the old CPU-compute pattern, then re-implement under this design. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -2,9 +2,9 @@
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Lay the substrate that all subsequent DQN v2 work depends on: state reset registry, ISV slot contract, audit docs, named-dimension constants, and the AdaptiveController unification refactor. Zero behavioural change to the trained policy — this plan only establishes scaffolding and refactors existing ad-hoc machinery into typed protocols.
|
||||
**Goal:** Lay the substrate that all subsequent DQN v2 work depends on: state reset registry, ISV slot contract, audit docs, named-dimension constants, and the AdaptiveMonitor unification refactor (GPU kernels compute; CPU observes, per spec §4.C.6 revision). Zero behavioural change to the trained policy — this plan only establishes scaffolding and refactors existing ad-hoc machinery into typed protocols.
|
||||
|
||||
**Architecture:** Three audit docs + one config file tracked by pre-commit hook. One `StateResetRegistry` struct categorising every piece of training state. One typed `IsvSlot<T>` contract with a compile-time **layout fingerprint** (structural hash of the slot list) at ISV[0] — fail-fast only, no migration path ever exists. One `AdaptiveController` trait with existing controllers migrated one-at-a-time. Named constants for every semantic index across portfolio state, ISV slots, plan_isv dimensions, plan_params, state-vector offsets, branch indices, and action sub-indices.
|
||||
**Architecture:** Three audit docs + one config file tracked by pre-commit hook. One `StateResetRegistry` struct categorising every piece of training state. One typed `IsvSlot<T>` contract with a compile-time **layout fingerprint** (structural hash of the slot list) at ISV[0..2) — fail-fast only, no migration path ever exists. One read-only `AdaptiveMonitor` trait with 6 GPU-kernel-plus-CPU-monitor pairs (atoms, gamma, kelly_cap, tau, epsilon, grad_balancer) and 3 static-config ISV writes (cql_alpha, conviction_floor, plan_threshold). **GPU drives, CPU reads** — CPU-side code never computes an adaptive value; all computation is in GPU kernels reading from and writing to ISV. Named constants for every semantic index across portfolio state, ISV slots, plan_isv dimensions, plan_params, state-vector offsets, branch indices, and action sub-indices.
|
||||
|
||||
**Tech Stack:** Rust (workspace crates: `ml`, `ml-core`, `ml-dqn`, `ml-supervised`), CUDA C++ via nvcc through `build.rs`, cudarc 0.19, Cargo 2021 edition, SQLx offline mode, sccache with `CARGO_INCREMENTAL=0`.
|
||||
|
||||
@@ -1470,215 +1470,293 @@ EOF
|
||||
|
||||
---
|
||||
|
||||
## Task 8: C.6 AdaptiveController trait + migrate first controller (atoms)
|
||||
## Task 8: C.6 AdaptiveMonitor trait + test harness (GPU-drives architecture)
|
||||
|
||||
**DESIGN NOTE:** Task 8 replaces the earlier `AdaptiveController` trait with a read-only `AdaptiveMonitor` trait. See spec §4.C.6 revision (2026-04-24). The reason: adaptive values are computed by GPU kernels from ISV signals; CPU-side code is pure observation. The old `update()` / `write_output()` trait methods are removed — CPU never computes an adaptive value.
|
||||
|
||||
**Files:**
|
||||
- Create: `crates/ml/src/trainers/dqn/adaptive_controller.rs`
|
||||
- Modify: `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` — use the trait for atom-support controller
|
||||
- Create: `crates/ml/src/trainers/dqn/adaptive_monitor.rs` (renamed from `adaptive_controller.rs`)
|
||||
|
||||
- [ ] **Step 8.1: Write failing test**
|
||||
### Step 8.1: Write the failing test
|
||||
|
||||
Create `crates/ml/src/trainers/dqn/adaptive_controller.rs`:
|
||||
Create `crates/ml/src/trainers/dqn/adaptive_monitor.rs` with ONLY the test module first:
|
||||
|
||||
```rust
|
||||
//! Unified protocol for adaptive controllers in the DQN trainer.
|
||||
//! Spec §4.C.6.
|
||||
//! Read-only observer for GPU-computed adaptive values in the DQN trainer.
|
||||
//! Spec §4.C.6 (2026-04-24 revision: GPU computes, CPU reads).
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn controller_fire_rate_stats_compute() {
|
||||
fn fire_rate_stats_compute() {
|
||||
let mut stats = FireRateStats::default();
|
||||
stats.record_fire(true);
|
||||
stats.record_fire(false);
|
||||
stats.record_fire(true);
|
||||
assert_eq!(stats.fire_count, 2);
|
||||
assert_eq!(stats.observation_count, 3);
|
||||
assert!((stats.fire_rate() - 2.0 / 3.0).abs() < 1e-9);
|
||||
assert!((stats.fire_rate() - 2.0f32 / 3.0f32).abs() < 1e-6);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn diag_snapshot_empty_by_default() {
|
||||
let snap = DiagSnapshot::default();
|
||||
assert!(snap.fields.is_empty());
|
||||
fn diag_snapshot_builder() {
|
||||
let snap = DiagSnapshot::new().with("v", 0.5).with("t", 1.0);
|
||||
assert_eq!(snap.fields.len(), 2);
|
||||
assert_eq!(snap.fields.get("v"), Some(&0.5f32));
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 8.2: Run test to verify it fails**
|
||||
### Step 8.2: Run — expect FAIL
|
||||
|
||||
Run:
|
||||
```bash
|
||||
SQLX_OFFLINE=true CARGO_INCREMENTAL=0 cargo test -p ml adaptive_controller 2>&1 | tail -10
|
||||
SQLX_OFFLINE=true CARGO_INCREMENTAL=0 cargo test -p ml adaptive_monitor 2>&1 | tail -10
|
||||
```
|
||||
|
||||
Expected: compile errors `cannot find type FireRateStats`.
|
||||
|
||||
- [ ] **Step 8.3: Implement the trait**
|
||||
### Step 8.3: Implement the trait
|
||||
|
||||
Prepend to `adaptive_controller.rs` (above the test module):
|
||||
Prepend to `adaptive_monitor.rs` above the test module:
|
||||
|
||||
```rust
|
||||
use std::collections::HashMap;
|
||||
|
||||
/// Per-controller fire-rate statistics. A "fire" is any observation where
|
||||
/// the controller emitted a control value different from its previous emission.
|
||||
/// Used by the `controller_activity` smoke test to verify load-bearing
|
||||
/// controllers don't fire in > 50% of epochs.
|
||||
#[derive(Debug, Clone, Default)]
|
||||
pub struct FireRateStats {
|
||||
pub fire_count: u64,
|
||||
pub observation_count: u64,
|
||||
/// Thin read/write view over the ISV pinned-memory bus.
|
||||
///
|
||||
/// Monitors use `read()` exclusively. The constructor / schedule-input path
|
||||
/// uses `write()` for CPU-born inputs (epoch_idx, static configs). GPU kernels
|
||||
/// write via their device pointer — not via `IsvBus`.
|
||||
pub(crate) struct IsvBus<'a> {
|
||||
signals: &'a mut [f32],
|
||||
}
|
||||
|
||||
impl<'a> IsvBus<'a> {
|
||||
pub(crate) fn new(signals: &'a mut [f32]) -> Self {
|
||||
Self { signals }
|
||||
}
|
||||
pub(crate) fn read(&self, slot: usize) -> f32 { self.signals[slot] }
|
||||
pub(crate) fn write(&mut self, slot: usize, value: f32) { self.signals[slot] = value; }
|
||||
pub(crate) fn len(&self) -> usize { self.signals.len() }
|
||||
}
|
||||
|
||||
/// Fire-rate statistics tracked by a monitor across observations.
|
||||
///
|
||||
/// A "fire" is a monitor observation where the value materially changed from
|
||||
/// the previous observation. Used by `controller_activity` smoke to verify
|
||||
/// load-bearing mechanisms do not fire in > 50% of epochs.
|
||||
#[derive(Debug, Default, Clone)]
|
||||
pub(crate) struct FireRateStats {
|
||||
fires: u32,
|
||||
epochs: u32,
|
||||
}
|
||||
|
||||
impl FireRateStats {
|
||||
pub fn record_fire(&mut self, fired: bool) {
|
||||
self.observation_count += 1;
|
||||
if fired {
|
||||
self.fire_count += 1;
|
||||
}
|
||||
pub(crate) fn record_fire(&mut self, fired: bool) {
|
||||
self.epochs += 1;
|
||||
if fired { self.fires += 1; }
|
||||
}
|
||||
|
||||
pub fn fire_rate(&self) -> f64 {
|
||||
if self.observation_count == 0 {
|
||||
0.0
|
||||
} else {
|
||||
self.fire_count as f64 / self.observation_count as f64
|
||||
}
|
||||
pub(crate) fn fire_rate(&self) -> f32 {
|
||||
if self.epochs == 0 { 0.0 } else { self.fires as f32 / self.epochs as f32 }
|
||||
}
|
||||
|
||||
pub fn reset(&mut self) {
|
||||
*self = Self::default();
|
||||
pub(crate) fn reset(&mut self) {
|
||||
self.fires = 0;
|
||||
self.epochs = 0;
|
||||
}
|
||||
}
|
||||
|
||||
/// Standardised diagnostic emission from an adaptive controller.
|
||||
/// Emitted at epoch boundary into HEALTH_DIAG.
|
||||
#[derive(Debug, Clone, Default)]
|
||||
pub struct DiagSnapshot {
|
||||
/// Named fields → value. Example: `{"gamma": 0.95, "fire_rate": 0.12}`.
|
||||
pub fields: HashMap<String, f64>,
|
||||
/// Diagnostic snapshot emitted by a monitor each epoch.
|
||||
///
|
||||
/// Uses `&'static str` keys (no allocation per key). Built via `.with()`.
|
||||
#[derive(Debug, Default, Clone)]
|
||||
pub(crate) struct DiagSnapshot {
|
||||
pub fields: HashMap<&'static str, f32>,
|
||||
}
|
||||
|
||||
impl DiagSnapshot {
|
||||
pub fn with(mut self, name: impl Into<String>, value: f64) -> Self {
|
||||
self.fields.insert(name.into(), value);
|
||||
pub(crate) fn new() -> Self { Self { fields: HashMap::new() } }
|
||||
pub(crate) fn with(mut self, key: &'static str, value: f32) -> Self {
|
||||
self.fields.insert(key, value);
|
||||
self
|
||||
}
|
||||
}
|
||||
|
||||
/// Unified protocol for adaptive controllers in the DQN trainer.
|
||||
/// Read-only observer for a GPU-computed adaptive value.
|
||||
///
|
||||
/// Each controller reads signals from the ISV bus, computes a control value
|
||||
/// (gamma, atom positions, Kelly floor, CQL alpha, etc.), writes its output
|
||||
/// back to ISV, and emits diagnostics.
|
||||
pub trait AdaptiveController {
|
||||
/// Opaque signal vector the controller reads.
|
||||
type Signal;
|
||||
/// Opaque control value the controller emits.
|
||||
type Control;
|
||||
/// All adaptive values are computed by GPU kernels from ISV signals. CPU-side
|
||||
/// code does NOT compute adaptive values; it only reads and reports. See
|
||||
/// spec §4.C.6 (2026-04-24 revision).
|
||||
pub(crate) trait AdaptiveMonitor {
|
||||
/// Read the GPU-computed current value from ISV.
|
||||
fn read(&self, isv: &IsvBus<'_>) -> f32;
|
||||
|
||||
/// Read inputs from the ISV bus.
|
||||
fn read_signals(&self, isv: &IsvBus) -> Self::Signal;
|
||||
/// Emit diagnostic snapshot for HEALTH_DIAG.
|
||||
fn diagnose(&self, isv: &IsvBus<'_>) -> DiagSnapshot;
|
||||
|
||||
/// Compute the new control value. May update internal state.
|
||||
fn update(&mut self, signals: Self::Signal) -> Self::Control;
|
||||
/// Record one observation; drives fire-rate tracking at epoch boundary.
|
||||
fn observe(&mut self, current: f32);
|
||||
|
||||
/// Write the control value back to ISV / consumer buffers.
|
||||
fn write_output(&self, ctrl: &Self::Control, isv: &mut IsvBus);
|
||||
|
||||
/// Per-controller fire-rate statistics.
|
||||
/// Fire-rate statistics across prior observations.
|
||||
fn fire_rate(&self) -> &FireRateStats;
|
||||
|
||||
/// Emit a diagnostic snapshot at epoch boundary.
|
||||
fn diagnose(&self) -> DiagSnapshot;
|
||||
|
||||
/// Controller name, used in HEALTH_DIAG field labels.
|
||||
/// Short name used in diagnostic output (e.g. "gamma", "kelly_cap").
|
||||
fn name(&self) -> &'static str;
|
||||
}
|
||||
|
||||
/// Abstraction over the ISV bus for controllers.
|
||||
/// In production, this wraps `isv_signals_pinned: *mut f32`.
|
||||
/// In tests, a plain `Vec<f32>` is used for isolation.
|
||||
pub struct IsvBus<'a> {
|
||||
pub(crate) slots: &'a mut [f32],
|
||||
}
|
||||
|
||||
impl<'a> IsvBus<'a> {
|
||||
pub fn new(slots: &'a mut [f32]) -> Self {
|
||||
Self { slots }
|
||||
}
|
||||
pub fn read(&self, idx: usize) -> f32 { self.slots[idx] }
|
||||
pub fn write(&mut self, idx: usize, v: f32) { self.slots[idx] = v; }
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 8.4: Run test to verify it passes**
|
||||
### Step 8.4: Run test — expect PASS
|
||||
|
||||
```bash
|
||||
SQLX_OFFLINE=true CARGO_INCREMENTAL=0 cargo test -p ml adaptive_controller 2>&1 | tail -5
|
||||
SQLX_OFFLINE=true CARGO_INCREMENTAL=0 cargo test -p ml adaptive_monitor 2>&1 | tail -5
|
||||
```
|
||||
|
||||
Expected: `test result: ok. 2 passed; 0 failed`.
|
||||
Expected: 2 passed.
|
||||
|
||||
- [ ] **Step 8.5: Commit the trait and harness**
|
||||
### Step 8.5: Export from mod.rs
|
||||
|
||||
In `crates/ml/src/trainers/dqn/mod.rs`:
|
||||
```rust
|
||||
pub(crate) mod adaptive_monitor;
|
||||
```
|
||||
|
||||
### Step 8.6: Compile-check
|
||||
|
||||
```bash
|
||||
git add crates/ml/src/trainers/dqn/adaptive_controller.rs \
|
||||
SQLX_OFFLINE=true CARGO_INCREMENTAL=0 cargo check -p ml 2>&1 | tail -3
|
||||
```
|
||||
|
||||
Expected: `Finished` with the 8-warning baseline.
|
||||
|
||||
### Step 8.7: Audit doc
|
||||
|
||||
Append to `docs/dqn-wire-up-audit.md`:
|
||||
```markdown
|
||||
| `trainers/dqn/adaptive_monitor.rs` | Read-only observer trait + harness (FireRateStats, DiagSnapshot, IsvBus<'a>); consumers added in Plan 1 Tasks 9-17 (atoms/gamma/kelly_cap/tau/epsilon/grad_balancer monitors) | Wired (consumers added in same plan) | C.6 GPU-drives-CPU-reads | — |
|
||||
```
|
||||
|
||||
### Step 8.8: Commit
|
||||
|
||||
```bash
|
||||
git add crates/ml/src/trainers/dqn/adaptive_monitor.rs \
|
||||
crates/ml/src/trainers/dqn/mod.rs \
|
||||
docs/dqn-wire-up-audit.md
|
||||
git commit -m "$(cat <<'EOF'
|
||||
feat(dqn-v2): C.6 AdaptiveController trait + test harness
|
||||
feat(dqn-v2): C.6 AdaptiveMonitor trait (GPU-drives, CPU-reads)
|
||||
|
||||
Unified protocol for every adaptive controller in the DQN trainer.
|
||||
FireRateStats for controller_activity smoke. DiagSnapshot for
|
||||
standardised HEALTH_DIAG emission. IsvBus abstraction over pinned
|
||||
device-mapped memory.
|
||||
Read-only observer trait for GPU-computed adaptive values. Replaces
|
||||
the earlier AdaptiveController trait which had CPU-side update() and
|
||||
write_output() — those violated the newly-codified principle that
|
||||
GPU kernels compute all adaptive decisions and CPU-side code only
|
||||
reads from ISV.
|
||||
|
||||
No concrete controller migration yet — Task 9 migrates atoms (first
|
||||
in the sequence per spec §8 C.6 decision: atoms → gamma → Kelly →
|
||||
cql_alpha → tau → epsilon → conviction_floor → plan_threshold →
|
||||
balancer).
|
||||
FireRateStats for controller_activity smoke. DiagSnapshot built via
|
||||
.with() chain. IsvBus read/write (write for CPU-born inputs only —
|
||||
epoch_idx, static configs; NEVER for adaptive values).
|
||||
|
||||
Tests: 2 unit tests on FireRateStats and DiagSnapshot.
|
||||
|
||||
Plan 1 Task 8. Spec §4.C.6.
|
||||
Plan 1 Task 8. Spec §4.C.6 (2026-04-24 revision).
|
||||
|
||||
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||||
EOF
|
||||
)"
|
||||
```
|
||||
|
||||
### Tasks 9 through 16: migrate each adaptive controller in spec-specified order
|
||||
|
||||
For each controller, follow this pattern:
|
||||
|
||||
1. **Write failing test** for the controller's `impl AdaptiveController`.
|
||||
2. **Run test to fail** (trait impl doesn't exist yet).
|
||||
3. **Implement the trait** for the controller; delete the old ad-hoc function.
|
||||
4. **Wire consumers** to call through the trait (may be via a central dispatcher that holds all `Box<dyn AdaptiveController>`).
|
||||
5. **Run test to pass.**
|
||||
6. **Run smoke tests** (`controller_activity` is the key one — it asserts fire rates).
|
||||
7. **Update docs/dqn-wire-up-audit.md and docs/isv-slots.md**.
|
||||
8. **Commit** with spec-referenced message.
|
||||
|
||||
Concrete controllers in spec order:
|
||||
- Task 9: atoms controller (`recompute_atom_positions` + `warm_start_atom_positions`)
|
||||
- Task 10: gamma controller (the `adaptive_gamma` in `training_loop.rs`)
|
||||
- Task 11: Kelly cap controller (`kelly_position_cap`'s `conviction` + `safety_multiplier` computation)
|
||||
- Task 12: cql_alpha controller (currently static; prepares for the seed-phase coupling in Plan 3)
|
||||
- Task 13: tau controller (stochastic SARSA temperature in `c51_loss_kernel`)
|
||||
- Task 14: epsilon controller (cosine-annealed with floor)
|
||||
- Task 15: conviction-floor controller (IQL branch_scales floor, currently static at ISV[36])
|
||||
- Task 16: plan-threshold controller (currently hardcoded 0.5, prepares for B.4 in Plan 3)
|
||||
- Task 17: grad-balancer controller (the `grad_balance_isv_update` kernel — wraps it under the trait)
|
||||
|
||||
After the last migration (Task 17) is complete, the old ad-hoc update functions are all removed in Task 17's commit — per spec §4.C.6 "Old scaffolding is removed in the SAME commit that migrates its last consumer".
|
||||
|
||||
---
|
||||
|
||||
## Tasks 9-17: Reactive GPU kernels + CPU monitors (new structure)
|
||||
|
||||
Each of Tasks 9-11, 13-14, 17 lands a GPU kernel + CPU `AdaptiveMonitor` pair. Tasks 12, 15, 16 are static-config ISV constructor writes (no monitor needed). All old CPU-compute scaffolding is removed in the same commit as the replacement lands (`feedback_no_partial_refactor.md`).
|
||||
|
||||
### Common pattern for GPU-kernel + monitor pairs
|
||||
|
||||
1. **Write failing CPU-monitor unit test** — monitor reads a fixture ISV slice, asserts value + diagnose fields.
|
||||
2. **Write failing GPU-kernel smoke test** — launch kernel with known ISV inputs, read back, assert expected output.
|
||||
3. **Implement kernel** in `crates/ml/src/cuda_pipeline/<name>_update_kernel.cu`. Reads ISV slots, writes to designated output slot.
|
||||
4. **Wire kernel launch** in `fused_training.rs::process_epoch_boundary` (cold path, per-epoch).
|
||||
5. **Implement monitor** in `crates/ml/src/trainers/dqn/monitors/<name>_monitor.rs`. Read-only.
|
||||
6. **Delete old CPU-compute code** (the pre-revert leftover, if any).
|
||||
7. **Run tests** — expect PASS.
|
||||
8. **Audit doc + isv-slots.md** — add rows for new slots + new monitor.
|
||||
9. **Commit.**
|
||||
|
||||
### Task 9: atoms kernel + monitor
|
||||
|
||||
- New kernel: `cuda_pipeline/atoms_update_kernel.cu`. Reads ISV v-range slots (`V_CENTER_*`, `V_HALF_*`); writes atom positions to device buffer (existing) + optional summary slot.
|
||||
- New monitor: `trainers/dqn/monitors/atoms_monitor.rs`. Reads atom-summary ISV slots; emits diagnostic.
|
||||
- Old CPU code deleted: `fused.recompute_atom_positions()` CPU path and `warm_start_atom_positions()` CPU path.
|
||||
|
||||
### Task 10: gamma kernel + monitor
|
||||
|
||||
- New ISV slot: `GAMMA_EFF_INDEX` (GPU-written).
|
||||
- New kernel: `cuda_pipeline/gamma_update_kernel.cu`. Reads ISV Q-stats util signals; writes to `GAMMA_EFF_INDEX`.
|
||||
- New monitor: `trainers/dqn/monitors/gamma_monitor.rs`.
|
||||
- Old CPU code deleted: the `adaptive_gamma` update block in `training_loop.rs`.
|
||||
|
||||
### Task 11: kelly_cap kernel + monitor
|
||||
|
||||
- New ISV slot: `KELLY_CAP_EFF_INDEX`.
|
||||
- New kernel: `cuda_pipeline/kelly_cap_update_kernel.cu`. Reads Kelly stats from ISV (via the already-ISV-exposed Kelly slot summary); writes to `KELLY_CAP_EFF_INDEX`.
|
||||
- New monitor: `trainers/dqn/monitors/kelly_cap_monitor.rs`.
|
||||
- Old CPU code deleted: Kelly f_mean / avg_win_ratio computation in `training_loop.rs`.
|
||||
|
||||
### Task 12: cql_alpha static ISV write (NO monitor, NO kernel)
|
||||
|
||||
- New ISV slot: `CQL_ALPHA_INDEX`.
|
||||
- Constructor writes `hyperparams.cql_alpha` to `CQL_ALPHA_INDEX` once.
|
||||
- CQL kernel launch reads from ISV slot instead of a config field.
|
||||
- No monitor — value is static, identity read.
|
||||
- Plan 3 B.3 will rewrite this into a kernel that updates `CQL_ALPHA_INDEX` from replay-seed-phase signals; at that point Task 12's "no kernel" becomes "GPU kernel". This plan stays static.
|
||||
|
||||
### Task 13: tau (Polyak EMA) kernel + monitor
|
||||
|
||||
- New ISV slots: `TAU_EFF_INDEX` (GPU-written), `EPOCH_IDX_INDEX` (CPU-written at epoch boundary), `TOTAL_EPOCHS_INDEX` (CPU-written at constructor).
|
||||
- New kernel: `cuda_pipeline/tau_update_kernel.cu`. Reads `EPOCH_IDX_INDEX`, `TOTAL_EPOCHS_INDEX`, `LEARNING_HEALTH_INDEX`; computes cosine schedule + health floor; writes to `TAU_EFF_INDEX`.
|
||||
- New monitor: `trainers/dqn/monitors/tau_monitor.rs`.
|
||||
- `iqn.set_tau_host()` rewritten to read from `TAU_EFF_INDEX` (via pinned pointer — zero-copy).
|
||||
- Old CPU code deleted: the cosine-schedule + `apply_health_coupled_tau_floor()` computation in `fused_training.rs`.
|
||||
|
||||
### Task 14: epsilon kernel + monitor
|
||||
|
||||
- New ISV slot: `EPSILON_EFF_INDEX` (GPU-written). Reuses `EPOCH_IDX_INDEX` / `TOTAL_EPOCHS_INDEX` from Task 13.
|
||||
- New kernel: `cuda_pipeline/epsilon_update_kernel.cu`. Reads `EPOCH_IDX_INDEX`, `TOTAL_EPOCHS_INDEX`, and any volatility-reactive signal; writes to `EPSILON_EFF_INDEX`.
|
||||
- New monitor: `trainers/dqn/monitors/epsilon_monitor.rs`.
|
||||
- Old CPU code deleted: `base_floor * (0.5 + volatility)` block in `initialize_epoch_state`.
|
||||
|
||||
### Task 15: conviction_floor static ISV write (NO monitor, NO kernel)
|
||||
|
||||
- New ISV slot: `CONVICTION_FLOOR_INDEX`. Constructor writes the configured floor (typically 0.1). IQL kernel reads from `CONVICTION_FLOOR_INDEX`.
|
||||
- No monitor.
|
||||
|
||||
### Task 16: plan_threshold static ISV write (NO monitor, NO kernel)
|
||||
|
||||
- New ISV slot: `PLAN_THRESHOLD_INDEX`. Constructor writes `0.5`. `backtest_plan_kernel.cu` and `experience_kernels.cu` read from `PLAN_THRESHOLD_INDEX`.
|
||||
- No monitor. Plan 3 B.4 rewrites these kernels to read from `PLAN_PARAMS_0_EMA_INDEX` — transforming this slot to dynamic.
|
||||
|
||||
### Task 17: grad_balancer monitor only (kernel already exists)
|
||||
|
||||
- Kernel `grad_balance_isv_update` already GPU-driven — no kernel work.
|
||||
- New monitor: `trainers/dqn/monitors/grad_balancer_monitor.rs`. Reads `GRAD_NORM_TARGET_*` and `GRAD_SCALE_LIMIT_INDEX`.
|
||||
|
||||
### Per-task-batch smoke validation
|
||||
|
||||
After each task's commit:
|
||||
|
||||
```bash
|
||||
SQLX_OFFLINE=true CARGO_INCREMENTAL=0 cargo check -p ml 2>&1 | tail -3
|
||||
SQLX_OFFLINE=true CARGO_INCREMENTAL=0 cargo test -p ml --lib -- monitors:: adaptive_monitor 2>&1 | tail -15
|
||||
```
|
||||
|
||||
### End-of-block scaffolding removal
|
||||
|
||||
Task 17's commit also removes any shared CPU-side controller-dispatcher scaffolding, per §4.C.6's "Old scaffolding is removed in the SAME commit that migrates its last consumer". If the pre-revert `controllers/` module directory contains stale files, they are removed here. The final layout is:
|
||||
|
||||
- `crates/ml/src/trainers/dqn/adaptive_monitor.rs` — trait + harness.
|
||||
- `crates/ml/src/trainers/dqn/monitors/` — 6 monitor impls (atoms, gamma, kelly_cap, tau, epsilon, grad_balancer).
|
||||
- `crates/ml/src/cuda_pipeline/` — 5 new `*_update_kernel.cu` files (atoms, gamma, kelly_cap, tau, epsilon). `grad_balance_isv_update` kernel stays.
|
||||
- ISV slots for 3 static configs (cql_alpha, conviction_floor, plan_threshold) written at constructor.
|
||||
|
||||
|
||||
## Task 18: Plan 1 validation run
|
||||
|
||||
**Files:**
|
||||
@@ -1739,7 +1817,7 @@ Plan 1 is "done" when:
|
||||
2. `docs/dqn-wire-up-audit.md`, `docs/isv-slots.md`, `docs/dqn-gpu-hot-path-audit.md`, `docs/dqn-named-dims.md`, `docs/ml-supervised-to-dqn-concept-audit.md`, `config/metric-bands.toml` all exist and are populated.
|
||||
3. Every raw `ps[N]`, `plan_isv[N]`, `plan_params[N]` access has been replaced with a named constant.
|
||||
4. `StateResetRegistry` is the sole fold-reset dispatch path.
|
||||
5. `AdaptiveController` trait has 9 concrete impls covering every previous ad-hoc controller.
|
||||
5. `AdaptiveMonitor` trait has 6 concrete impls (atoms, gamma, kelly_cap, tau, epsilon, grad_balancer — all read-only observers of GPU-computed values). 3 static configs (cql_alpha, conviction_floor, plan_threshold) written to ISV at constructor. All adaptive computation happens in GPU kernels; CPU-side code never computes adaptive values (spec §4.C.6, 2026-04-24 revision).
|
||||
6. ISV layout fingerprint at ISV[0] is populated at construct (via compile-time structural hash) and fail-fast-checked at checkpoint load. No `migrate_*` functions exist. No integer version sequence.
|
||||
7. Hot-path purity audit has zero `MIGRATE` entries remaining.
|
||||
8. All smoke tests pass.
|
||||
|
||||
@@ -373,38 +373,77 @@ warmup_steps = tracked via ISV, proportional to the seed-phase decay in B.3
|
||||
|
||||
While seed trajectories dominate the buffer, `cql_alpha` is near zero (no pessimism). As seed trajectories displace, `cql_alpha` ramps to its final value.
|
||||
|
||||
#### C.6 Adaptive Controller Unification (major refactor)
|
||||
#### C.6 Adaptive Mechanism Unification (major refactor)
|
||||
|
||||
Current: the DQN has many adaptive controllers (atoms, gamma, grad-balancer, Kelly cap, tau, cql_alpha, epsilon, conviction-floor, plan-threshold, and new ones added by this spec). Each was bolted on independently with its own update rule and ISV interaction.
|
||||
Current: the DQN has many adaptive mechanisms (atoms, gamma, grad-balancer, Kelly cap, tau, cql_alpha, epsilon, conviction-floor, plan-threshold, and new ones added by this spec). Each was bolted on independently with its own update rule and ISV interaction. Most were CPU-computed and pushed to GPU consumers.
|
||||
|
||||
Design: unify under `trait AdaptiveController`:
|
||||
**Design — GPU computes, CPU reads (non-negotiable):**
|
||||
|
||||
All adaptive values and all schedules are computed by GPU kernels from ISV-visible inputs and written to ISV slots. CPU-side code has two roles only:
|
||||
|
||||
1. **CPU-born inputs to ISV** — values that fundamentally originate on CPU (epoch_idx, total_epochs, static config constants) are written to dedicated ISV slots once at the appropriate cadence (epoch boundary for epoch_idx, constructor for static configs). This is a pinned-memory zero-copy write.
|
||||
2. **Read-only observation** — CPU reads ISV slots to emit HEALTH_DIAG diagnostics and track fire-rate statistics for the `controller_activity` smoke.
|
||||
|
||||
CPU never computes an adaptive value or schedule output. If the value is f(ISV_signals), a GPU kernel computes it.
|
||||
|
||||
**Rationale:**
|
||||
|
||||
- Unified adaptive machinery: every adaptive mechanism follows the same on-GPU pattern. No CPU-orchestrated special cases.
|
||||
- Per-sample granularity available — adaptive mechanisms can be per-sample if useful (e.g., Expected SARSA τ already does this inline in c51_loss_kernel), whereas CPU controllers are bounded to per-epoch.
|
||||
- Zero CPU→GPU config transfer in the hot path. CPU writes to ISV via the pinned pointer (no copy); GPU reads from ISV directly.
|
||||
- ISV is the single source of truth for every adaptive value. CPU-side code is pure observability, enforcing Invariant 3's spirit beyond the hot path into epoch-boundary code as well.
|
||||
- No conflation of "schedule" (epoch-deterministic) and "adaptive" (reactive to observed signals): both compute on GPU. The input signals differ; the compute location does not.
|
||||
|
||||
**The observer trait:**
|
||||
|
||||
```rust
|
||||
trait AdaptiveController {
|
||||
type Signal;
|
||||
type Control;
|
||||
|
||||
fn read_signals(&self, isv: &IsvBus) -> Self::Signal;
|
||||
fn update(&mut self, signals: Self::Signal) -> Self::Control;
|
||||
fn fire_rate(&self) -> FireRateStats;
|
||||
fn diagnose(&self) -> DiagSnapshot;
|
||||
pub(crate) trait AdaptiveMonitor {
|
||||
/// Read the GPU-computed current value from ISV.
|
||||
fn read(&self, isv: &IsvBus<'_>) -> f32;
|
||||
/// Emit diagnostic snapshot for HEALTH_DIAG.
|
||||
fn diagnose(&self, isv: &IsvBus<'_>) -> DiagSnapshot;
|
||||
/// Record one observation; drives fire-rate tracking across epoch boundaries.
|
||||
fn observe(&mut self, current: f32);
|
||||
fn fire_rate(&self) -> &FireRateStats;
|
||||
fn name(&self) -> &'static str;
|
||||
}
|
||||
```
|
||||
|
||||
Each existing controller becomes an `impl AdaptiveController`. Shared:
|
||||
No `update()`, no `write_output()`, no `Signal`/`Control` associated types. Pure observation.
|
||||
|
||||
- ISV-read via typed slot handles (A.2).
|
||||
- ISV-write of the control value via typed slot handles.
|
||||
- Fire-rate tracking instrumented by `controller_activity` smoke test.
|
||||
- HEALTH_DIAG emission via `diagnose()` returning a standardised snapshot.
|
||||
**Mechanism → GPU kernel mapping:**
|
||||
|
||||
Benefits:
|
||||
| Mechanism | Nature | Kernel behaviour | CPU monitor? |
|
||||
|---|---|---|---|
|
||||
| atoms | Reactive to Q-distribution | reads ISV v-range; writes atom positions to device buffer + summary to ISV | yes |
|
||||
| gamma | Reactive to Q-stats util | reads ISV Q-stats util; writes γ to ISV | yes |
|
||||
| kelly_cap | Reactive to trade P&L distribution | reads ISV Kelly stats; writes cap to ISV | yes |
|
||||
| tau (Polyak EMA) | Schedule + health-reactive | reads ISV epoch_idx + total_epochs + health; writes τ_eff to ISV | yes |
|
||||
| epsilon | Schedule + volatility-reactive | reads ISV epoch_idx + total_epochs + volatility; writes ε to ISV | yes |
|
||||
| grad_balancer | Reactive to per-branch grad norms | existing `grad_balance_isv_update` kernel (already GPU-driven) | yes (new) |
|
||||
| cql_alpha | Static config | CPU writes to ISV at constructor; no kernel | no |
|
||||
| conviction_floor | Static config | CPU writes to ISV at constructor; no kernel | no |
|
||||
| plan_threshold | Static config | CPU writes to ISV at constructor; no kernel (Plan 3 §B.4 makes reactive) | no |
|
||||
|
||||
- Adding a new adaptive mechanism follows a protocol, not an ad-hoc accretion.
|
||||
- Fire-rate auditing is built-in (addresses `controller_activity` smoke's assertion that load-bearing controllers shouldn't fire in >50% of epochs).
|
||||
- Standardised diagnostics — the HEALTH_DIAG `controllers [anti_lr=true gamma=false clip=true …]` line becomes derived state, not manually-coded.
|
||||
Expected SARSA τ in `c51_loss_kernel` is already per-sample GPU-computed from ISV — no change needed; diagnostic surface already wired via `last_sarsa_tau_factor()`. Exemplar of the pattern.
|
||||
|
||||
Biggest-scope piece in the spec. Executed as a series of internal commits — one per existing controller migrated to the protocol. Old scaffolding (the ad-hoc update functions being replaced) is removed in the SAME commit that migrates its last consumer to the new protocol, not in a deferred cleanup pass. Per `feedback_no_partial_refactor.md` — when a contract changes, every consumer migrates together.
|
||||
**New ISV slots for computed outputs + CPU-born inputs:**
|
||||
|
||||
Allocated in dedicated ranges (§6 appendix). The fingerprint seed updates automatically as slots are added; fingerprint position shifts to the new tail.
|
||||
|
||||
- `EPOCH_IDX_INDEX`, `TOTAL_EPOCHS_INDEX` — CPU-written at epoch boundary and constructor respectively.
|
||||
- `EPSILON_EFF_INDEX`, `TAU_EFF_INDEX`, `GAMMA_EFF_INDEX`, `KELLY_CAP_EFF_INDEX` — GPU-written by the respective kernels.
|
||||
- `CQL_ALPHA_INDEX`, `CONVICTION_FLOOR_INDEX`, `PLAN_THRESHOLD_INDEX` — CPU-written at constructor for the three static configs.
|
||||
|
||||
Rationale for CPU→ISV static writes (rather than Rust constants): consumers read via ISV for consistency with the adaptive mechanisms. A reader cannot tell whether a value is static or GPU-updated; both use the same slot-read pattern. This eliminates a class of "wrong lookup" bugs and gives static values a uniform diagnostic surface.
|
||||
|
||||
**Benefits preserved from the original C.6 design:**
|
||||
|
||||
- Adding a new adaptive mechanism follows a protocol — write a GPU kernel + a CPU monitor.
|
||||
- Fire-rate auditing built in via `controller_activity` smoke.
|
||||
- Standardised diagnostics — HEALTH_DIAG's `controllers [...]` line derived from monitor state.
|
||||
|
||||
**Execution:** a series of internal commits — spec+plan revision first, then revert any prior CPU-compute implementation, then per-kernel implementation + CPU monitor pairs. Old CPU-side scaffolding is removed in the SAME commit that lands its GPU-kernel replacement, per `feedback_no_partial_refactor.md`.
|
||||
|
||||
### Part D — Temporal Architecture
|
||||
|
||||
|
||||
Reference in New Issue
Block a user