plan(dqn-v2): Plan 3 revision — GPU-drives-CPU-reads, ISV tail-append, AdaptiveMonitor

Aligns Plan 3 with post-Plan-2 reality:

1. AdaptiveController → AdaptiveMonitor throughout (spec §4.C.6
   2026-04-24 revision). Tasks 4 (plan-threshold) and 9 (cql_alpha
   seed-coupled) become GPU kernel + read-only CPU monitor pairs
   instead of CPU-compute controllers — uniform with Plan 1's
   tau/epsilon/gamma/kelly_cap and Plan 2's per-branch γ pattern.

2. Stale ISV slot indices [49..65) replaced with tail-append language.
   Current post-Plan-2 ISV_TOTAL_DIM = 63; new slots start at 63 and
   grow. Fingerprint at 61-62 auto-re-tails.

3. Architecture paragraph updated to document replay-seed orchestration
   (Task 8 dqn_replay_seed.rs) as constructor-time cold-path pre-training
   — not a GPU-drives violation.

4. Prerequisites section clarified: Plan 2 state — ISV_TOTAL_DIM 63,
   TLOB at SL_TLOB_START, layout fingerprint auto-recomputes.

No scope changes to individual Tasks 1-10. Just terminology + slot
indexing aligned with current main state.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-04-24 21:46:38 +02:00
parent 3c18ebd637
commit a59e7599c8

View File

@@ -4,14 +4,19 @@
**Goal:** Land the behavioural machinery that escapes the val-Flat-trap: ISV-scaled continuous reward shaping, novelty-weighted trade-attempt bonus, adaptive plan-MLP activation threshold, per-component reward attribution, state-distribution divergence detection, temporal timing rewards, and a scripted-portfolio replay warm-start with CQL-α decoupling. Every change is ISV-driven (Invariant 1) and wires producers to consumers in the same commit (Invariant 2).
**Architecture:** 15 new ISV slots spanning `[49..65)`, grouped by Part: B = 3 slots ([49..52)), C.2 = 6 slots ([52..58)), C.3 = 3 slots ([58], [63..65)), D.4 = 3 slots ([59..62)). One new controller (`PlanMlpThresholdController`) via the AdaptiveController trait (Plan 1 C.6). One new replay-buffer seed harness module (`dqn_replay_seed.rs`) driven by a portfolio of 4 scripted policies. CQL α becomes ISV-coupled to the seed-displacement signal.
**Architecture:** ~15 new ISV slots appended AT TAIL after current `ISV_TOTAL_DIM = 63` (post-Plan-2). Grouped by Part: C.2 = 6 slots (reward-component EMAs), B = 3 slots (Flat-scale, trade-attempt, trade-target), B.4 = 1 slot (plan-threshold-eff), D.4 = 3 slots (temporal reward EMAs), C.3 = 3 slots (state-KL). Layout fingerprint re-tails automatically; final `ISV_TOTAL_DIM ≈ 78`.
**GPU-drives-CPU-reads architecture (spec §4.C.6 2026-04-24 revision):** every adaptive value in Plan 3 is computed by a GPU kernel reading ISV signals and written to an ISV slot. CPU-side `AdaptiveMonitor` impls are READ-ONLY observers. What the original plan called "controllers" — `PlanMlpThresholdController` (Task 4), `CqlAlphaSeedCoupledController` (Task 9) — become GPU kernel + read-only CPU monitor pairs, uniform with Plan 1's tau/epsilon/gamma/kelly_cap and Plan 2's per-branch-gamma pattern.
One new replay-buffer seed harness module (`dqn_replay_seed.rs`) driven by a portfolio of 4 scripted policies (Task 8). Replay-seed orchestration runs PRE-training (cold path, one-shot at constructor-time before epoch 1); does not violate GPU-drives.
**Tech Stack:** Rust workspace + CUDA C++ via nvcc, cudarc 0.19, sccache with `CARGO_INCREMENTAL=0`. No new crate dependencies. New CUDA kernel files: `reward_component_audit_kernel.cu` (host-readable, pinned), `state_kl_divergence_kernel.cu`.
**Authority:** `docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md` §4.B, §4.C.2C.5, §4.D.4. All 9 invariants apply.
**Dependencies on prior plans:**
- Plan 1: `StateResetRegistry`, `AdaptiveController` trait, `ISV_LAYOUT_FINGERPRINT_{LO,HI}_INDEX`, audit docs.
- Plan 1: `StateResetRegistry`, `AdaptiveMonitor` trait (read-only observer, spec §4.C.6 2026-04-24 revision), `ISV_LAYOUT_FINGERPRINT_{LO,HI}_INDEX`, audit docs.
- Plan 2: layout-fingerprint auto-recompute, current `ISV_TOTAL_DIM = 63`, TLOB cuBLAS integration, per-branch γ, quantile atoms. New slots in Plan 3 go AT TAIL after index 60 (TLOB_REGIME_FOCUS_EMA); fingerprint (currently 61-62) shifts to new tail.
- Plan 2: per-branch γ ISV slots at `[37..41)` already populated; `Q_P05/Q_P95_EMA` at `[41..49)` already populated; layout fingerprint updated (seed bytes cover Plan 2 slots); `PLAN_ISV_REMAINING_FRACTION` at `plan_isv[6]`.
---
@@ -107,7 +112,7 @@ pub const REWARD_OPP_COST_EMA_INDEX: usize = 56;
pub const REWARD_BONUS_EMA_INDEX: usize = 57;
```
Keep `ISV_TOTAL_DIM` unchanged if Plan 2 sized it to 72 already. If Plan 2 bumped to exactly the slots it needed (e.g. `ISV_TOTAL_DIM = 49`), grow to `58`:
Current `ISV_TOTAL_DIM` post-Plan-2 is **63**. New slots append at 63 and grow from there; fingerprint re-tails to the new highest indices. Per-task slot allocation is documented at the start of each task; exact indices computed during implementation based on ISV_TOTAL_DIM at task start.
```rust
pub const ISV_TOTAL_DIM: usize = 58; // Plan 3 Task 1 grows from <49..58>.
@@ -445,7 +450,7 @@ pub const TRADE_ATTEMPT_RATE_EMA_INDEX: usize = 50;
pub const TRADE_TARGET_RATE_INDEX: usize = 51;
```
Bump `ISV_TOTAL_DIM` if it was at 58 from Task 1 — now 58 still, but the slots at 50/51 were unallocated in Plan 2. If Plan 2 left `ISV_TOTAL_DIM = 49`, this task grows to `52`. Any inconsistency here indicates Task 1's ISV_TOTAL_DIM bump was wrong; fix Task 1 first, not this one.
Grow `ISV_TOTAL_DIM` by +N slots for whatever this task requires; re-tail the fingerprint. Implementation computes exact indices from ISV_TOTAL_DIM at task start.
- [ ] **Step 3.2: Write the failing test**
@@ -652,7 +657,7 @@ Expected: no adaptive path, both counts similar.
Create `crates/ml/src/trainers/dqn/adaptive_controllers/plan_mlp_threshold.rs`:
```rust
use super::AdaptiveController;
use super::AdaptiveMonitor;
pub struct PlanMlpThresholdController {
alpha: f32,
@@ -681,7 +686,7 @@ impl PlanMlpThresholdController {
}
}
impl AdaptiveController for PlanMlpThresholdController {
impl AdaptiveMonitor for PlanMlpThresholdController {
type Signal = PlanParamsBatchSummary;
type Control = f32; // the threshold
@@ -1295,7 +1300,7 @@ pub struct CqlAlphaSeedCoupledController {
seed_fraction_ema: f32,
}
impl AdaptiveController for CqlAlphaSeedCoupledController {
impl AdaptiveMonitor for CqlAlphaSeedCoupledController {
type Signal = f32; // current seed fraction in replay buffer
type Control = f32; // cql_alpha
@@ -1460,7 +1465,7 @@ git commit -m "plan3(task10): Tier 1 + Tier 2 (behavioural) validation passes
- [ ] All 9 invariants preserved across every commit in this plan.
- [ ] 15 new ISV slots allocated [49..65), all with named constants, all in `docs/isv-slots.md`.
- [ ] 4 scripted policies (uniform, momentum, mean_reversion, vwap_deviation) wired into replay seed.
- [ ] 2 new AdaptiveController impls (`PlanMlpThresholdController`, `CqlAlphaSeedCoupledController`) registered with the controller harness.
- [ ] 2 new AdaptiveMonitor impls (`PlanMlpThresholdController`, `CqlAlphaSeedCoupledController`) registered with the controller harness.
- [ ] HEALTH_DIAG emits `reward_split[..]` + `temporal_reward[..]` lines.
- [ ] Multi-seed × multi-fold Tier 1 + Tier 2 (behavioural subset) exit pass.
- [ ] No stubs, no TODO/FIXME, no deferred work (Invariant 9).