diff --git a/docs/superpowers/plans/2026-04-24-dqn-v2-plan-3-behavioural.md b/docs/superpowers/plans/2026-04-24-dqn-v2-plan-3-behavioural.md index d8c7255c5..2e3401eac 100644 --- a/docs/superpowers/plans/2026-04-24-dqn-v2-plan-3-behavioural.md +++ b/docs/superpowers/plans/2026-04-24-dqn-v2-plan-3-behavioural.md @@ -4,14 +4,19 @@ **Goal:** Land the behavioural machinery that escapes the val-Flat-trap: ISV-scaled continuous reward shaping, novelty-weighted trade-attempt bonus, adaptive plan-MLP activation threshold, per-component reward attribution, state-distribution divergence detection, temporal timing rewards, and a scripted-portfolio replay warm-start with CQL-α decoupling. Every change is ISV-driven (Invariant 1) and wires producers to consumers in the same commit (Invariant 2). -**Architecture:** 15 new ISV slots spanning `[49..65)`, grouped by Part: B = 3 slots ([49..52)), C.2 = 6 slots ([52..58)), C.3 = 3 slots ([58], [63..65)), D.4 = 3 slots ([59..62)). One new controller (`PlanMlpThresholdController`) via the AdaptiveController trait (Plan 1 C.6). One new replay-buffer seed harness module (`dqn_replay_seed.rs`) driven by a portfolio of 4 scripted policies. CQL α becomes ISV-coupled to the seed-displacement signal. +**Architecture:** ~15 new ISV slots appended AT TAIL after current `ISV_TOTAL_DIM = 63` (post-Plan-2). Grouped by Part: C.2 = 6 slots (reward-component EMAs), B = 3 slots (Flat-scale, trade-attempt, trade-target), B.4 = 1 slot (plan-threshold-eff), D.4 = 3 slots (temporal reward EMAs), C.3 = 3 slots (state-KL). Layout fingerprint re-tails automatically; final `ISV_TOTAL_DIM ≈ 78`. + +**GPU-drives-CPU-reads architecture (spec §4.C.6 2026-04-24 revision):** every adaptive value in Plan 3 is computed by a GPU kernel reading ISV signals and written to an ISV slot. CPU-side `AdaptiveMonitor` impls are READ-ONLY observers. What the original plan called "controllers" — `PlanMlpThresholdController` (Task 4), `CqlAlphaSeedCoupledController` (Task 9) — become GPU kernel + read-only CPU monitor pairs, uniform with Plan 1's tau/epsilon/gamma/kelly_cap and Plan 2's per-branch-gamma pattern. + +One new replay-buffer seed harness module (`dqn_replay_seed.rs`) driven by a portfolio of 4 scripted policies (Task 8). Replay-seed orchestration runs PRE-training (cold path, one-shot at constructor-time before epoch 1); does not violate GPU-drives. **Tech Stack:** Rust workspace + CUDA C++ via nvcc, cudarc 0.19, sccache with `CARGO_INCREMENTAL=0`. No new crate dependencies. New CUDA kernel files: `reward_component_audit_kernel.cu` (host-readable, pinned), `state_kl_divergence_kernel.cu`. **Authority:** `docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md` §4.B, §4.C.2–C.5, §4.D.4. All 9 invariants apply. **Dependencies on prior plans:** -- Plan 1: `StateResetRegistry`, `AdaptiveController` trait, `ISV_LAYOUT_FINGERPRINT_{LO,HI}_INDEX`, audit docs. +- Plan 1: `StateResetRegistry`, `AdaptiveMonitor` trait (read-only observer, spec §4.C.6 2026-04-24 revision), `ISV_LAYOUT_FINGERPRINT_{LO,HI}_INDEX`, audit docs. +- Plan 2: layout-fingerprint auto-recompute, current `ISV_TOTAL_DIM = 63`, TLOB cuBLAS integration, per-branch γ, quantile atoms. New slots in Plan 3 go AT TAIL after index 60 (TLOB_REGIME_FOCUS_EMA); fingerprint (currently 61-62) shifts to new tail. - Plan 2: per-branch γ ISV slots at `[37..41)` already populated; `Q_P05/Q_P95_EMA` at `[41..49)` already populated; layout fingerprint updated (seed bytes cover Plan 2 slots); `PLAN_ISV_REMAINING_FRACTION` at `plan_isv[6]`. --- @@ -107,7 +112,7 @@ pub const REWARD_OPP_COST_EMA_INDEX: usize = 56; pub const REWARD_BONUS_EMA_INDEX: usize = 57; ``` -Keep `ISV_TOTAL_DIM` unchanged if Plan 2 sized it to 72 already. If Plan 2 bumped to exactly the slots it needed (e.g. `ISV_TOTAL_DIM = 49`), grow to `58`: +Current `ISV_TOTAL_DIM` post-Plan-2 is **63**. New slots append at 63 and grow from there; fingerprint re-tails to the new highest indices. Per-task slot allocation is documented at the start of each task; exact indices computed during implementation based on ISV_TOTAL_DIM at task start. ```rust pub const ISV_TOTAL_DIM: usize = 58; // Plan 3 Task 1 grows from <49..58>. @@ -445,7 +450,7 @@ pub const TRADE_ATTEMPT_RATE_EMA_INDEX: usize = 50; pub const TRADE_TARGET_RATE_INDEX: usize = 51; ``` -Bump `ISV_TOTAL_DIM` if it was at 58 from Task 1 — now 58 still, but the slots at 50/51 were unallocated in Plan 2. If Plan 2 left `ISV_TOTAL_DIM = 49`, this task grows to `52`. Any inconsistency here indicates Task 1's ISV_TOTAL_DIM bump was wrong; fix Task 1 first, not this one. +Grow `ISV_TOTAL_DIM` by +N slots for whatever this task requires; re-tail the fingerprint. Implementation computes exact indices from ISV_TOTAL_DIM at task start. - [ ] **Step 3.2: Write the failing test** @@ -652,7 +657,7 @@ Expected: no adaptive path, both counts similar. Create `crates/ml/src/trainers/dqn/adaptive_controllers/plan_mlp_threshold.rs`: ```rust -use super::AdaptiveController; +use super::AdaptiveMonitor; pub struct PlanMlpThresholdController { alpha: f32, @@ -681,7 +686,7 @@ impl PlanMlpThresholdController { } } -impl AdaptiveController for PlanMlpThresholdController { +impl AdaptiveMonitor for PlanMlpThresholdController { type Signal = PlanParamsBatchSummary; type Control = f32; // the threshold @@ -1295,7 +1300,7 @@ pub struct CqlAlphaSeedCoupledController { seed_fraction_ema: f32, } -impl AdaptiveController for CqlAlphaSeedCoupledController { +impl AdaptiveMonitor for CqlAlphaSeedCoupledController { type Signal = f32; // current seed fraction in replay buffer type Control = f32; // cql_alpha @@ -1460,7 +1465,7 @@ git commit -m "plan3(task10): Tier 1 + Tier 2 (behavioural) validation passes - [ ] All 9 invariants preserved across every commit in this plan. - [ ] 15 new ISV slots allocated [49..65), all with named constants, all in `docs/isv-slots.md`. - [ ] 4 scripted policies (uniform, momentum, mean_reversion, vwap_deviation) wired into replay seed. -- [ ] 2 new AdaptiveController impls (`PlanMlpThresholdController`, `CqlAlphaSeedCoupledController`) registered with the controller harness. +- [ ] 2 new AdaptiveMonitor impls (`PlanMlpThresholdController`, `CqlAlphaSeedCoupledController`) registered with the controller harness. - [ ] HEALTH_DIAG emits `reward_split[..]` + `temporal_reward[..]` lines. - [ ] Multi-seed × multi-fold Tier 1 + Tier 2 (behavioural subset) exit pass. - [ ] No stubs, no TODO/FIXME, no deferred work (Invariant 9).