diff --git a/docs/superpowers/plans/2026-04-24-dqn-v2-plan-4-supervised-concepts.md b/docs/superpowers/plans/2026-04-24-dqn-v2-plan-4-supervised-concepts.md index ddf051a1d..1298acd10 100644 --- a/docs/superpowers/plans/2026-04-24-dqn-v2-plan-4-supervised-concepts.md +++ b/docs/superpowers/plans/2026-04-24-dqn-v2-plan-4-supervised-concepts.md @@ -4,46 +4,83 @@ **Goal:** Wire the six IN concepts from Part E of the DQN v2 spec into the DQN training pipeline: full TFT Variable Selection Network, Gated Residual Network as a trunk building block (if absent), multi-quantile IQN decomposition, explicit encoder-decoder separation, ISV-exposed attention interpretability, and auxiliary regression/classification heads. Each concept lands wired, diagnostic, and completely finished — no "evaluate later" (Invariant 9). -**Architecture:** Most adoption is additive: a TFT VSN layer in front of the existing trunk, attention-weight ISV plumbing from Mamba2 + TLOB + trunk heads into new diagnostic slots `[63..72)`, auxiliary heads branching off the trunk output via a lightweight two-layer MLP. Encoder-decoder separation is a refactor-not-rewrite: the existing trunk is renamed `StateEncoder`, the four Q-heads become `ValueDecoder`s with an explicit interface boundary. Multi-quantile IQN extends the existing IQN branch — no new branch needed. +**Status (2026-04-25, before any Plan 4 task lands):** Plan 3 substrate fully landed (11/12 tasks; Task 10 Argo validation pending but not blocking). `ISV_TOTAL_DIM = 87`, `PS_STRIDE = 43`, fingerprint at ISV [85, 86]. Local smoke baseline shows healthy convergence with 4 reward-shaping signals + state-KL amplification + GPU-only replay seed. + +| Task | Scope | Adds parameters? | Difficulty | +|---|---|---|---| +| 1 — E.1 Full TFT VSN | extend `vsn_mag`/`vsn_dir` masks to all 6 feature groups | yes (per-group VSN params) | medium-heavy | +| 2 — E.2 GRN | audit-then-decide; if GRN absent in trunk, adopt; else canonicalise | conditional | medium or no-op | +| 3 — E.3 Multi-quantile IQN | extend IQN output from 1 to 5 quantiles (5/25/50/75/95) | yes (head expansion 1→5) | medium-heavy | +| 4 — E.4 Encoder-decoder separation | refactor — `StateEncoder` + 4× `ValueDecoder` with explicit interface | no | medium (cross-cutting) | +| 5 — E.5 Attention-weight interpretability | ISV-expose attention weights from Mamba2 + TLOB | no | **light — ISV-only** | +| 6 — E.6 Multi-task auxiliary heads | 2 new auxiliary heads (next-return regression + regime classification) | yes (2 small MLPs) | medium | +| 7 — Part E audit close-out | docs only; close E rows to "complete" | no | trivial | +| 8 — Plan 4 validation run | Argo L40S multi-seed × 60 epochs | no | infrastructure | + +**Architecture:** Most adoption is additive: a TFT VSN layer in front of the existing trunk, attention-weight ISV plumbing from Mamba2 + TLOB + trunk heads into new diagnostic slots tail-appended after Plan 3's slot 86, auxiliary heads branching off the trunk output via a lightweight two-layer MLP. Encoder-decoder separation is a refactor-not-rewrite: the existing trunk is renamed `StateEncoder`, the four Q-heads become `ValueDecoder`s with an explicit interface boundary. Multi-quantile IQN extends the existing IQN branch — no new branch needed. **Tech Stack:** Rust workspace + CUDA C++ via nvcc, cudarc 0.19, sccache with `CARGO_INCREMENTAL=0`. One new CUDA kernel: `vsn_feature_selection_kernel.cu`. Attention-weight plumbing reuses existing ISV/pinned-memory infrastructure. **Authority:** `docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md` §4.E. All 9 invariants apply. **Dependencies on prior plans:** -- Plan 1: `StateResetRegistry`, `AdaptiveController` trait, audit docs, named-dimension registry. +- Plan 1: `StateResetRegistry`, read-only `AdaptiveMonitor` trait (post-2026-04-24 GPU-drives revision; was `AdaptiveController`), audit docs, named-dimension registry. - Plan 2: Per-branch γ, Q-quantile atoms, Mamba2 backward wired. - Plan 3: Behavioural reward shaping + replay-seed warm-start. Multi-quantile IQN (E.3) can interact with C.4/D.4 temporal rewards. - Part E audit at `docs/ml-supervised-to-dqn-concept-audit.md` must list E.2 GRN decision (IN-adopt vs IN-canonicalise) before Task 2 starts. +**Reality reconciliation note (2026-04-25):** Plan 4 was drafted 2026-04-24 against an earlier Plan 3 design. After Plan 3 landed, several pre-plan checks reference slot names that don't exist (`PLAN_PARAMS_0_EMA_INDEX` → became `READINESS_EMA_INDEX`; `STATE_KL_THRESHOLD_EMA_INDEX` → eliminated, kernel uses internal trailing EMA; `TEMPORAL_REWARD_*_EMA_INDEX` → consolidated into rc[5] / ISV[68] `REWARD_BONUS_EMA_INDEX`). The pre-plan slot-existence check below is updated to match landed reality. Validation-doc gate softened: Argo Plan 3 Tier 1 PASS becomes `OPTIONAL` — local smoke baseline is sufficient if the user wants to proceed without burning Argo cycles for the Plan 3 gate. + --- ## Pre-plan: verify Plan 3 state -- [ ] **Step 0.1: Verify Plan 3 exit criteria satisfied** +- [ ] **Step 0.1: Verify Plan 3 substrate landed (post-2026-04-25 revision)** ```bash -# All 15 Plan 3 ISV slots must exist. +# Plan 3 ISV slots that actually landed (not the original-plan names). +# Reward-component EMAs (Task 1): for name in REWARD_POPART_EMA_INDEX REWARD_CF_EMA_INDEX REWARD_TRAIL_EMA_INDEX \ - REWARD_MICRO_EMA_INDEX REWARD_OPP_COST_EMA_INDEX REWARD_BONUS_EMA_INDEX \ - TRADE_ATTEMPT_RATE_EMA_INDEX TRADE_TARGET_RATE_INDEX \ - PLAN_PARAMS_0_EMA_INDEX \ - STATE_KL_TRAIN_VAL_EMA_INDEX STATE_KL_THRESHOLD_EMA_INDEX STATE_KL_AMPLIFICATION_INDEX \ - TEMPORAL_REWARD_PERSIST_EMA_INDEX TEMPORAL_REWARD_REGIME_SHIFT_EMA_INDEX \ - TEMPORAL_REWARD_CONSISTENCY_EMA_INDEX; do + REWARD_MICRO_EMA_INDEX REWARD_OPP_COST_EMA_INDEX REWARD_BONUS_EMA_INDEX; do grep -q "pub const $name" crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs || { - echo "Plan 3 ISV slot missing: $name"; exit 1 + echo "Plan 3 Task 1 slot missing: $name"; exit 1 } done +# Trade-attempt novelty (Task 3): +for name in TRADE_ATTEMPT_RATE_EMA_INDEX TRADE_TARGET_RATE_INDEX; do + grep -q "pub const $name" crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs || { + echo "Plan 3 Task 3 slot missing: $name"; exit 1 + } +done +# Adaptive plan-threshold (Task 4 — readiness EMA at 75; PLAN_THRESHOLD_INDEX=49 reused): +grep -q "pub const READINESS_EMA_INDEX" crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs \ + || { echo "Plan 3 Task 4 slot missing: READINESS_EMA_INDEX"; exit 1; } +# State-KL (Task 7 — note: only TRAIN_VAL + AMPLIFICATION; threshold is kernel-internal): +for name in STATE_KL_TRAIN_VAL_EMA_INDEX STATE_KL_AMPLIFICATION_INDEX; do + grep -q "pub const $name" crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs || { + echo "Plan 3 Task 7 slot missing: $name"; exit 1 + } +done +# Replay seed + CQL ramp (Task 8/9): +for name in SEED_STEPS_TARGET_INDEX SEED_STEPS_DONE_INDEX SEED_FRAC_EMA_INDEX; do + grep -q "pub const $name" crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs || { + echo "Plan 3 Task 8/9 slot missing: $name"; exit 1 + } +done +# Temporal-reward suite consolidated into rc[5] / ISV[68] REWARD_BONUS_EMA — no separate +# TEMPORAL_REWARD_PERSIST/REGIME_SHIFT/CONSISTENCY slots; this is the post-revision design. +# C.4 timing-bonus uses PS-state PEAK_PNL_BAR (slot 38) — verify PS_STRIDE grew accordingly: +grep -q "PS_STRIDE *43\|PS_STRIDE *= *43" crates/ml/src/cuda_pipeline/state_layout.cuh \ + || { echo "PS_STRIDE not at expected 43 — Tasks 5+6a+6b+6c may not all be landed"; exit 1; } +# ISV total dim should be 87 post-Tasks 8+9: +grep -q "ISV_TOTAL_DIM: usize = 87" crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs \ + || { echo "ISV_TOTAL_DIM not 87 — Plan 3 not fully landed"; exit 1; } -# Plan 3 validation doc must exist with a "Tier 2 (behavioural subset) PASS" marker. -grep -q "Tier 1.*PASS" docs/dqn-v2-plan-3-validation.md || { - echo "Plan 3 Tier 1 gate not documented as passing"; exit 1 -} - -echo "Plan 3 exit criteria satisfied. Plan 4 may begin." +echo "Plan 3 substrate verified. Plan 4 may begin." ``` +**Note on validation gate (softened):** The original plan required `docs/dqn-v2-plan-3-validation.md` with a "Tier 1 PASS" marker before Plan 4 starts. That gate was an Argo cloud deploy requirement. As of 2026-04-25 the user explicitly chose to proceed with Plan 4 without first burning Argo cycles for Plan 3 validation. Local 5-epoch multi-fold smoke baseline (`SQLX_OFFLINE=true cargo test -p ml --lib -- multi_fold_convergence --ignored`) is the de-facto gate. If a Plan 4 task regresses this smoke, fall back and investigate before continuing. + - [ ] **Step 0.2: Verify baseline compiles** ```bash