Lands the static-configuration branch of Plan 1 C.6: cql_alpha,
conviction_floor (no-op if IQL_BRANCH_SCALE_FLOOR already serves),
plan_threshold written to dedicated ISV slots at constructor. Consumer
kernels (CQL, backtest_plan, experience) read from ISV instead of
config fields / hardcoded literals.
Also pre-allocates ISV slots for the 6 upcoming GPU-kernel tasks
(atoms, gamma, kelly_cap, tau, epsilon outputs + CPU-born epoch inputs).
Those slots start at 0; GPU kernels in follow-up commits fill them.
New ISV slots:
- EPOCH_IDX_INDEX=39, TOTAL_EPOCHS_INDEX=40 (CPU-born inputs)
- EPSILON_EFF_INDEX=41, TAU_EFF_INDEX=42, GAMMA_EFF_INDEX=43,
KELLY_CAP_EFF_INDEX=44 (GPU-written in follow-up tasks)
- CQL_ALPHA_INDEX=45, PLAN_THRESHOLD_INDEX=46 (static config; this commit)
- Task 15 confirmed no-op: IQL_BRANCH_SCALE_FLOOR_INDEX=36 already
serves conviction-floor role (constructor + ISV read in iql kernel).
Layout fingerprint auto-updated via seed-byte edits; fingerprint
re-tail at [47..49). ISV_TOTAL_DIM 39 -> 49.
GpuDqnTrainConfig gains total_epochs field; fused_training.rs passes
hyperparams.epochs at construction; default 0 for smoke tests.
write_isv_signal_at bound extended from ISV_DIM(23) to ISV_TOTAL_DIM(49)
so tail slots are writable by CPU.
cql_alpha consumer: compute_cql_logit_gradients reads base from
ISV[CQL_ALPHA_INDEX] instead of config.cql_alpha; adaptive formula
(base x health x (1-regime_stability)) unchanged.
plan_threshold consumers: experience_kernels.cu (experience_state_gather,
experience_action_select, experience_env_step) and backtest_plan_kernel.cu
(backtest_plan_state_isv) read plan_thr from ISV[ISV_PLAN_THRESHOLD_IDX=46]
via isv_signals pointer already present in both kernels; null-guard defaults
to 0.5f for smoke-scale runs without ISV warmup.
ISV_PLAN_THRESHOLD_IDX=46 defined in state_layout.cuh (included by both
plan kernel files); value must match PLAN_THRESHOLD_INDEX in gpu_dqn_trainer.rs.
StateResetRegistry entries added for all new slots: SchemaContract for
TOTAL_EPOCHS/CQL_ALPHA/PLAN_THRESHOLD; FoldReset for EPOCH_IDX and
GPU-written per-fold slots.
No behavioural change: all threshold/base values remain at their prior
defaults; consumers now read adaptive values from ISV instead of baked-in
config or hardcoded literals.
Plan 1 Tasks 12, 15, 16 + pre-allocation for 9, 10, 11, 13, 14.
Spec §4.C.6 (2026-04-24 GPU-drives-CPU-reads revision).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reverts commits d76849f31 (Batch A: atoms, gamma, kelly_cap, cql_alpha)
and 4189da563 (Batch B: tau, epsilon, conviction_floor, plan_threshold).
The reverted commits implemented 8 controllers under the old
AdaptiveController trait which had CPU-side update() that computed
adaptive values and write_output() that pushed them to ISV. This
violates the architectural principle codified in the §4.C.6 revision
(commit cf36091ee): GPU kernels compute all adaptive decisions; CPU is
pure observation.
The 8 migrations will be re-implemented under the new AdaptiveMonitor
pattern:
- 6 reactive mechanisms (atoms, gamma, kelly_cap, tau, epsilon,
grad_balancer) get new GPU kernels + read-only CPU monitors.
- 3 static mechanisms (cql_alpha, conviction_floor, plan_threshold)
get ISV constructor-writes (no kernel, no monitor).
After this revert:
- AdaptiveController trait is back on main (from Task 8's 419c24b4f).
It will be replaced with AdaptiveMonitor in the next commit per the
revised Plan 1 Task 8.
- StateResetRegistry (from Task 2's b688827d6) stays intact.
- Tasks 1-7 completed work unchanged.
Tests: cargo check -p ml at 8-warning baseline after revert.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User raised that the CPU-controller pattern (where CPU computes adaptive
values from ISV and writes back to ISV) violates the architectural
principle: GPU kernels should compute adaptive decisions; CPU-side code
is pure observation.
Replaces the AdaptiveController trait with read-only AdaptiveMonitor:
- No update() — CPU doesn't compute adaptive values.
- No write_output() — CPU doesn't write adaptive values to ISV.
- read() returns the GPU-computed value from ISV.
- diagnose() emits HEALTH_DIAG snapshot.
- observe() + fire_rate() track how often the GPU-computed value changes.
Reclassifies the 9 adaptive mechanisms:
- 6 reactive get GPU kernel + CPU monitor: atoms, gamma, kelly_cap, tau
(Polyak EMA), epsilon, grad_balancer (last is already GPU-driven).
- 3 static get ISV constructor-write, no monitor: cql_alpha,
conviction_floor, plan_threshold.
New ISV slots for GPU-written adaptive outputs (EPSILON_EFF, TAU_EFF,
GAMMA_EFF, KELLY_CAP_EFF) and CPU-born inputs (EPOCH_IDX, TOTAL_EPOCHS),
plus slots for the 3 static configs.
Rationale:
- Unified adaptive machinery, no CPU-side special cases.
- Per-sample granularity available (Expected SARSA τ in c51_loss_kernel
is already the exemplar — reads ISV q_gap + health per sample).
- Zero CPU→GPU config transfer in any path.
- ISV is single source of truth for every adaptive value.
Plan 1 Tasks 8-17 restructured:
- Task 8 creates AdaptiveMonitor trait (not AdaptiveController).
- Tasks 9, 10, 11, 13, 14, 17: GPU kernel + monitor pairs.
- Tasks 12, 15, 16: static ISV writes only.
Follow-up commits will revert Batch A (d76849f31) and Batch B (4189da563)
which implemented the old CPU-compute pattern, then re-implement under
this design.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Unified protocol for every adaptive controller in the DQN trainer.
FireRateStats for controller_activity smoke. DiagSnapshot for
standardised HEALTH_DIAG emission. IsvBus abstraction over pinned
device-mapped memory (&mut [f32] wrapper).
No concrete controller migration yet — Tasks 9-17 migrate existing
controllers (atoms → gamma → Kelly → cql_alpha → tau → epsilon →
conviction_floor → plan_threshold → balancer) one per sub-commit.
Old scaffolding for each controller is removed in the SAME commit
that migrates its last consumer to the new protocol, per
feedback_no_partial_refactor.md.
Tests: 2 unit tests on FireRateStats and DiagSnapshot.
Plan 1 Task 8. Spec §4.C.6.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Enumerated every DtoH/HtoD/memcpy call in the DQN hot path
(run_full_step → child graphs). Classified 55 call sites:
31 OK-pinned, 20 COLD-PATH, 4 MIGRATED.
Fix 1 (gpu_dqn_trainer.rs): removed dead cuMemcpyDtoHAsync_v2 in
run_causal_intervention_unconditional — result (causal_mean_scratch)
was never consumed; now stays on device.
Fix 2 (fused_training.rs + training_loop.rs): compute_iqr() was
called inside submit_aux_ops (captured aux_child graph). Its sync
cuMemcpyDtoH_v2 cannot be graph-captured — silently ran only during
capture, then HtoD replayed stale IQR data on every step. Removed
from submit_aux_ops; added refresh_iqn_iqr() called once per epoch
in process_epoch_boundary.
Fix 3 (gpu_iqn_head.rs): tau_buf (CudaSlice<f32>) + tau_host (f32) +
cuMemcpyHtoDAsync_v2 each step replaced by tau_pinned (*mut f32) +
tau_dev_ptr (u64) via cuMemAllocHost_v2(DEVICEMAP) +
cuMemHostGetDevicePointer_v2. CPU writes *tau_pinned = tau; EMA
kernel reads via tau_dev_ptr — zero PCIe overhead. Drop updated.
Audit table populated in docs/dqn-gpu-hot-path-audit.md.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Populated docs/dqn-wire-up-audit.md with every pub module and CUDA
kernel in the DQN path. Each entry classified Wired / Partial / Orphan /
Ghost / OUT-of-DQN-scope with the action plan linking to the plan+task
that resolves any non-Wired status.
No Orphan left unclassified. Orphans fall into three buckets:
1. Scheduled for wiring by a later Plan (gpu_statistics → Plan 2 D.2;
tlob_loader → Plan 2 D.8).
2. OUT-of-DQN-scope because supervised consumers exist (PPO kernels,
xLSTM, KAN trainable adapter, flash_attention, benchmarks).
3. Genuinely unused — escalated to user review in the task output,
not deleted autonomously (streaming_dbn_loader, unified_data_loader,
training/orchestrator, training_pipeline, inference_validator,
model_loader_integration, paper_trading/mod.rs,
portfolio_transformer, regime_detection/mod.rs).
Summary: 109 total modules/kernels, 74 wired, 7 partial, 11 orphan,
0 ghost, 17 OUT-of-DQN-scope.
Plan 1 Task 6. Spec §4.A.5, Invariant 2.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Implements spec §4.A.2 structural layout fingerprint with tail placement
rather than head placement (spec alternative: §4.A.2 Step 5.3 alt).
Head placement (ISV[0..2)) was rejected because isv_signals[0] and [1]
are actively written by the isv_signal_update kernel (Q-drift EMA and
gradient-norm EMA). Shifting those would require updating every literal
reference in experience_kernels.cu — a larger change than warranted for
pure contract enforcement. Tail placement leaves all existing indices
intact, touches zero kernel .cu files, and fulfils the same design contract.
Key changes:
- ISV_LAYOUT_FINGERPRINT_LO_INDEX = 37, HI_INDEX = 38 (u64 across 2×f32).
- LAYOUT_FINGERPRINT_CURRENT: u64 = FNV-1a of slot-list seed bytes.
Value: 0x85d4d76b578a7c17. Any slot change updates seed bytes,
which updates the hash automatically.
- Constructor writes fingerprint after zero-init; calls
check_layout_fingerprint() to self-verify before returning.
- check_layout_fingerprint(): reads pinned slots [37..39), recomposes u64,
fails-fast on mismatch with "retrain required" message.
- Error message does NOT mention migration as an option.
- Pre-commit hook rejects `fn migrate_isv|upgrade_isv` names — makes the
no-migration rule structurally enforced (check_no_isv_migrations).
- ISV_TOTAL_DIM: 37 → 39.
- Zero existing index shifts (no kernel literal sites affected).
- StateResetRegistry entry renamed ISV_SCHEMA_VERSION → ISV_LAYOUT_FINGERPRINT.
- ResetCategory::SchemaContract docstring updated to remove "migration" framing.
- docs/isv-slots.md: updated table + design note for tail placement.
Tests: state_reset_registry 3 unit tests pass with renamed entry.
cargo check -p ml clean (pre-existing warnings only).
Plan 1 Task 5. Spec §4.A.2 (tail-placement alternative).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User raised a backward-compat concern: integer "schema version"
semantically implies a family of coexisting versions with upgrade
paths between them, inviting the forbidden pattern
(feedback_no_legacy_aliases.md, feedback_no_partial_refactor.md).
Replace with a compile-time structural fingerprint:
- ISV[0..2) stores a u64 FNV-1a hash of the slot layout (split across
two f32 lanes preserving raw bits).
- The hash is computed by a const fn over the slot list; any slot
change automatically updates the fingerprint. No human decides
"what version is this now".
- Checkpoint load is fail-fast only. Error message does NOT mention
migration as an option.
- Pre-commit hook rejects any `fn migrate_isv|upgrade_isv` to make
the no-migration rule structurally enforced (landed as part of
Plan 1 Task 5 hook extension).
- StateResetRegistry entry renamed ISV_SCHEMA_VERSION →
ISV_LAYOUT_FINGERPRINT (Task 2's landed code touched in Task 5's
commit per no-partial-refactor).
Updated:
- spec §4.A.2 (fingerprint design + rationale)
- spec §5 landing-order note (fingerprint auto-updates on layout change)
- Plan 1 architecture line
- Plan 1 Task 2 StateResetRegistry test + entry naming
- Plan 1 Task 5 — full rewrite of implementation steps
- Plan 1 exit criteria #6
- Plan 2 pre-plan verification (grep fingerprint constants, not version==0)
- Plan 2 Task 6D.1 (update fingerprint seed, not bump version)
- Plan 2 Task 6D exit criteria #7
- Plan 3 pre-plan verification
- docs/isv-slots.md ISV[0..2) row
No code changed. Plan 1 Task 5 is not yet implemented — this commit
realigns the spec + plans so the implementer subagent works from the
corrected design.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Define MAG_QUARTER=0, MAG_HALF=1, MAG_FULL=2, NUM_MAGNITUDES=3 in
state_layout.cuh. Migrate the one unambiguous semantic comparison in
trade_physics.cuh (compute_target_position_4branch magnitude scaling).
Arithmetic operations on mag_idx (mag_idx±1, 2-mag_idx, mag_idx<2) are
runtime values, not semantic comparisons, and are not migrated.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Define DIR_SHORT=0, DIR_HOLD=1, DIR_LONG=2, DIR_FLAT=3, NUM_DIRECTIONS=4 in
state_layout.cuh. Migrate all literal dir_idx/raw_dir comparisons to named
constants across experience_kernels.cu (15 sites), trade_physics.cuh (3 sites),
and backtest_metrics_kernel.cu (2 sites). Raw integer comparisons (== 0/1/2/3)
eliminated from all direction-branch consumers.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add BRANCH_DIR/BRANCH_MAG/BRANCH_ORD/BRANCH_URG/NUM_BRANCHES to
state_layout.cuh. Migrate the 4 clear literal branch-index accesses
in branch_grad_balance_kernel.cu (branch_norms_dev[0..3] in
grad_balance_isv_update).
Other branch-keyed arrays (branch_starts, branch_lens, branch_norms,
branch_scales) use the runtime `branch = blockIdx.y` variable or loop
counter — no raw literal index, so no migration needed. Rust call sites
use struct fields (branch_0_size etc.) or loop vars — not literals.
No behavioural change; pure refactor.
Plan 1 Task 4D. Spec §3 Invariant 8.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add PLAN_PARAM_* constants to state_layout.cuh and replace raw
plan_params/pp slot accesses in experience_kernels.cu and
backtest_plan_kernel.cu.
Sites migrated: pp[0..5] in both files (12 sites), plan_params_ptr
indexed accesses at slots 0, 1, 4 in experience_kernels.cu (4 sites),
plan_params[w*6+4] in backtest_plan_kernel.cu (1 site). Loop variable
pp[p] in noisy-plan path left as-is (p is a runtime loop counter, not
a literal slot index).
No behavioural change; pure refactor.
Plan 1 Task 4C. Spec §3 Invariant 8.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add PLAN_ISV_* constants to state_layout.cuh and replace raw
plan_isv[N] and pisv[N] accesses in experience_kernels.cu and
backtest_plan_kernel.cu. backtest_plan_kernel.cu gains an
#include "state_layout.cuh" so the constants are visible.
6 code sites migrated (plan_isv[0..5] in experience_kernels.cu),
plus 6 pisv[N] local-array accesses in backtest_plan_kernel.cu
(same semantic slots).
No behavioural change; pure refactor.
Plan 1 Task 4B. Spec §3 Invariant 8.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace raw ps[N] access across all CUDA kernels with PS_* named
constants defined in state_layout.cuh. 32 constants total covering
the full 38-slot portfolio state: position/cash/value, DSR stats,
Kelly stats, plan fields, and OFI scratch range.
Notable deviations from pre-written plan mapping: the plan's
PS_VALUE/PS_POSITION/PS_CASH values (0,1,2) had the wrong semantics.
Code wins (feedback_trust_code_not_docs): ps[0]=position, ps[1]=cash,
ps[2]=portfolio_value. All 148 raw ps[digit] accesses in
experience_kernels.cu and trade_stats_kernel.cu migrated. In-comment
range references (e.g. "ps[30..37]") left as documentation.
trade_stats_kernel.cu: migrated base+14 offset to PS_KELLY_WIN_COUNT.
docs/dqn-named-dims.md: PS_* table corrected to match actual code layout.
No behavioural change; pure refactor.
Plan 1 Task 4A. Spec §3 Invariant 8.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the ad-hoc scattered fold-boundary reset calls with a single
registry-driven iteration. Adding new fold-reset state now requires
adding a registry entry AND a dispatch arm in reset_named_state in the
same commit (Invariant 2 Wire-It-Up).
Step 3.4 correction: plan_state entry removed from the registry —
plan_state_buf exists only in GpuBacktestEvaluator (val path), not in
the training-path fused ctx. No training-side fold reset is applicable.
New behaviour: isv_learning_health, isv_sharpe_ema, isv_q_means are
now properly reset to baseline at each fold boundary (previously unset,
which allowed signals from fold N to bias fold N+1 initialisation).
Tests: 3 registry unit tests pass; cargo check -p ml clean (8 pre-existing
warnings only, no new).
Authority: spec §4.A.1. Plan 1 Task 3.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Typed classification of every piece of training state by reset
lifecycle: FoldReset, WindowReset, SoftReset(decay_bars),
TrainingPersist, SchemaContract.
Replaces the previous vibes-driven fold-boundary reset logic with an
explicit registry. Adding new state requires adding an entry in the
same commit (Invariant 2 Wire-It-Up).
Consumer wiring (reset_fold_state, reset_window_state helpers that
iterate the registry) follows in Task 3 of Plan 1.
Tests: 3 unit tests covering category lookup, fold-reset filtering,
unknown-name handling.
Authority: docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md §4.A.1
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Plan 1 Task 1. Creates the five audit docs plus config/metric-bands.toml
that track Invariants 2, 7, 8 per the DQN v2 spec, and extends the
pre-commit hook with two checks:
- component-adding commits must touch an audit doc (Invariant 7)
- added code may not contain TODO/FIXME/XXX/HACK/TBD/unimplemented!/
todo! markers (Invariant 9)
Tests: manually verified by staging a TODO-marked file; commit
rejected with the correct error message.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Self-review caught that the CqlAlphaSeedCoupledController read_signals
example used unimplemented!("plumbed by caller"), which violates the very
Invariant 9 the plan enforces. Replaced with a concrete implementation
that reads SEED_FRACTION_EMA_INDEX from the ISV bus (a Plan 1 reserved
controller-signal slot).
No other plan had placeholder violations. The remaining TBD/TODO/FIXME
grep hits across Plans 1-5 are either the pre-commit hook's rejection
patterns (correctly quoted) or enforcement scans that DETECT the markers
in audit docs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fifth and final plan decomposing the DQN v2 unified spec
(docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md §2, §4.A.3,
§4.A.4, §4.A.4.1). No new policy mechanisms — orchestration, automation,
regression detection, and the final sign-off run.
Covers spec sections:
- §4.A.3 Multi-seed × multi-fold Argo harness (N=5, K=6 defaults via
--multi-seed / --folds flags on argo-train.sh)
- §4.A.4 Regression detection hard-stop (2N consecutive error-band
metrics → self-SIGTERM with TerminationReason)
- §4.A.4.1 nsys profile harness with --profile flag and
compare-nsys-profiles.py for 20% per-epoch regression detection
- §2 Tier 1 + Tier 2 + Tier 3 final exit criteria via scripts/validation/
suite (tiered tier1/tier2/tier3 check scripts + wrapper)
- §2 DQN v2 rollout close-out (spec marked LANDED, retrospective,
plan-level validation docs archived)
Plan structure: 6 tasks — 4 infrastructure (harness, regression-detect,
nsys, validation-scripts) + final-run task + rollout-closeout task. The
final-run task is the sole exit-gate for the DQN v2 rollout: ALL THREE
tiers must PASS on the first final-validation run (per stop-on-anomaly:
retrying with different seeds hoping for a different outcome is anti-
pattern).
Preserves all 9 invariants. Every mechanism wired, no stubs, no TODO/FIXME.
Retrospective written from actual implementation experience after Plans
1-5 all land.
This is the terminal plan. Plans 1-5 together constitute the complete
DQN v2 rollout. After Plan 5's final-run exit passes, the spec is marked
LANDED and the rollout is declared complete.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fourth of five sequential plans decomposing the DQN v2 unified spec
(docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md §4.E).
Covers the six Part E IN decisions:
- §4.E.1 TFT Variable Selection Network across 6 feature groups
(market/OFI/TLOB/MTF/portfolio/plan_isv) with vsn_feature_selection kernel
- §4.E.2 Gated Residual Network — CONDITIONAL branch (ADOPT vs CANONICALISE)
based on docs/ml-supervised-to-dqn-concept-audit.md row
- §4.E.3 Multi-quantile IQN heads (5/25/50/75/95) replacing single CVaR output
- §4.E.4 Encoder-Decoder separation — explicit StateEncoder + per-branch
ValueDecoder with D.3 horizon-decomposed V_short/V_long sub-heads
- §4.E.5 Attention-weight interpretability — 7 new ISV slots [65..72) for
per-group attention focus EMAs + Mamba2 retention proxy
- §4.E.6 Multi-task auxiliary heads — next-bar return MSE + 5-bar regime CE,
ISV-coupled aux-weight schedule (sharpe-reactive)
ISV_TOTAL_DIM seals at 72 with this plan's allocations. Part E audit doc
closes out all rows (zero TBD/evaluate remain per Invariant 9).
Plan structure: 8 tasks (6 feature tasks + audit close-out + validation).
TDD-disciplined steps. Task 2 documents the CONDITIONAL branch decision
pathway explicitly (ADOPT vs CANONICALISE Branch A/B structure).
Plan 4 exit gate: Tier 1 convergence RETAINED (no regression vs Plan 3
baseline) + aux heads produce measurable signal. Tier 2 + Tier 3 checked
by Plan 5.
Preserves all 9 invariants. Zero stubs. Zero TODO/FIXME. Every Part E item
either lands fully or is explicit OUT (xLSTM, KAN); no third path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Second of five sequential plans decomposing the DQN v2 unified spec
(docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md).
Covers spec sections:
- §4.C.1 Quantile-based atom support (8 new ISV slots, q_quantile_reduce kernel)
- §4.D.1 Mamba2 backward pipeline completion (no atomicAdd — per-sample arrays + host reduce)
- §4.D.2 Per-branch gamma via AdaptiveController (4 new ISV slots)
- §4.D.5 Soft fold-boundary transitions (extends StateResetRegistry with SoftReset category)
- §4.D.7 Liquid Time-constant audit (trace fire rate, decide wire-or-delete)
- §4.D.3 + §4.D.6 + §4.D.8 coordinated state-layout migration (horizon-decomposed V +
plan_isv[6] + TLOB integration with atomic ISV schema version bump 1→2)
Plan structure: 7 tasks + pre-plan verification, all TDD-disciplined with
bite-sized steps (test → fail → implement → pass → commit). Task 6 is
explicitly atomic to preserve Invariant 5 (state-layout consistency).
Plan 2 prerequisites: Plan 1 landed (StateResetRegistry, AdaptiveController
trait, ISV schema version, audit docs). Plan 2 exit: all 9 invariants
preserved; convergence-scaffolding run passes with 16 new ISV slots
populated; ISV schema version == 2.
Preserves all 9 invariants. No stubs. No TODO/FIXME. Every new module
wired to production path in the same task it lands in.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First of 5 implementation plans for the DQN v2 unified integrated
policy system spec (d13b53586, 336ee40b9). Plan 1 covers:
Task 1: Audit doc scaffolding + pre-commit hook (Invariant 7, 9)
Task 2: StateResetRegistry definition (A.1)
Task 3: Wire StateResetRegistry into fold-boundary reset (A.1)
Task 4: Named-dimension refactor — ps[], plan_isv[], plan_params[],
branch indices, dir/mag sub-indices (Invariant 8)
Task 5: ISV schema version at ISV[0], fail-fast on checkpoint
mismatch (A.2)
Task 6: Orphan audit populated (A.5)
Task 7: Hot-path purity audit populated; MIGRATE calls fixed (A.6)
Task 8: AdaptiveController trait + harness (C.6)
Tasks 9–17: Migrate each adaptive controller to the trait in spec-
specified order (atoms → gamma → Kelly → cql_alpha → tau
→ epsilon → conviction_floor → plan_threshold → balancer)
Task 18: Plan 1 validation run (smoke + 3-epoch L40S)
Plan 1 landing criteria: all 9 invariants preserved across every
commit, all smoke tests pass, audit docs fully populated. Plan 2
(temporal core) starts only after Plan 1 passes exit criteria.
Plans 2–5 cover Parts B, C (minus C.6 already in Plan 1), D, E, and
final validation. Planned as separate documents in docs/superpowers/
plans/2026-04-24-dqn-v2-plan-{2..5}-*.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User-mandated addition:
Invariant 9 — No deferred work, no stubs, no TODO/FIXME
Every commit lands complete. No stubs (placeholder return values), no
TODO/FIXME/XXX/HACK/TBD markers, no half-finished implementations. If
it can't finish in this commit, it doesn't start. Stubs and deferrals
train the network on semantic emptiness — invisible in convergence
metrics but burn GPU time optimising against partly-fake signal.
Authority: feedback_no_stubs.md, feedback_no_todo_fixme.md,
feedback_no_quickfixes.md elevated to first-class spec enforcement.
Pre-commit hook greps for forbidden markers.
Part E decisions made concrete (per Invariant 9):
- E.1 TFT VSN: IN (full VSN extension)
- E.2 GRN: IN if A.5 audit finds absent (otherwise mark existing as canonical)
- E.3 Multi-quantile heads: IN (5/25/50/75/95 quantile decomposition)
- E.4 Encoder-decoder: IN (extends D.3 to full trunk/head separation)
- E.5 Interpretability: IN (attention-weight ISV diagnostics)
- E.6 Auxiliary heads: IN (next-return + regime-classification)
- xLSTM: OUT (redundant with Mamba2+TLOB; YAGNI)
- KAN: OUT (function approx not a bottleneck)
No "evaluate later" items. Every concept either lands or explicitly
doesn't, with rationale.
§8 decisions tightened:
- D.8 TLOB mode: decision criterion = per-step cost benchmark ≤ 5ms
- C.6 controller order: fixed in spec (atoms→gamma→Kelly→cql_alpha→
tau→epsilon→conviction_floor→plan_threshold→balancer)
- A.3 resource plan: auto-switch L40S→H100 at 12 GPU-hour threshold
- A.4 metric band init: [mean − 3σ, mean + 3σ] from last 3 good runs
No discretionary "we'll see" remain. All decisions data/order/budget/
statistic-driven.
Counter updates: "seven invariants" → "nine" in §3 header and §5
cleanup contract.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Self-review pass on the DQN v2 unified design spec (d13b53586). Fixes
applied inline:
Ambiguity fixes:
- §3 Invariant 3: explicit definition of "training step loop" (per-step
code in fused_training::step_fused and captured graph children; NOT
per-epoch / per-fold / per-checkpoint code).
- §2 Tier 1: "stable epochs" defined as epochs [warmup_end..run_end)
with warmup_end from B.3's seed-phase decay or explicit flag.
- §4 A.2: migration mechanism clarified as fail-fast initially; helpers
added only when a specific migration need arises (no speculative
scaffolding).
- §4 A.3: N=5, K=6 stated as DEFAULTS with ≥ MINIMUMS per Invariant 6;
resource budget flagged (~8-10 GPU-hours per validation pass).
- §4 B.3: "≥ 100K experiences" (minimum), warm-restart clarified as
runtime condition not feature flag.
- §4 C.3: adaptive KL threshold mechanism made explicit (second ISV
slot for threshold EMA, third for amplification multiplier).
- §4 C.6: cleanup timing explicit — old scaffolding removed in the SAME
commit that migrates its last consumer (no deferred pass).
- §4 D.1: DQN-path-specific scope clarified; validation via grad-norm
smoke + integration test.
- §4 D.6: plan_isv index naming made consistent with existing layout;
coordinated state-layout migration commit called out.
New Invariant 8 — Named dimensions, not indices:
Every dimension, slot, offset, or semantic position in a multi-field
buffer has a named constant. Raw numeric indices (`[0]`, `[6]`, `[23]`)
appear ONLY in the definition site. Named constants specified for ISV
slots (existing), portfolio state ps[0..30), plan_isv[0..7), plan_params
[0..6), state-vector offsets, branch indices (BRANCH_DIR/MAG/ORD/URG),
action sub-indices (DIR_SHORT/HOLD/LONG/FLAT, MAG_QUARTER/HALF/FULL).
Enforcement: audit pass during A.1/A.2, lint via grep for raw-index
access outside definition modules.
Landing order revision:
- Dependency graph explicit (A.2 before new ISV slots, D.1 before
D.2-4-8, state-layout changes in ONE coordinated commit).
- Spec decomposition note added — writing-plans may produce multiple
sequential plans rather than one monolithic plan.
New doc tracked by pre-commit: docs/dqn-named-dims.md (Invariant 8).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Eliminates the f64→f32 cudarc ABI trap (feedback_cudarc_f64_f32_abi.md,
task #82) at the type level: hyperparameters consumed by CUDA kernels
now live as f32 in Rust, cast once at the TOML/PSO ingest boundary
instead of at every kernel call site.
Structs changed:
- DQNHyperparameters (crates/ml/src/trainers/dqn/config.rs) —
~85 scalar fields migrated from f64 → f32. Covers all
kernel-facing scalars: reward weights (w_pnl/w_dd/w_idle,
dd_threshold, cea_weight, micro_reward_*, price_confirm_weight,
book_aggression_weight, hold_quality_weight), exploration
(epsilon_* and the 4 branch mults, noisy_sigma_*, count_bonus,
noise_sigma, q_gap_threshold), distributional RL (v_min, v_max,
reward_scale, iqn_lambda, qr_kappa, spectral_*,
gradient_collapse_multiplier), fill simulation (5 fill_*
fields), risk/Kelly (kelly_fractional, kelly_max_fraction,
max_leverage, max_position_absolute, minimum_profit_factor),
ensemble/curiosity (curiosity_weight,
curiosity_q_penalty_lambda, ensemble_*, beta_*, variance_cap),
anti-LR (anti_lr_*, adversarial_dd_threshold,
beta_penalty_strength), walk-forward (wf_*),
experience (avg_spread, transaction_cost_multiplier,
holding_cost_rate, churn_penalty_scale, contract_multiplier,
margin_pct, tick_size, bars_per_day, cash_reserve_percent),
misc kernel scalars (gamma, tau, huber_delta, q_clip_*,
shrink_perturb_*, regime_replay_decay, per_alpha,
per_beta_start, dt_target_return, etc.). Also
`noisy_epsilon_floor: Option<f32>` and
`count_bonus_coefficient: Option<f32>`.
- `computed_v_min` / `computed_v_max` now return f32.
- `compute_max_position` returns f32 (f64 internally for the
notional division).
Fields preserved as f64 (precision-sensitive, NOT kernel-facing
scalars — per task spec and feedback_cudarc_f64_f32_abi.md):
- `learning_rate` — tested at 1e-10 tolerance; f32 rounds to
2e-12 for a 1e-5 LR.
- `entropy_coefficient` — tested at 1e-9 tolerance.
- `weight_decay` — tiny 1e-5..1e-3 range.
- `adam_epsilon` — 1e-8 default; f32 preserves denorms here but
paired with weight_decay/learning_rate for symmetry.
- `gradient_clip_norm: Option<f64>` — grad norms are f64
accumulators by project convention.
- `min_loss_improvement_pct`, `q_value_floor` — early-stopping
long-horizon stats.
- `cql_alpha` — flows into DQNConfig (ml-dqn) still f64.
- `min_learning_rate`, `lr_min` — paired with learning_rate.
- Family intensity scalars (6 `*_intensity` fields) — f64 PSO
search space; the intensity applies via `as f32` at each call
site in `apply_family_scaling`.
No checkpoint format change: DQNHyperparameters has
`#[derive(Debug, Clone)]` only (not Serialize/Deserialize), so the
TOML-ingest path is the only serde boundary and already casts
explicitly via `hp.field = v as f32;` in
`DqnTrainingProfile::apply_to`. PSO hyperopt bounds stay f64 in
`SearchSpaceSection` and cast at the adapter boundary.
Call-site impact:
- ~30 `as f32` casts removed from hot paths (fused_training
FusedConfig builder, training_loop kernel launches,
constructor DQNConfig builder, action.rs GPU action selector,
trainer/mod.rs WF config). Kernels now receive the hp field
directly via `&hp.x`.
- ~75 `as f32` casts added at the `apply_to` / hyperopt-adapter
ingest boundary — the single conversion point.
- Cross-crate contracts (DQNConfig in ml-dqn, PortfolioTracker
in ml-core, GAECalculator, DropoutScheduler, NoisySigmaScheduler,
KellyOptimizerConfig, RewardConfig) retain their f64 signatures;
ml calls cast at the boundary with `f64::from(hp.x)` so the
contract is explicit and greppable.
Two test-side adjustments:
- `test_kelly_fields_are_public` now asserts `f32` for
kelly_fractional / kelly_max_fraction (these migrated).
- `test_early_stopping_termination` uses `f64::from(...)` to
preserve the f64 threshold computation against the now-f32
`gradient_collapse_multiplier`.
Verified:
- `SQLX_OFFLINE=true CARGO_INCREMENTAL=0 RUSTC_WRAPPER=sccache
cargo check --workspace` — clean, zero new warnings.
- `cargo check --workspace --tests` — clean.
- `cargo test -p ml --lib training_profile::tests` — all 18
migrated tests pass. The one pre-existing failure
(`test_production_profile_applies_all_sections` n_steps
mismatch, 1 vs 5) reproduces on stashed baseline, so it is
unrelated to this change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Final commit toward task #94 val plan_isv parity. Wires the plan
machinery inside the chunked val backtest loop so state positions
[86..92) carry real plan signals matching training instead of zero-
fill. This is the structural distribution-shift fix behind the
observed val trade count of 21 / 214,654 bars (0.0098%).
Phase 6 inside `submit_dqn_step_loop_cublas`, per chunk, after env
step advances portfolio:
1. Forward on the LAST-STEP N rows of chunked_states — a targeted
`compute_q_values_to(last_step_states, N, scratch_q)` rewrites
`save_h_s2` to exactly [N, SH2] for the final step. Needed
because the batched `compute_q_values_to` for N*chunk_len rows
leaves save_h_s2 at an arbitrary last-sub-batch slice.
2. `compute_plan_params(plan_params_buf, N)` runs the trade-plan
MLP on that clean save_h_s2, producing [N, 6] plan_params.
3. `backtest_plan_state_isv` — Flat↔Positioned activation /
deactivation on plan_state[N, 7] and writes plan_isv_buf[N, 6]
consumed by the next chunk's first `launch_gather` for state
positions [86..92).
ch_q_values is reused as the Q-scratch for step (1) — its original
chunk Q-values were already consumed by Phase 4 action-select.
Epoch-boundary diagnostic `launch_plan_diag_and_log` after the step
loop: launches `backtest_plan_diag_reduce` (1-block, 256 threads),
DtoH-copies the 8-float summary, syncs, and emits a single
`tracing::info!(target: "val_plan_diag", ...)` line with the six
labelled plan_isv means plus active_frac / n_active.
Scalars passed with exact i32 types (current_step, max_len,
feat_dim, n_windows) per feedback_cudarc_f64_f32_abi. Kernels
pulled via `plan_state_isv_kernel.as_ref().ok_or_else(...)` — they
are loaded in `ensure_action_select_ready` on first eval.
Not captured inside a CUDA Graph: the backtest step loop uses direct
kernel launches (evaluator comment: graphs SIGSEGV on 500K+ launches).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Second commit toward task #94 val plan_isv parity. Adds structural
wiring in GpuBacktestEvaluator:
- plan_params_buf [N, 6] : live plan MLP output per step
- plan_state_buf [N, 7] : persistent stored plan across bars
- plan_diag_buf [8] : epoch-end diagnostic reduction output
- plan_state_isv_kernel : lazy-loaded CudaFunction from
backtest_plan_kernel.cubin
- plan_diag_reduce_kernel : diagnostic reduction kernel
Buffers allocated zero-initialised in the constructor; kernels loaded
alongside the action_select and scatter_intent kernels in
`ensure_action_select_ready` (same module-load pattern).
Remaining work (follow-up commit):
- Wire QValueProvider::compute_plan_params into the chunked loop to
populate plan_params_buf from the trainer's save_h_s2 after each
forward pass.
- Launch backtest_plan_state_isv after the env_batch_kernel each
chunk to update plan_state_buf and plan_isv_buf for the next
chunk's state gather.
- Launch backtest_plan_diag_reduce + DtoH of plan_diag_buf at epoch
boundary for HEALTH_DIAG emission.
The chunked batch layout (N*chunk_len samples per forward pass)
requires careful slicing to extract the last-step's plan_params for
the plan_state update — handled in the follow-up commit to avoid
rushing a subtle integration.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Val-Flat-collapse fix#5 infrastructure (task #94, 2026-04-24). The
val backtest's `plan_isv_buf` was zero-filled, causing state positions
[86..92) to be OOD relative to training and biasing val argmax toward
Flat/Hold on 99.99% of bars. Research-agent audit confirmed this is a
documented architectural gap, not a bug per se: val's portfolio state
has no plan slots (8 vs training's 30+), so there was no mechanism to
carry plan signals across bars in val.
This commit lays the foundation for closing that gap:
1. `backtest_plan_kernel.cu` (new) — two kernels:
- `backtest_plan_state_isv`: per-window Flat→Positioned plan
activation + plan deactivation on return to Flat + plan_isv[0..5]
computation matching training's formula at
experience_kernels.cu:596-619. Reads live plan_params from the
plan MLP, writes plan_state (persistent across bars) and
plan_isv_out. Extracts raw_close from features buffer for
unrealized P&L ratios.
- `backtest_plan_diag_reduce`: single-block reduction emitting 8
diagnostic floats for plan activity logging (active fraction,
per-slot mean magnitudes, raw active count).
2. `QValueProvider::compute_plan_params` trait method — allows the val
evaluator to run the plan MLP on the trainer's most recent trunk
hidden state (save_h_s2), filling plan_params[N, 6] for use in the
state-update kernel.
3. `FusedTrainingQValueProvider::compute_plan_params` impl — runs
`launch_trade_plan_forward` then DtoD-copies `plan_params_buf` to
the caller's output in the same chunk-loop pattern as
`compute_q_values_to`.
4. build.rs — register `backtest_plan_kernel.cu` for compilation.
Next commit will wire these into `GpuBacktestEvaluator`:
- Add `plan_params_buf[N, 6]`, `plan_state_buf[N, 7]`,
`plan_diag_buf[8]` device allocations.
- Load the two kernels into CudaFunction slots.
- Call sequence per step: `compute_q_values_to` →
`compute_plan_params` → action_select → env_step →
`backtest_plan_state_isv` (reads updated portfolio, writes
plan_state + plan_isv_buf for next step).
- Epoch boundary: `backtest_plan_diag_reduce` + DtoH + HEALTH_DIAG
emission.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Val-Flat-collapse fix#3 revision (task #94, 2026-04-24). Prior
commit 543e3c11b used `-0.5 × holding_cost_rate × vol_proxy` — the
0.5 is a hardcoded tuned constant violating
`feedback_isv_for_adaptive_bounds.md` and
`feedback_adaptive_not_tuned.md`.
Replace with ISV-driven per-sample conviction, already computed by
the action-select kernel and threaded into env_step via
`conviction_ptr → conviction_core`:
reward_flat = -shaping_scale * holding_cost_rate
* conviction_core * vol_proxy_flat
`conviction_core ∈ [0, 1]` = direction-branch Q-range normalised by
`ISV[21]` (q_dir_abs_ref EMA). Self-adapting properties:
- Cold start / ISV[21] uninitialised → fallback 1.0 → full penalty,
encourages early exploration out of the flat equilibrium.
- Low conviction (uncertain direction) → penalty scales toward 0
→ doing nothing is acceptable when there's no signal (matches
real-world: flat cost is only real when there's opportunity).
- High conviction (strong directional edge) → penalty scales up
→ Flat becomes expensive ONLY where the model itself says
there's an edge. Forces the policy to take action exactly
where it has belief, not blindly.
Continuity: conviction is continuous ∈ [0, 1], no step function. The
temporal Mamba2 layers in the trunk feed the Q-values that drive
conviction, so conviction inherits temporal history — the model can
learn "market has been signalling for N bars, time to try" without
an explicit time-since-last-trade feature.
Training-time exploration (Boltzmann sampling, epsilon floor 2%,
NoisyNets σ, count bonus) is unchanged. The Flat-cost shifts the
learned Q-values so that deterministic val argmax picks trade-
actions where the training-time exploration already found edge.
Hold semantics retained:
- Hold while position≠0 (in-trade stance) → positioned-bar
holding-cost branch (unchanged)
- Hold while position=0 AND Flat (no-op outcomes) → this branch
→ conviction-scaled opportunity cost.
The distinction is structural by portfolio state, not by dir label.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Val-Flat-collapse fix#4 (task #94, 2026-04-24). Research-agent audit
of train-bv2n5 identified a positive-feedback loop driving atom
utilisation to 1%:
util < 0.4 → γ -= 0.01 (toward 0.90 floor)
γ × delta_z shrinks → TD targets concentrate on fewer bins
→ more collapse → util lower → γ lower ...
The original controller had the right structure but the wrong
direction on the low-util branch. Flipping the sign (`-=` → `+=`)
breaks the loop: when atoms collapse, raising γ widens the effective
TD-target support `γ × atom_support`, pushing targets across more
bins of the C51 grid and giving the network more distributional
signal to learn from.
Evidence from train-bv2n5 (pre-fix): atom_util stuck at 0.01 for 16
epochs; γ slowly drifted toward 0.90. No mechanism recovered.
Change localised to a single `else if util < 0.4` branch at
`training_loop.rs:660-675`. Same step size (0.005) as the healthy
branch avoids overshoot on the recovery path. Clamps retained
[0.90, 0.95] at this layer; the fused trainer re-clamps to
[0.90, 0.995] after regime adjustment downstream.
Other pending atom-util fixes from the audit (deferred pending
validation):
- Atom-entropy regularisation term in C51 loss (medium risk)
- Absolute v_half floor in update_eval_v_range (low risk)
- TD-target dithering before Bellman projection (low risk)
This single-line fix addresses the dominant mechanism — research
agent labelled it "lowest risk, highest leverage". If util remains
low after this, the others stack additively.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Val-Flat-collapse fix#3 (task #94, 2026-04-24). Research agent
traced the dominant val failure mode: reward = 0.0 for Flat bars
produces structural Flat-beats-trading-actions in the argmax —
Flat accumulates Q ≈ 0 while CQL pulls trading-action Q's toward
pessimistic negative estimates. Result: val argmax picks Flat for
~all states (22 trades/epoch vs 30K+ in training via Boltzmann).
Prior comment defended the design: "No signal is correct — the
model should not be rewarded or penalized for correctly staying
flat when there's no edge." That reasoning was correct in isolation
but broken in context: the trading-action branch is penalized by
CQL's lower-bound estimator anyway, so Flat's neutral Q becomes the
de-facto upper bound on all action Q-values → Flat wins argmax
regardless of the policy's actual directional belief.
Add a per-bar opportunity cost for Flat bars, proportional to
realised volatility (ATR fraction):
reward = -shaping_scale * holding_cost_rate * 0.5 * vol_proxy
This:
- Symmetric to the positioned-bar holding penalty (positioned
pays `holding × |pos|`; Flat pays `holding × 0.5 × vol_proxy`).
- Vol-scaled: Flat during quiet markets approaches zero cost;
Flat during volatile markets has a small negative signal
representing missed-move expectation.
- Caps vol_proxy at 0.01 (1%) as numerical safety against
single-bar vol spikes dominating reward.
- Recomputes the ATR locally (the segment-complete branch that
normally does this is mutually exclusive with this Flat branch).
- Guarded on `features != NULL` and `bar_idx` range for smoke-test
compatibility.
The `0.5` factor is a numerical-safety symmetric-to-holding-cost
bound per the carve-out in `feedback_isv_for_adaptive_bounds.md`,
not a tuned hyperparameter. The `holding_cost_rate` itself flows
from config (already adaptive via shaping_scale annealing).
Not part of the triad yet:
- plan_isv train/val parity (task #94 item #2): documented
architectural gap requiring val-time plan-state generator; no
trivial safe fix.
- Atom utilization (task #94 item #4): deep C51 representation
work (atom-level entropy regularisation or target-variance
engineering); deferred pending tie-break + opp-cost validation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Val-Flat-collapse fix#1 (task #94, 2026-04-24). Research agent audit
of train-bv2n5 identified that val picks Flat for ~all states while
training picks diverse directions via Boltzmann sampling. Evidence:
val produced 16-27 trades per epoch while training produced 30K-200K
with the same policy. val_Sharpe sat at 0.00 for 16 consecutive
epochs (raw magnitudes 1e-6 to 3e-5 — noise floor).
Root cause: C51 atom utilization collapsed to ~1% (all probability
mass on a handful of atoms). With v_range≈240 and 50 atoms, each
atom spans ~9.6 units; direction E[Q] values drifted by 1e-4 to
1e-5 from floating-point precision alone — below atom resolution
but above the existing `1e-6f` tie-break threshold. The strict
argmax then gave one bin (typically Flat via initialization /
accumulated-drift bias) a consistent-but-meaningless edge.
Fix: widen the tie-break epsilon adaptively using the per-branch
Q-support half-width stored in the ISV bus (V_HALF_{DIR,MAG,ORD,URG}
at indices 24, 26, 28, 30). `tie_eps = max(1e-6, 0.01 × isv_half)` —
Q-values within 1% of the learned support are numerically
indistinguishable from atom jitter and fire the existing Philox
uniform-among-tied sampler. NULL ISV falls back to 1e-6 (pre-fix
behaviour) for smoke-test callers that don't wire the bus.
Applied to all four branches in `experience_action_select`:
- direction: half_dir from isv[24]
- magnitude: half_mag from isv[26]
- order: half_ord from isv[28]
- urgency: half_urg from isv[30]
The 1% fraction is a numerical-safety bound per
`feedback_isv_for_adaptive_bounds.md` (safety carve-out: "Safety
hard limits can stay as constants; adaptive training-dynamics bounds
cannot"). The bound IS a safety floor; the adaptation flows through
isv_half which is itself an EMA-driven adaptive signal.
Complementary fixes still pending (task #94 remainder):
- opportunity_cost in training reward (structural Flat-Q-≈-0 fix)
- atom utilization / v-range improvements (deep C51 fix)
The plan_isv train/val gap is a known documented trade-off
(gpu_backtest_evaluator.rs:524-535) deferred pending infrastructure
to compute plan signals at val time.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Root-cause bug #3 from the C-audit (task #92). The IQL advantage-weighting
kernel `iql_per_branch_advantage` computed:
iql_scale = a_branch[d] / max_a // per-branch advantage / max
branch_scales[b, d] = r * iql_scale + (1-r) * 0.25
When IQL readiness `r` → 1 and a sample's branch advantage `a_branch[d]`
is near zero, the scale collapsed to ~0. That zero multiplies the C51
per-sample gradient to zero, starving (b, d) of learning signal. For
direction branch this happens disproportionately on Hold (d==1 action)
and Flat (d==3 action) samples where the direction advantage is
structurally small. Combined with the ~22% Hold+Flat sample fraction,
direction accumulates gradient from only a subset of the batch — part of
the compound 150x grad_ratio_mag_dir observed in train-5wb4n.
Add an ISV-driven floor:
- New ISV slot (36): IQL_BRANCH_SCALE_FLOOR_INDEX
- ISV_TOTAL_DIM 36 → 37
- Bootstrap at 0.1 (safety bound, 10% of uniform=0.25 — per-sample
scale can never fall below this)
- `iql_per_branch_advantage` kernel takes ISV pointer + slot index,
reads floor, takes `max(scale, floor)` before writing branch_scales
- `compute_branch_scales` in `gpu_iql_trainer.rs` threads ISV ptr +
slot index through (mirrors existing `compute_per_sample_support`
pattern)
- `fused_training.rs` step 7 passes `self.trainer.isv_signals_dev_ptr()`
+ `IQL_BRANCH_SCALE_FLOOR_INDEX` at the call site
The floor at 0.1 is a numerical-safety bound per the carve-out in
`feedback_isv_for_adaptive_bounds.md` ("Safety hard limits can stay as
constants"). It's addressable via ISV for future adaptive tuning if
observed starvation patterns warrant, but is not actively EMA-updated
today.
Completes the three-part fix triad for task #92:
1. d61aefe2b — ISV-driven grad balancer (safety net, per-branch norm)
2. dc6a034a4 — C51 backward standardization (math bug, per-atom)
3. this commit — IQL branch_scales floor (per-sample starvation)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Root-cause bug from C-audit of train-5wb4n magnitude-branch Q-saturation
(task #92). Forward kernel `c51_loss_batched` (c51_loss_kernel.cu
lines 736-752) applies per-atom advantage standardization for the
magnitude branch:
centered[j] = (adv[a_d,j] - a_mean[j]) / (a_std[j] + 1e-6) // d==1 only
combined[j] = value[j] + centered[j]
lp[j] = log_softmax(combined)
But the backward kernel `c51_grad_kernel` treated the forward as if
`combined[j] = value[j] + (adv[a_d,j] - a_mean[j])` — no a_std divisor.
That is chain-rule-inconsistent: the computed `d_combined / d_adv[a,j]`
is the gradient of a *different* loss than the one forward-pass'd.
Gradient points in the wrong direction relative to the actual loss
surface.
Fix: pass the magnitude-branch online advantage logits pointer
(`on_adv_logits_b1`) to the backward kernel. For each (b, j) in the
d==1 dueling-grad block, recompute a_std from the 3 advantage values
and multiply `grad_val *= 1/(a_std + 1e-6)`. Counterfactual magnitude
gradient (Hold samples) inherits the same factor automatically —
single correction point.
Approximation: treats a_std as constant w.r.t. advantage values (skips
`da_std/d_adv` chain-rule terms). This is the standard simplification
used in batch-norm with `track_running_stats=False`; produces the
dominant scale correction without the full Jacobian complexity.
Call site updates the single launch in `launch_c51_grad`. Kernel
signature guards on NULL for backward compatibility with any smoke-
test launcher that may not wire the argument; d==1 block falls back
to pre-fix behaviour (identity) in that case.
Part of task #92 fix triad. Previous commit d61aefe2b addressed the
balancer (safety-net fix). Next: IQL branch_scales floor for
direction-branch Hold/Flat per-sample starvation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The pre-spec balancer used `cap = 4 × median` to suppress outlier
branches. On L40S train-5wb4n (epoch 9 norms `[dir=130, mag=20000,
ord=18000, urg=16000]`), that made cap = 68000 — exceeding every
branch's norm. No cap ever fired; the kernel was a geometric no-op
for 11 consecutive epochs while direction starved at 1/150 the
magnitude gradient. The outlier-cap algorithm can only handle "one
branch above the pack", not "three co-elevated, one starved".
Replace with ISV-driven equalisation:
- 4 new ISV slots (31..34): per-branch grad-norm target
- 1 new ISV slot (35): scale clamp limit
- ISV_TOTAL_DIM 31 → 36
- New on-GPU producer kernel `grad_balance_isv_update` runs after
`branch_grad_norm_reduce`, before `branch_grad_rescale`. Computes
current-step median + max/min spread, applies adaptive-rate EMA
to the ISV slots. Single-threaded (1 warp), graph-capture safe,
no atomicAdd, no host syncs.
- Rescale kernel now reads ISV target + limit, computes per-branch
`scale[b] = clamp(target[b] / norm[b], 1/limit, limit)`. Every
branch equalises toward the shared median target over time;
starved branches boost, elevated branches suppress. Limit is the
observed max/min spread EMA, hard-bounded [2.0, 1e4] as a
numerical-safety cap (not tuning).
Adaptive-rate EMA: `alpha = clamp(err / (err + baseline), 0.01, 0.3)`.
Fast response when error is large relative to current value, slow
drift when stable. No fixed alpha constant.
Bootstrap at construction + fold reset: targets = 1.0, limit = 2.0.
Pre-first-EMA-update behaviour: scale clamped to [0.5, 2.0], effectively
the original identity on warm-up, until real observations flow in.
Complies with feedback_isv_for_adaptive_bounds.md (2026-04-23):
adaptive bounds always live in ISV, never hardcoded. No partial
refactor — all consumers of the balancer contract (launch, kernels,
rescale algorithm) migrate together.
Still pending from the task-#92 triad: (a) C51 backward kernel must
replicate the forward advantage-standardization for d==1 (chain-rule
violation produces wrong gradient; separate commit), (b) IQL
branch_scales floor for direction-branch Hold/Flat starvation
(separate commit). This commit is the safety net; the two root-cause
fixes come next.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The project's Cargo.toml and .cargo/config.toml both set
`incremental = true`, which makes rustc emit per-query save-analysis
artifacts that sccache does not cache. Result: even though
RUSTC_WRAPPER=sccache was set, the rustc calls wrote incremental
state that subsequent builds saw as stale and recompiled anyway.
Setting CARGO_INCREMENTAL=0 in the ensure-binary env forces rustc
to emit pure object output that sccache hashes uniformly on
(source + args). Same SHA → full cache hit; small source change →
only dirty crates recompile.
Does not change the service compile-and-deploy-template path,
which intentionally uses cargo-incremental against a persistent
/cargo-target-cpu PVC for repeated rebuilds of a narrow set of
service binaries.
Applies from the next workflow submission onward (train-5wb4n
already compiled under the old config and is running).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Root cause of Q-value saturation at +/-50 seen in train-6nbx5 after
ISV v-range unification (9deda5f65, 11df03785): cold-start path in
`update_eval_v_range` latched `q_std_ema = q_std.max(0.01)` on the
first epoch. But that first `q_std` is dominated by the +/-50
bootstrap atom-spread (scaffolding set in construct/reset), NOT by
real Q-distribution spread. Result: `3*std_ema` exceeds
`min_half_floor=10` and approaches `abs_half=50` immediately, atoms
stay wide next epoch, next `q_std` confirms that width, EMA never
escapes. Q saturated at +/-abs_half every run.
Fix 1 (gpu_dqn_trainer.rs:3106-3120): seed
`eval_q_std_ema = min_half_floor / 3.0` at cold start so initial
`half = 3 * std_ema = min_half_floor` exactly. Adaptive-rate EMA
(alpha clamped to [0.01, 0.3]) then relaxes upward only if genuine
Q-spread warrants it. Breaks the self-confirming initialization.
Fix 2 (training_loop.rs:479-494): remove leftover pre-clamp of
reward quantiles to `config.v_{min,max}`. That was from the earlier
quantile-clamp fix (d38a8cf99). Phase 2c (9deda5f65) moved the
per-branch clamp inside `warm_start_atom_positions`, which reads
each branch's [centre-half, centre+half] from the ISV pinned bus.
An outer static clamp to the wider config bound is redundant
double-clamping and hides which layer owns the support. Pass raw
quantiles through to warm_start.
Validated locally: SQLX_OFFLINE cargo check -p ml passes (only
pre-existing warnings).
Next: push + L40S validation run. Diagnostic instrumentation from
423ac460b remains in place to confirm (center, half) trajectory on
the next run — will be removed once validated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds targeted tracing::info! at three call sites to diagnose why Q-value
range hits exactly +/-50 (config hard safety clamp) at every epoch after
the ISV v-range unification commits 9deda5f65 and 11df03785.
Instrumentation (all under target="isv_vrange_diag"):
1. update_eval_v_range (called at epoch-final Q-stats): per-branch
(q_mean, q_std, q_gap, center, half, initialized_before) — captures
whether cold init or warm EMA path produced the ISV write.
2. recompute_atom_positions (called at epoch N start): per-branch host
read of ISV (center, half) — what the adaptive_atom kernel will
consume on this epoch's forward pass.
3. warm_start_atom_positions (called mid-epoch, after experience
collection): per-branch clamp bounds (b_min, b_max, isv_ptr_null),
plus post-write per-branch (amin, amax) of the atoms just uploaded.
NOT PUSHED — local diagnostic commit for root-cause investigation.
Completes the ISV-unified Q-support range spec
(docs/superpowers/specs/2026-04-23-isv-v-range-unification.md) by
migrating the per_sample_support buffer from per-sample [B, 3] to
per-sample-per-branch [B, 4, 3] stride-12. Without this phase the
atom_positions grid already spanned per-branch adaptive ranges (landed
in 9deda5f65 via ISV slots 23..30) while the loss-projection
Bellman step still read a single V(s)-centred range — atoms and
projection disagreed, which is the exact pathology the spec fixes.
Producers:
* iql_value_kernel.cu::iql_compute_per_sample_support — new kernel
signature adds isv_signals pointer; writes 4 branch triples per
sample where centre = isv_signals[23 + 2*d] and half-width = V(s)-
derived Q spread. Bootstrap identity: ISV centres=0 + readiness=0
falls back to [-1, 1] across all 4 branches, byte-identical to the
pre-Phase-2d single-range default at epoch 1.
* iql_value_kernel.cu::iql_support_floor — Frugal-1U p5 estimator now
aggregates half-widths across all (sample, branch) pairs and applies
the floor per (sample, branch) independently.
* gpu_iql_trainer.rs — per_sample_support_buf sized b*4*3, seed writes
12 floats per sample, compute_per_sample_support takes isv_dev_ptr
and forwards it to the kernel; launch-site arg order aligned.
* fused_training.rs — passes trainer.isv_signals_dev_ptr() into the
IQL call.
Consumers (all migrated to stride-12 indexing `b*12 + d*3 + {0,1,2}`):
* c51_loss_kernel.cu::c51_loss_batched — per-branch (v_min, v_max,
delta_z) read INSIDE the d-loop; degenerate-support skip is now
per-branch (continue instead of whole-sample early exit).
* c51_grad_kernel.cu::c51_grad_kernel — per-branch z_norm and
delta_z for the q-gap floor gradient path.
* experience_kernels.cu::compute_expected_q — per-branch (v_min, dz)
inside the d-loop that iterates all 4 branches.
* experience_kernels.cu::mag_concat_qdir — reads direction branch
(d=0) slots from the stride-12 tile.
* experience_kernels.cu::quantile_q_select — per-branch (v_min, dz)
inside the d-loop.
Experience-collector parity:
* gpu_experience_collector.rs — its OWN per_sample_support_buf grows
to alloc_episodes*4*3; update_per_sample_support tiles the same
(v_min, v_max, delta_z) triple to all 4 branches so the layout
matches the IQL buffer and the consumer kernels read uniformly.
Safety:
* Bootstrap byte-identical at epoch 1 preserved (ISV centres default 0,
readiness ramps from 0 → 1).
* No stub values, no TODO/FIXME/XXX markers introduced.
* Kernel scalar arg (gamma) is already f32 in GpuIqlConfig — no f64→f32
cast needed at the call site (feedback_cudarc_f64_f32_abi compliance
via type, not cast).
* c51_loss branch-degenerate `continue` is uniform across the block
(all threads read the same support_base) so __syncthreads inside
the loop body remains collective.
Compile verified: cargo check -p ml + --workspace pass (SQLX_OFFLINE,
CARGO_INCREMENTAL=0, sccache) and cargo build -p ml compiles all
CUDA kernels via nvcc. cargo test -p ml --lib --no-run succeeds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Unifies the Q-support range source across atom grid / warm-start quantile
clamp consumers via the ISV signal bus. One broadcast written at epoch
boundary from per-branch Q-stats EMAs, read by the two consumers that
previously held disagreeing ranges. Target observation: atom utilisation
≥40% (up from 11-15% on train-fpxnw).
Phase 0 — per-branch Q-stats kernel Rust plumbing:
* Load q_stats_per_branch_reduce alongside legacy q_stats_reduce
* Add per_branch_q_stats_pinned (28 f32 = 4 × 7, device-mapped)
* PerBranchQValueStats struct: [QValueStatsResult; 4]
* reduce_current_q_stats_per_branch launches the new kernel with the
four branch (off, size) pairs derived from config.branch_N_size
Phase 1 — ISV v-range plumbing (zero behavioural change at epoch 1):
* ISV_NETWORK_DIM=23 preserved for w_isv_fc1 sizing; ISV_TOTAL_DIM=31
allocates 8 additional slots for per-branch (centre, half-width)
* Slot constants V_CENTER_DIR..V_HALF_URG covering slots 23..30
* eval_q_mean_ema / eval_q_std_ema / eval_ema_initialized promoted
to [f32; 4] / [bool; 4]; scalar setters preserved for trajectory
backtracking (broadcast same value to all branches)
* Bootstrap at construction: centre=0, half=(v_max-v_min)/2 → the
byte-identical [config.v_min, config.v_max] span per branch before
any Q observations arrive
* reset_eval_v_range_state resets the 4 per-branch EMAs AND the 8 ISV
slots to bootstrap values; legacy eval_v_range_pinned[2] still reset
(deferred removal — spec Phase 3)
* update_eval_v_range reworked: signature takes PerBranchQValueStats and
per_branch_q_gaps. Maintains 4 independent adaptive-rate EMAs,
computes (centre, half) per branch with min_half_floor=0.1×(v_max-v_min)
and clamps to config bounds, writes 8 ISV slots. Branch-0 (direction)
centre±half is also mirrored into the legacy eval_v_range_pinned for
consumers that have not yet migrated to the per-branch bus.
Phase 2a/2b — atom grid per-branch v-range:
* adaptive_atom_positions kernel signature changed from
(v_min: float, v_max: float) to (branch_idx: int, isv_signals: float*);
reads centre/half from ISV slots 23+2·b, 24+2·b. Eliminates the f64→f32
ABI trap (spec Phase 2 side-effect) since the only per-branch range
path is now pointer-based.
* recompute_atom_positions passes branch_idx + isv_signals_dev_ptr per
branch; no scalar v_min/v_max arg remains.
Phase 2c — warm-start quantile clamp per-branch from ISV:
* warm_start_atom_positions reads per-branch (centre, half) from pinned
ISV host memory, clamps shared reward-quantile vector into each
branch's adaptive range before tiling into atom_positions_buf.
Bootstrap makes this equivalent to the pre-spec config.v_{min,max}
clamp until the first Q observation lands.
Deviations from spec:
* Phase 2d (per_sample_support_buf → [N, 4, 3]) NOT implemented. The
spec's premise was that per_sample_support is host-tiled from
eval_v_range, but the active path in this codebase has it filled by
iql_compute_per_sample_support (V(s)-centered, per-sample, already
adaptive) — orthogonal to the ISV bus. Migrating that kernel to
per-branch output would require rewriting iql_value_kernel +
iql_support_floor + C51/MSE loss kernel indexing in lockstep, which
the "no unrelated refactoring" constraint disallows. The loss-kernel
Bellman projection today uses V-centered bounds that are themselves
adaptive; the ISV v-range fix still lands the primary win (atom grid
+ warm-start agreement) without touching IQL.
Compile verified: cargo check -p ml + --workspace pass (SQLX_OFFLINE,
CARGO_INCREMENTAL=0, sccache). No TODO/FIXME/XXX introduced.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Root cause of the Q=±333k explosion at every run's epoch 2.
`warm_start_atom_positions` writes quantiles from the raw environment
reward distribution directly into `atom_positions_buf`. Raw rewards
are unbounded PnL-scaled values — a single extreme sample in the
first experience buffer becomes `atom_positions[num_atoms-1]`, and
the C51 expected-value readback `Q = Σ prob × atom_pos` inherits
that magnitude.
Observed deterministically across train-7rgqd, train-5gzpn, and
train-gj54m: epoch 1 Q in `[0, ~6]` (initial Xavier atoms), epoch 2
Q at exactly `±333406` once warm-start writes the sorted-reward-tail
into the atom grid. Every downstream path — the C51 loss projection,
eval_v_range EMA, IQL support, HEALTH_DIAG q_gap — assumes
`atom_positions ∈ [v_min, v_max]`. The warm-start path was the only
one bypassing that assumption.
Clamp each quantile to the configured `[v_min, v_max]` before writing.
This is a safety rail, not a tuning parameter: config.v_{min,max} are
already derived from reward_scale (±15 default), which is the support
range the rest of the system expects.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
target_params_buf was initialized once via DtoD copy at first train
step and then only moved toward online via slow Polyak EMA (tau≈0.005).
At fold boundaries the online weights are shrink-and-perturb'd with
alpha=0.8, which modifies params_buf in-place — but target_params_buf
still held the end-of-previous-fold values. The Bellman target would
then use stale weights against freshly perturbed online predictions,
producing a large TD error gap in the first fold-N+1 training steps.
Polyak averaging at tau=0.005 is far too slow to close that gap before
the oversized gradients compound through Adam into runaway updates —
one of the drivers of the fold-1 gradient explosion observed in both
train-7rgqd and train-5gzpn.
- Add GpuDqnTrainer::sync_target_from_online() — DtoD memcpy of the
full params_buf into target_params_buf.
- Call it from FusedTraining::reset_for_fold right after shrink-and-
perturb and before reset_adam_state, so target = perturbed online
and Adam moments zero out from the same starting point.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
