nsys profile of multi_fold_convergence on L40S identified backtest_state_gather as
the #1 GPU consumer at 37.2% (596 ms / 60,588 calls — kernel launch latency
dominated compute) and the per-step TLOB cuBLAS gemvx calls as #2 at 25%
(242K calls). Both share the same per-step amplification: chunk_len=512 separate
gather launches + 512 separate TLOB.forward calls (4 SGEMMs each) per chunk
before any Q-values can be computed.
This commit replaces the per-step gather + DtoD pattern with a single batched
launch, and reuses the same chunked buffer for a single chunk-wide TLOB forward.
Per-chunk launch reduction: from 2*chunk_len + chunk_len*7 to 1 + 7 for the
gather+TLOB phase (4608 -> 8 with chunk_len=512, a 576x reduction).
New kernel `backtest_state_gather_chunk` (experience_kernels.cu):
- Writes [chunk_len, N, padded_sd] directly into chunked_states_buf
- chunk_len * N threads, 1 thread per output row
- Mathematically identical to per-step gather: portfolio_buf and plan_isv_buf
are CONSTANT within a chunk (env_step + plan_state_isv update at chunk
boundary only). Each thread reads independent feature offsets, no atomics,
no reordering.
Chunked val TLOB (gpu_backtest_evaluator.rs + metrics.rs):
- Val TLOB instance now sized to DQN_BACKTEST_CHUNK_SIZE * n_windows via new
GpuBacktestEvaluator::val_tlob_batch_size() helper.
- submit_dqn_step_loop_cublas calls tlob.forward(chunked_states, batch) ONCE
per chunk on the chunk-wide buffer instead of chunk_len times on states_buf.
- Partial last chunks reuse the same buffers (forward(b) accepts any
b <= construction_batch).
Borrow restructure:
- Removed top-of-function `let ch_states = self.chunked_states_buf.as_ref()?`
binding (TLOB needs &mut). Replaced with per-chunk ch_states_base raw u64
device pointer extracted in tight scope, reused by Phase 1 (gather) and
Phase 2+3 (compute_q_values_to + last_step_states_ptr). The pointer is
stable across the chunk because the Option<CudaSlice<f32>> does not
reallocate.
Per-step gather kernel `backtest_state_gather` retained unchanged for
evaluate() / evaluate_ppo() / evaluate_supervised() paths that still need
a single-step writer (closure-based callers with no chunked buffer).
Audit doc dqn-gpu-hot-path-audit.md updated with Fix 18 entry per Invariant 7.
Build: SQLX_OFFLINE=true cargo check -p ml --lib clean (12 warnings, baseline).
Tests: cargo test -p ml --lib --no-run clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per feedback_mbp10_mandatory.md, MBP-10 + trades are mandatory inputs.
multi_fold_convergence now reads FOXHUNT_MBP10_DATA / FOXHUNT_TRADES_DATA
env vars, forwards them to train_baseline_rl as --mbp10-data-dir /
--trades-data-dir, and hard-fails if either path is missing.
Argo sanitizer-test + nsys-test templates set the env vars to PVC paths
/data/test-data/{mbp10,trades} and FOXHUNT_TEST_DATA points at
/data/test-data/ohlcv (PVC layout has lost+found/trades/mbp10/ohlcv as
top-level subdirs; the symbol ES.FUT lives under ohlcv/).
Without this, the smoke would silently fall back to tick-rule proxy
classification and validate a degraded model variant (no real OFI),
defeating the purpose of L40S validation. Audit doc updated per
Invariant 7.
After Fix 1..16 migrated all 80+ production callers off
`super::htod_f32` and `super::clone_htod_f32`, the helper bodies in
`cuda_pipeline/mod.rs:129-145` had zero non-test consumers. Deleted
both function definitions per `feedback_no_legacy_aliases.md` (no
deprecated wrappers).
Per `feedback_no_partial_refactor.md` (when a shared contract is
deleted, every consumer migrates together — including tests), the
two surviving test-block callers in `gpu_tlob.rs::tests` (lines
1017 and 1132) are migrated to `mapped_pinned::upload_f32_via_pinned`
in the same commit. The other test-only callers in
`signal_adapter.rs::tests`, `gpu_action_selector.rs::tests`, and
`cuda_pipeline/mod.rs::tests` use bare `stream.memcpy_htod` /
`stream.memcpy_stod` against the cudarc handle directly (not the
deleted helpers) — no change needed.
A docstring was added at the deletion site recording when and why
the helpers were removed, pointing future readers at the canonical
replacements `mapped_pinned::clone_to_device_f32_via_pinned` and
`mapped_pinned::upload_f32_via_pinned`.
Final state of the HtoD migration sequence:
- production callers of `stream.memcpy_htod` / `memcpy_stod`: 0
- production callers of `htod_f32` / `clone_htod_f32`: 0
- helper definitions: removed from `mod.rs`
docs/dqn-gpu-hot-path-audit.md updated with Fix 17 entry.
cargo check -p ml --lib clean at 12 warnings.
cargo check -p ml --tests clean at 23 warnings (12 lib duplicates +
11 test-specific, baseline unchanged).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The two `crate::cuda_pipeline::clone_htod_f32` callsites in
`training_loop.rs::set_raw_market_data` (lines 1250 and 1254) are
rewritten to `mapped_pinned::clone_to_device_f32_via_pinned` per
`feedback_no_htod_htoh_only_mapped_pinned.md`.
These run once per fold immediately before the step loop begins; the
`_raw` suffix indicates the un-normalized arrays kept on-GPU for the
eval-time critic.
Error-type continuity: the helper already returns `Result<_, String>`,
which feeds `anyhow::anyhow!("…raw upload: {e}")` directly — the prior
.map_err wrapper is preserved verbatim. 1:1 path swap.
docs/dqn-gpu-hot-path-audit.md updated with Fix 16 entry.
cargo check -p ml --lib clean at 12 warnings.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The remaining `super::clone_htod_f32` call in `upload_ofi_features`
(`:4258`) is rewritten to `mapped_pinned::clone_to_device_f32_via_pinned`
per `feedback_no_htod_htoh_only_mapped_pinned.md`. Called from
training_loop's data-load phase before each fold's training run, so
the OFI tensor (4M bars × 32 dims) now stages through mapped-pinned
+ DtoD just like all other large data uploads.
Error-type shift: `mapped_pinned::clone_to_device_f32_via_pinned`
returns `Result<CudaSlice<f32>, String>` whereas `clone_htod_f32`
returned `Result<CudaSlice<f32>, MLError>`. Wrapped with
`.map_err(|e| MLError::ModelError(format!("upload_ofi_features: {e}")))`
to preserve the call-site label in error messages.
docs/dqn-gpu-hot-path-audit.md updated with Fix 15 entry.
cargo check -p ml --lib clean at 12 warnings.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Residual COLD ctor sites missed by the original audit are now migrated
off explicit HtoD per `feedback_no_htod_htoh_only_mapped_pinned.md`:
- `:9922` sel_clip_buf 1-element init (sigmoid head clip-norm seed)
- `:10075-10078` spectral norm init_u_s1/v_s1/u_s2/v_s2 (4 calls)
- `:10095-10096` `alloc_spec_pair!` macro body — both u/v init uploads
inside the macro now go through `upload_f32_via_pinned`; the macro's
`$lbl_u`/`$lbl_v` are reused in the error-message format so per-pair
failures stay diagnostically distinct
- `:11276` graph_params_host (60 floats: cross-branch graph message
passing weights)
- `:11330` denoise_params_host (1800 floats: 2-step diffusion Q-refinement)
- `:11476` qlstm_weights_host (528 floats: QLSTM Xavier init)
All sites use `mapped_pinned::upload_f32_via_pinned` (the canonical
mapped-pinned + DtoD staging helper). The helper returns
`Result<_, String>` whereas this constructor returns
`Result<_, MLError>`, so each site wraps the error via
`.map_err(|e| MLError::ModelError(format!("<site> upload via pinned: {e}")))`.
Site labels preserved so backtraces remain readable.
docs/dqn-gpu-hot-path-audit.md updated with Fix 14 entry.
cargo check -p ml --lib clean at 12 warnings.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Agent 1 — bulk COLD/WARM migration across init/loader files.
Sites: 39 production HtoD calls migrated to MappedF32Buffer / MappedI32Buffer /
new MappedU32Buffer. Cross-file infrastructure includes new staging helpers
(clone_to_device_{f32,i32}_via_pinned, upload_{f32,i32,u32}_via_pinned),
MappedU32Buffer type, and host_slice_mut accessor on MappedF32/U32Buffer.
Per feedback_no_htod_htoh_only_mapped_pinned.md — first of 3 parallel batches.
# Conflicts:
# docs/dqn-gpu-hot-path-audit.md
HOT sites:
- L2992 hindsight bar_indices: was `alloc + memcpy_htod(total i32s)` on
every collect_experiences_gpu when hindsight_fraction > 0. Added
persistent `bar_indices_pinned: MappedI32Buffer` (capacity =
alloc_episodes * alloc_timesteps * 2) allocated at constructor.
Per-call host writes go through host_ptr; relabel kernel reads via
dev_ptr u64.
- L3186 feature_mask: was alloc + memcpy_htod each epoch at t==0.
Promoted `feature_mask_buf` to `Option<MappedF32Buffer>`. Reallocates
only when mask size changes; CPU writes via host_ptr, state_gather
reads via dev_ptr.
- L3816 update_per_sample_support: per_sample_support_buf is read-only
by the kernel — promoted to MappedF32Buffer (kernel args now
dev_ptr u64). per-epoch tile fill becomes a direct host_ptr write.
WARM sites:
- L3097 episode_starts upload: episode_starts_buf is GPU-mutated by
domain_rand_episode_starts so must remain CudaSlice. Stage via
MappedI32Buffer + memcpy_dtod_async through new helper
`upload_host_to_cuda_i32_via_pinned`.
- L2009 upload_expert_actions: promoted expert_actions_gpu from
Option<CudaSlice<i32>> to Option<MappedI32Buffer>. Direct host_ptr
write replaces alloc + memcpy_htod.
COLD sites:
- L1076,3848 portfolio_states init/reset (GPU-mutated): use
upload_host_to_cuda_f32_via_pinned (mapped-pinned staging + DtoD).
- L1093 epoch_state init (GPU-mutated): replace clone_htod_f32 with
explicit alloc_zeros + upload_host_to_cuda_f32_via_pinned.
- L1341 saboteur_base init (GPU-mutated): use the same helper.
- L2784 trade_stats_buf pre-reduction zero: replaced htod_f32 of an
all-zeros vec with stream.memset_zeros — fully GPU-side, no PCIe.
Imports: added DevicePtrMut for memcpy_dtod_async pointer extraction.
Per `feedback_no_partial_refactor.md`, every consumer of the migrated
buffers is updated in this commit:
- per_sample_support_buf kernel args at L3426 (compute_expected_q) and
L3546 (quantile_q_select) now consume dev_ptr u64.
- feature_mask_buf accessor at L3195 reads via .dev_ptr.
- expert_actions_gpu field type change is contained (no external
consumers in this crate).
cargo check -p ml --lib: 11 warnings (unchanged from prior commit on
this branch; 2 below 13-warning baseline because two unrelated trivial
warnings disappeared as a side-effect of the refactor).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per `feedback_no_htod_htoh_only_mapped_pinned.md`, mapped pinned
(cuMemHostAlloc DEVICEMAP) is the only allowed CPU↔GPU path.
Migrates 3 WARM HtoD sites:
- `fused_training.rs:505` HER source_indices Vec<i32> upload
- `fused_training.rs:511` HER reward_ones Vec<f32> upload
- `training_loop.rs:470` curriculum episode_starts Vec<i32> upload
For `fused_training.rs` adds module-level `staging_upload_{i32,f32}`
helpers. For `training_loop.rs` uses an inline staging+DtoD block
because the destination is reached via the out-of-scope accessor
`collector.episode_starts_buf_mut() -> &mut CudaSlice<i32>`.
All destination buffer types remain `CudaSlice<T>` so no downstream
consumer (incl. `gpu_her::relabel_batch_with_strategy` and any
`episode_starts_buf` reader) needs to change.
Note: if Agent 2's `gpu_experience_collector` merge changes
`episode_starts_buf_mut`'s return type to a mapped pinned buffer,
the inline staging block in `training_loop.rs:470` can be simplified
to a direct `write_from_slice` (follow-up).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per `feedback_no_htod_htoh_only_mapped_pinned.md`, mapped pinned
(cuMemHostAlloc DEVICEMAP) is the only allowed CPU↔GPU path.
Both `upload_params` and `upload_target_params` previously did
`stream.memcpy_htod` of TOTAL_PARAMS f32 (~MB) at fold boundaries /
external weight loads. Now route through the helper
`update_via_mapped_f32` introduced in the previous commit — stages
CPU bytes through a transient mapped pinned buffer and DtoD-copies
into the existing `params_buf` / `target_params_buf` `CudaSlice<f32>`.
No public API change. The destination buffer types remain
`CudaSlice<f32>` so every downstream kernel-arg consumer is untouched.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cold-path constructor sites (2):
- portfolio_states init: htod_f32 → upload_f32_via_pinned
- rng_states init: memcpy_htod → upload_u32_via_pinned
Warm per-collect sites (2):
- episode_starts_buf: memcpy_htod → upload_i32_via_pinned
- barrier_config: htod_f32 → upload_f32_via_pinned
Warm per-epoch reset (2 sites + persistent staging refactor):
The Vec<f32>/Vec<u32> host staging fields (portfolio_init_staging,
rng_seed_staging) replaced with persistent MappedF32Buffer /
MappedU32Buffer allocated once at construction. Resets fill via
host_slice_mut() (zero-allocation, zero-HtoD); a single async DtoD
seeds the persistent device-resident portfolio_states / rng_states
(which the kernel mutates each step and must remain in VRAM).
Adds host_slice_mut() accessor on MappedF32Buffer and MappedU32Buffer
to support structured per-element writes against the mapped pages.
Audit row appended (Fix 11) in docs/dqn-gpu-hot-path-audit.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per `feedback_no_htod_htoh_only_mapped_pinned.md`, mapped pinned
(cuMemHostAlloc DEVICEMAP) is the only allowed CPU↔GPU path.
Adds module-level `upload_via_mapped_{f32,i32,u32,u64}` helpers and an
in-place `update_via_mapped_f32`. Each stages CPU data through a
transient mapped pinned buffer and DtoD-copies into the destination
`CudaSlice<T>`, then stream-syncs so the staging buffer is safe to
drop. Destination buffer types remain `CudaSlice<T>` so every existing
consumer (raw_ptr / kernel arg) is untouched, satisfying
`feedback_no_partial_refactor.md` for these one-shot init paths.
Migrated COLD sites in `GpuDqnTrainer::new`:
- weight_decay_mask (TOTAL_PARAMS f32)
- branch_slice_starts_dev, branch_slice_lens_dev ([i32; 4])
- branch_grad_scales_dev ([f32; 4])
- per_branch_gamma_base/max_dev ([f32; 4] each)
- q_quantile_branch_offsets/sizes_dev ([i32; 4] each)
- spectral_norm_descriptors_dev ([u64; 78])
- stochastic_depth_scale_buf ([f32; 3])
- stochastic_depth_rng_state ([u32; 1])
- vsn_group_begins_buf, vsn_group_ends_buf ([i32; num_groups] each)
- mamba2_params Xavier init (mamba2_param_count f32)
11 COLD HtoD sites eliminated. cargo check clean (15 warnings; +2 over
baseline 13 are the new MappedU32/U64 struct visibility warnings,
matching the existing MappedF32/I32 pattern).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- 1 constructor site (rng_states u32 init) rewritten to
mapped_pinned::upload_u32_via_pinned.
- 2 warm-path sites (source_indices, donor_indices in relabel_batch)
rewritten to mapped_pinned::upload_i32_via_pinned. The warm sites
re-allocate pinned staging per relabel call — acceptable on this
cold path (HER relabel runs once per epoch).
Adds MappedU32Buffer type to mapped_pinned.rs mirroring MappedI32Buffer,
plus upload_u32_via_pinned helper. Manual Debug impl so warning count
stays at 13.
Audit row appended (Fix 10) in docs/dqn-gpu-hot-path-audit.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
HOT site: `train_step_gpu` was uploading the 4-byte Adam step counter via
`stream.memcpy_htod(&[step], &mut scratch.adam_t)` on every backward pass.
Promoted `DtScratch.adam_t` from `CudaSlice<i32>[1]` to `MappedI32Buffer(1)`
allocated once in `alloc_dt_scratch`. Per-step CPU writes go through
`host_ptr`; the Adam kernel reads via `dev_ptr` u64. Trivial fix, massive
frequency — runs once per gradient step.
Other sites:
- L478 ctor params init: `params` is mutated by Adam (must stay CudaSlice).
Upload via mapped-pinned staging + `memcpy_dtod_async` through new
helper `upload_host_to_cuda_f32`.
- L501 ctor wd_mask init: wd_mask is read-only by Adam. Promoted to
`MappedF32Buffer`; CPU writes 1.0s through host_ptr, kernel reads dev_ptr.
Caller switched from `wd_mask.raw_ptr()` to `wd_mask.dev_ptr`.
- L1105 trajectory build episode_starts: one-time, replaced `clone_htod`
with `MappedI32Buffer` (direct host write, kernel read via dev_ptr).
- L1147 trajectory build ep_rewards: one-time, replaced `htod_f32` with
`MappedF32Buffer` (direct host write, return_to_go reads via dev_ptr).
Also added `#[allow(missing_debug_implementations)]` to MappedI32Buffer
and MappedF32Buffer in `mapped_pinned.rs` — raw pointers + CUdeviceptr
have no useful Debug impl, and the workspace `missing_debug_implementations`
lint now reaches them through the public DT type that holds them.
Audit doc: docs/dqn-gpu-hot-path-audit.md updated.
cargo check -p ml --lib: 11 warnings (down from 13 baseline; net-zero
regression, 2 minor unrelated warnings vanished as a side-effect).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- 4 stream.clone_htod sites in IQN constructor: cos_features precompute,
online_taus + target_taus tile broadcast, init_iqn_xavier_weights
upload — all rewritten to mapped_pinned::clone_to_device_f32_via_pinned.
- clone_cuda_slice helper rewritten to a single DtoD copy via the
existing super::clone_cuda_slice_f32 helper. Previously did
device→host→device, double-violating both no-DtoH and no-HtoD rules.
Sole callsite is constructor seeding of target params from online.
Audit row appended (Fix 9) in docs/dqn-gpu-hot-path-audit.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per `feedback_no_htod_htoh_only_mapped_pinned.md`, mapped pinned
(cuMemHostAlloc DEVICEMAP) is the only allowed CPU↔GPU path.
`GpuActionSelector::new` previously did one HtoD memcpy to upload
RNG seeds. Now allocates `MappedU32Buffer`, writes seeds via
host_ptr, and passes `dev_ptr` (CUdeviceptr) to the three kernels
that consume rng_states (epsilon_greedy, epsilon_greedy_routed,
branching_action_select).
Adds `MappedU32Buffer` and `MappedU64Buffer` to `mapped_pinned.rs`
mirroring the existing `MappedF32Buffer`/`MappedI32Buffer` API
(new/write_from_slice/read_all/Drop). The U64 variant is staged
for the upcoming spectral-norm host_desc[78] descriptor table
migration in `gpu_dqn_trainer::new`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- trainers/ppo.rs: 2 sites (features, targets in set_raw_market_data)
- hyperopt/adapters/ppo.rs: 3 sites (features, targets in upload path;
action_indices in run_gpu_backtest forward closure)
- hyperopt/adapters/mamba2.rs: 1 site (gpu_to_stream)
All clone_htod_f32 and bare stream.clone_htod calls rewritten to
mapped_pinned::clone_to_device_{f32,i32}_via_pinned.
Audit row appended (Fix 7) in docs/dqn-gpu-hot-path-audit.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
HOT site at simulate_batch_gpu — `actions: &[i32]` was uploaded
per-batch via stream.memcpy_htod into a persistent CudaSlice. Promoted
the field to `MappedI32Buffer` allocated once at constructor; per-call
host writes now go through `host_ptr` (no HtoD copy). Kernel arg switched
to passing `dev_ptr` as a u64.
The two COLD sites (ctor init_state, reset init_state) writing into the
GPU-mutated `portfolio_state_buf` cannot replace the destination buffer
type (env_step kernel writes back to it). Use a mapped-pinned staging
buffer + memcpy_dtod_async via a new `upload_portfolio_state` helper.
Retains zero direct HtoD copies on the only path the violation rule
permits per `feedback_no_htod_htoh_only_mapped_pinned.md`.
Sites migrated:
- L122 ctor init_state (htod_f32 -> staging+DtoD via upload_portfolio_state)
- L174 simulate_batch_gpu actions upload (HOT: persistent MappedI32Buffer)
- L296 reset init_state (htod_f32 -> staging+DtoD)
Audit doc: docs/dqn-gpu-hot-path-audit.md updated with new MIGRATED entries.
cargo check -p ml --lib: 13 warnings, baseline preserved.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Migrates the 3 clone_htod_f32 calls in walk-forward GPU data upload
(features, targets, OFI) to mapped_pinned::clone_to_device_f32_via_pinned.
Audit row appended (Fix 6) in docs/dqn-gpu-hot-path-audit.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per `feedback_no_htod_htoh_only_mapped_pinned.md`, mapped pinned
(cuMemHostAlloc DEVICEMAP) is the only allowed CPU↔GPU path.
`set_count_bonuses` is called per action-selection — converting the
three bonus buffers from CudaSlice<f32> (HtoD memcpy) to
MappedF32Buffer eliminates 3-6 HtoD memcpys per call.
Before: 3 reused-buffer htod_f32 (lines 92/96/100) + 3 first-call
clone_htod_f32 (lines 93/97/101) on every call = 3 HtoD steady-state.
After: zero HtoD — direct host_ptr writes via write_from_slice.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Migrates the 7 in-file callers of clone_htod_f32 in DqnGpuData/PpoGpuData
constructors and helpers to mapped_pinned::clone_to_device_f32_via_pinned.
Sites: features+targets+ofi upload (DqnGpuData::upload_slices,
::upload_ofi), portfolio scratch (build_batch_states), htod_copy_into,
PpoGpuData::upload.
Helper bodies htod_f32 / clone_htod_f32 left intact at lines 129-145 —
other parallel agents are still migrating files that call them.
Audit row appended (Fix 5) in docs/dqn-gpu-hot-path-audit.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds clone_to_device_{f32,i32}_via_pinned and upload_{f32,i32}_via_pinned
in `mapped_pinned.rs`. They allocate a temporary MappedXxxBuffer
(cuMemHostAlloc DEVICEMAP), write the payload via host_ptr, then async
DtoD into a target CudaSlice<T>. Used to migrate cold-path uploads off
explicit cudaMemcpy HtoD per `feedback_no_htod_htoh_only_mapped_pinned.md`
without changing struct field types where the kernel mutates the buffer
every step (e.g. portfolio_buf, weight buffers consumed by Adam).
Audit table updated in docs/dqn-gpu-hot-path-audit.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- 660f02ff4 split compute_validation_loss into launch_validation_loss + consume_validation_loss; mirror split on GpuBacktestEvaluator (launch_metrics_and_record_event + consume_metrics_after_event); evaluate_dqn_graphed_async pipelined path; mapped-pinned mirrors for actions_history/intent_mag/picked_action; lazy eval_done_event re-recorded each epoch; post-loop drain so smoke tests see fresh data
The async-diag merge (3c0d26292, building on d9cb14f1b + 673b04a8d) wired the
GPU side of cross-stream eval pipelining correctly (training stream `cuda_stream`,
eval stream `validation_stream`, `cuStreamWaitEvent` barrier on `train_done_event`)
but left a host-side `stream.synchronize()` inside
`gpu_backtest_evaluator::launch_metrics_and_download` that blocked the CPU thread
for the FULL eval drain (~25-30s/epoch on L40S). The host thread is the same one
that submits the next epoch's training kernels via `run_full_step` — so until eval
drained, training submission was gated on it, defeating the dedicated
`validation_stream`.
Fix: split metrics readback into record/consume halves at the buffer-readback layer:
* `launch_metrics_and_record_event` — submits the metrics kernel + DtoD-async
copies of `actions_history_buf` / `intent_mag_buf` / `picked_action_history_buf`
into mapped-pinned mirrors, then records `eval_done_event`. Returns immediately.
* `consume_metrics_after_event` — `event.synchronize()` (the SOLE host wait per
epoch boundary), then `read_volatile`s mapped-pinned `metrics_buf`. After the
event syncs, the four action-distribution helpers read directly from the
now-coherent mapped-pinned mirrors — eliminating four synchronous DtoH copies
that violated `feedback_no_htod_htoh_only_mapped_pinned.md`.
Caller migration (per `feedback_no_partial_refactor.md`):
* `evaluate_dqn_graphed` (existing public API) becomes a thin sync wrapper:
launch_async → consume. ABI unchanged.
* New `evaluate_dqn_graphed_async` for the pipelined path.
* `evaluate` / `evaluate_ppo` / `evaluate_supervised` migrated inline to
launch+consume (no caller pipelines them).
* `compute_validation_loss` replaced by `launch_validation_loss` +
`consume_validation_loss`. Training loop now consumes pending at epoch start
(cached_async_val_loss = epoch N-1's val_sharpe) and launches at epoch end
(sentinel-only return). Final post-loop consume drains the LAST epoch's launch
so smoke tests / hyperopt that read `last_eval_direction_dist` see the most
recent epoch's data.
Mathematical identity preserved: every val_sharpe consumed is bit-identical to
what the prior synchronous flow would have produced for the same epoch — only
the timing of the host parse differs (one epoch later, matching the existing
`cached_async_val_loss` lag semantics; HEALTH_DIAG `val [...]` / `val_dir_dist` /
`val_picked_dir_dist` emit moves with the consume).
Audit: docs/dqn-gpu-hot-path-audit.md Fix 4. Out of scope: periodic chunk-level
`stream.synchronize` calls in `submit_dqn_step_loop_cublas` (lines ~1596 + ~1863)
for kernel-error detection — host-blocking but smaller in aggregate; future work.
Workspace baseline preserved: 13 warnings.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the proposed static `include_str!` regression guard for the
fold-boundary IQN target hard-sync (issue #84, root-cause fix in commit
`7c19b5903`) with a real GPU runtime test that exercises the contract
end-to-end. The static guard only caught literal deletion of the call
line — a stub body returning `Ok(())`, a copy against the wrong buffer,
the wrong copy direction, or a queue against the wrong stream all pass
the textual assertion silently.
Test
(`cuda_pipeline::gpu_iqn_head::tests::iqn_sync_target_from_online_makes_target_equal_online`):
1. Construct a `GpuIqnHead` with default `GpuIqnConfig` on the
default CUDA stream.
2. Fill `online_params` ← 0.42 and `target_params` ← 0.99 via a
single mapped-pinned staging buffer + `cuMemcpyDtoDAsync`. No
HtoD copy is issued; the host write through `MappedF32Buffer::host_ptr`
reaches the GPU through the device-mapped alias and the DtoD
copies the staged values into each parameter buffer. The
witnesses 0.42 / 0.99 are arbitrary distinct fp32 constants —
the contract asserted is buffer equality, independent of magnitude.
3. Sanity: read both buffers back via fresh mapped-pinned destinations
+ DtoD, assert they differ pointwise.
4. Call `iqn.sync_target_from_online()`.
5. `stream.synchronize()` so the queued DtoD has retired.
6. Read both buffers back and assert bit-for-bit equality across all
`total_params` slots using `f32::to_bits` (so any future NaN-bearing
implementation also fails loud).
Per `feedback_no_htod_htoh_only_mapped_pinned.md`, all CPU↔GPU
communication routes through `cuMemHostAlloc(DEVICEMAP|PORTABLE)`
mapped pinned memory. Tests are not exempt — fills and read-backs
both use `MappedF32Buffer` + `cuMemcpyDtoDAsync`.
Buffer access exposed via four new `#[cfg(test)] pub(crate)` accessors
on `GpuIqnHead` (`online_params_slice`, `target_params_slice`,
`total_params_for_test`, `stream_for_test`) so the public API is not
widened.
Test carries `#[ignore = "gpu"]` matching the smoke-test convention
already used in `regression_detection.rs`. `cargo test -p ml --lib`
on a CPU-only host (the worktree environment) skips it cleanly; the
L40S smoke validation pool runs it via `--ignored`.
Paired with a strengthened doc-block at the call site in
`fused_training.rs::reset_for_fold` (boxed `DO NOT DELETE` warning +
reference to the new test name and issue #84) so anyone touching the
line sees the regression context inline before deleting.
Touched: `gpu_iqn_head.rs` (4 cfg(test) accessors + tests mod with
helpers + the runtime test, +217 LOC), `fused_training.rs` (boxed
comment + test reference, +16 LOC, no behaviour change),
`docs/dqn-wire-up-audit.md` (audit entry replacing the
static-test entry from the previous proposal, +33 LOC).
Verified:
* `cargo check -p ml --lib` — clean at 13 warnings (workspace
baseline).
* `cargo test -p ml --lib --no-run` — clean at 24 warnings (test
profile baseline).
* `cargo test -p ml --lib state_reset_registry` — 3/3 existing
tests pass (no 4th static-source test added).
* `cargo test -p ml --lib gpu_iqn_head` — 1 test discovered,
correctly reports `ignored, gpu` on this CPU-only worktree.
Local run not attempted — worktree environment lacks a GPU. The test
runs as part of L40S smoke validation via `--ignored`.
No fingerprint change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous absolute floor (`|q_mean| > 1.5`) in the Q-drift kill
criterion was a tuned constant in violation of
`feedback_adaptive_not_tuned.md` and
`feedback_isv_for_adaptive_bounds.md`. Replace with an ISV-driven
adaptive threshold:
kill_floor = max(0.5, 3.0 × max(ISV[Q_ABS_REF_INDEX=16],
ISV[Q_DIR_ABS_REF_INDEX=21]))
Both ISV slots are per-branch EMAs of `max(|Q_mean|)` already
maintained on-GPU by `q_stats_kernel.cu` and consumed by
`c51_loss_kernel`/`c51_grad_kernel`. The kill floor now anchors on
the same recently-observed healthy Q scale that the loss kernels
already use to normalise their collapse-fraction signals.
The 3.0× multiplier is architectural ("RL Q-divergence shows up at
2-4× healthy scale"); the 0.5 cold-start floor is an Invariant-1
numerical-stability bound active only while both ISV slots are still
≤ 1e-6 in fold-0 epoch-1, then dominated by the runtime formula. The
2× ratio gate is unchanged — already an architectural rate-of-change
bound.
ISV reads use the existing pinned/device-mapped path
(`fused.trainer().read_isv_signal_at`) — no HtoD/DtoH per
`feedback_no_htod_htoh_only_mapped_pinned.md`.
The bug-signature trajectory that motivated the original 1.5 floor
(F1 ep2 Q=+2.23 from F1 ep1 Q=+0.82) still trips: ISV[16] would have
been ~0.6 with α=0.05 EMA tracking, giving kill_floor ≈ 1.8, and
Q=+2.23 > 1.8 with ratio 2.7× trips both gates.
Touched: `training_loop.rs` (+70 LOC, two new use-list imports),
`docs/dqn-wire-up-audit.md` (Invariant 7 audit entry).
cargo check clean at 13 warnings (workspace baseline). No
fingerprint change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The IQN readiness gauge on `GpuDqnTrainer` is a streaming improvement
fraction `(iqn_loss_initial - iqn_loss_ema) / iqn_loss_initial`
(gpu_dqn_trainer.rs:5805-5818). `iqn_loss_initial` is captured once on
the first `update_iqn_readiness` call where it is still ~0 and never
resets — across folds it stays pinned to the fold-0 epoch-1 IQN loss.
The pinned device-mapped readiness slot is read by `c51_loss_kernel`
as the CVaR α and by IQN gradient-weight gating, so a stale anchor
either pins readiness near 1 (over-confident, full IQN gradient
weight on a still-recovering head) or near 0 (under-weighting a
converged head).
Fix:
- New `pub fn reset_iqn_readiness_state` on `GpuDqnTrainer` zeroes the
three coupled scalars (iqn_loss_initial / iqn_loss_ema /
iqn_readiness) and writes 0.0 through the device-mapped pinned host
slot — no HtoD copy, the mapping propagates the host write directly.
- New `pub(crate) fn reset_iqn_readiness_state` wrapper on
`FusedTrainingCtx` mirrors the existing `reset_eval_v_range_state`
pattern.
- Three new `FoldReset` registry entries (`isv_iqn_loss_initial`,
`isv_iqn_loss_ema`, `isv_iqn_readiness`) dispatch through one match
arm in `reset_named_state` (training_loop.rs).
`update_iqn_readiness` already handles `iqn_loss_initial < 1e-12` as
the bootstrap branch, so the new fold's first batch is recaptured as
the anchor on the first call after reset, by design.
Per `feedback_no_partial_refactor.md` — same fold-boundary
state-migration contract, all three coupled scalars migrate together.
This commit completes the 3-fix sequence (Fix 1 IQN target sync,
Fix 2 aux Adam states, Fix 3 IQN readiness) closing the fold-boundary
contract gap surfaced by the audit. Bug signature being resolved:
F0 ep5 Q=+0.79 (healthy)
F1 ep1 Q=+0.82 (boundary fine; c51_alpha warmup blends IQN low)
F1 ep2 Q=+2.23 (drift starts — IQN online↔target gap widens)
F1 ep3 Q=+4.05
F1 ep4 Q=+10.66 (geometric ~2.3×/epoch)
F1 ep5 NaN at step 5 (fp32 overflow past atom support)
Fix 1 (IQN target sync) is the load-bearing fix for the geometric
drift itself; Fix 2 prevents first-epoch Adam-step overshoot in
seven aux optimizers; Fix 3 closes the gauge-staleness path.
cargo check -p ml --lib clean at 13 warnings (workspace baseline).
cargo test -p ml --lib --no-run clean. All three existing
`state_reset_registry::tests` pass without modification.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extend the fold-boundary Adam reset contract to every auxiliary
optimizer added after the main DQN's `reset_adam_state` was wired:
- `GpuDqnTrainer::reset_adam_state` now also memsets q_attn /
sel / denoise / mamba2 / ofi_embed (m, v) buffer pairs and
zeroes the corresponding adam_step counters.
- New `pub(crate) fn reset_adam_state` on `GpuTlob` (memsets
adam_m/v, zeroes adam_step + writes 0 to the device-mapped
pinned host counter so the next adam_step kernel reads t=1
after `increment_adam_step`).
- New `pub fn reset_adam_state` on `GpuIqlTrainer` (same pattern,
m_buf/v_buf/adam_step/t_pinned).
- `FusedTrainingCtx::reset_for_fold` invokes the three new
helpers immediately after the existing IQN Adam reset block.
Both `gpu_iql` and `gpu_iql_low` are reset.
Per `feedback_no_partial_refactor.md` — the fold-boundary
state-migration contract has multiple consumers (main DQN trunk,
IQN head, aux Adam states across q_attn / sel / denoise / mamba2
/ ofi_embed / tlob / iql / iql_low). Adding `sync_target_from_online`
to the main DQN years ago without extending it to subsequently-
added optimizers left a "partial migration strictly worse than
the original" state. This commit + Fix 1 + Fix 3 close all three
remaining gaps surfaced by the fold-boundary audit.
All new resets follow the existing main-DQN pattern: DtoD memset-
zero only, pinned host counter written through the device-mapped
page (no HtoD/HtoH/DtoH paths). cargo check -p ml --lib clean
at 13 warnings (workspace baseline). cargo test -p ml --lib
--no-run clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add `GpuIqnHead::sync_target_from_online` (DtoD copy of online_params →
target_params) and call it from `FusedTrainingCtx::reset_for_fold`
alongside the existing IQN Adam reset.
Root cause: at fold boundary the main DQN does shrink-and-perturb +
hard target sync (gpu_dqn_trainer.rs:12495 — the latter explicitly added
to prevent fold-1 grad explosion). The IQN head was added later but
only had Polyak EMA `target_ema_update` — no hard-sync existed. Polyak
at τ=0.005/step closes ~0.5%/step, far too slow to close the
fold-boundary online↔target gap before inflated Bellman TD-error
compounds geometrically through the IQN backward pass.
Bug signature this resolves:
F0 ep5 Q=+0.79 (healthy)
F1 ep1 Q=+0.82 (boundary fine; c51_alpha warmup blends IQN low)
F1 ep2 Q=+2.23 (drift starts — c51_alpha ramp + IQN online↔target gap widens)
F1 ep3 Q=+4.05
F1 ep4 Q=+10.66 (geometric ~2.3×/epoch)
F1 ep5 NaN at step 5 (fp32 overflow past atom support)
Per `feedback_no_partial_refactor.md`: when a shared contract
(fold-boundary state migration) changes, every consumer must migrate
together. The main DQN sync was added explicitly — the IQN, TLOB, IQL,
and aux Adam states were added later but never extended to that
contract. This commit closes the IQN gap (primary). Aux Adam + IQN
readiness gaps follow in subsequent commits.
Touched:
- gpu_iqn_head.rs: new `sync_target_from_online` (mirrors gpu_dqn_trainer.rs:12495 exactly)
- fused_training.rs: call the new sync inside the existing IQN reset block
- docs/dqn-wire-up-audit.md: Invariant 7 entry covering Fix 1 of 3
cargo check -p ml --lib clean at 13 warnings (workspace baseline). No
fingerprint change. DtoD memcpy only — no HtoD/HtoH/DtoH paths.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Best-checkpoint save (val Sharpe improvement, ~30% of epochs in
convergent runs) blocked the epoch loop for 20-40s on each improvement:
serialize_model() chained N (~26) per-tensor DtoH downloads via
GpuTensor::to_host → memcpy_dtoh, each forcing an implicit stream sync
on a busy training stream. The DtoH chain also violates
feedback_no_htod_htoh_only_mapped_pinned.md (only cuMemHostAlloc
DEVICEMAP allowed for CPU↔GPU paths).
Plan B:
- Introduce snapshot_model_to_pinned (mod.rs): allocate one
MappedF32Buffer sized for all named weight slices concatenated,
cuMemcpyDtoDAsync each slice into the buffer's device pointer
(which aliases the host page), single stream sync, copy bytes out
to a Send + 'static Vec<u8>. One sync per snapshot, replaces N.
- serialize_snapshot_bytes (mod.rs): pure-CPU safetensors construction
from CheckpointSnapshot. Static — callable without &self, so the
worker can move the snapshot across thread boundary.
- handle_epoch_checkpoints_and_early_stopping on val-Sharpe
improvement: save_best_gpu_params (DtoD, fast) + snapshot to pinned
+ tokio::task::spawn_blocking the safetensors construction +
checkpoint_callback invocation. JoinHandle parked on
pending_checkpoint_handles. Training loop continues immediately.
- await_pending_checkpoint_handles drains in-flight workers at
training end (success branch + early-stop branches) and before
any synchronous cold-path checkpoint write to keep disk ordering
deterministic.
- F bound on train / train_walk_forward / train_fold_from_slices
gains + 'static so the callback can be moved into the worker.
All public callers already use 'static-compatible move closures
(test fixtures with shared mutable state migrate to Arc<Mutex<T>>).
Internal pipeline uses CheckpointCallbackHandle =
Arc<std::sync::Mutex<Box<dyn FnMut + Send + 'static>>> so the
same callback flows through multi-fold walk-forward into every
fold's worker.
- serialize_model itself rewritten via the snapshot path: the
no-DtoH rule now holds across ALL checkpoint paths (best, periodic,
early-stop, plateau-exhausted). The pre-existing GpuTensor::to_host
path is no longer reachable from the DQN trainer.
The audit's spec called for an mpsc channel(1) drop-old worker, but
the multi-fold + &mut F pre-existing API made the simpler
fire-and-forget spawn_blocking pattern a cleaner fit (Mutex
serialises any concurrent invocations; Vec<JoinHandle> drain at end
guarantees disk writes complete before the trainer returns). Same
overlap benefit (training rolls while serialize+disk run on a
blocking thread); upper bound on in-flight work is one-per-improved-
epoch which approximates the spec's depth=1 in realistic training
runs.
Per feedback_no_partial_refactor: every site that constructs a
checkpoint payload migrated in lockstep — best-improvement uses
the worker; periodic / plateau-exhausted / early-stop call the
shared Arc<Mutex<F>> handle inline. All paths read params via
snapshot_model_to_pinned, so the no-DtoH rule applies uniformly.
Test fixtures (8 .rs files) updated for the + 'static bound (move
closures + cloned PathBufs / Arc<Mutex<T>> for shared mutable state).
Verified: SQLX_OFFLINE=true cargo check --workspace --tests clean
(warnings unchanged from baseline). cargo test -p ml --lib --no-run
clean. No fingerprint change.
Wire-up audit entry extended with Plan B file:line edit sites
(rides under the same Async-validation overlap section started by
the companion Plan A commit).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The train-multi-seed-p526h repro (with MSE-clamp + PopArt-carry already
landed) reproduced a geometric Q-divergence in fold 1 that train_loss
can no longer hide:
F0 ep5 Q=+0.79 (healthy carryover)
F1 ep1 Q=+0.82 (boundary handoff fine)
F1 ep2 Q=+2.23 (2.7× jump — drift starts)
F1 ep3 Q=+4.05
F1 ep4 Q=+10.66
F1 ep5 NaN at step 5
A model with this Q dynamic is catastrophically unsafe for live trading:
Kelly cap floats with Q-confidence, so runaway Q drives oversized
positions and can blow up the strategy before any downstream safety
trips. This commit installs a production safety net — NOT a root-cause
fix — that hard-halts training the moment Q-drift is detected:
if |q_mean| > 2× prev AND |q_mean| > 1.5 production-unsafe floor
→ return Err, training halts, model rejected from deployment
Inserted in training_loop.rs immediately before the existing
`self.prev_epoch_q_mean = q_mean` update so the comparison uses the
same source-of-truth values that already feed downstream diagnostics.
The 2× ratio catches genuine geometric divergence (typical healthy
growth <30%/epoch); the 1.5 absolute floor prevents false positives
in early training where small Q magnitudes oscillate by large ratios
(verified: F0 ep4→ep5 ratio 5.3× would NOT trigger because |0.79|<1.5).
Both thresholds are numerical-stability bounds (Invariant 1 carve-
out), not tuned hyperparameters. Skips first epoch (no prev_q). Does
NOT skip fold-boundary epochs — those are exactly the failures
we're catching.
This is a safety net. The actual root cause is a fold-boundary reset
gap (which reset isn't firing correctly?). A systematic audit is
queued to identify it. Suspects: prev_grad_buf, PopArt GPU Welford
buffers (popart_count huge from fold 0 → new-fold stats track too
slowly), isv_q_abs_ref_* magnitude EMAs, spectral norm σ EMAs, TLOB
Adam state.
Per user's "can't happen at production" philosophy: production
deployments must have this safety net regardless of whether the
root-cause fix lands first.
Validation backtest was serialised on the training stream by passing
self.cuda_stream as parent_stream into GpuBacktestEvaluator::new (the
forked eval-stream still depends on training-stream completion, but the
metrics + plan_diag readbacks went through clone_dtoh / memcpy_dtoh,
each forcing an implicit sync that blocked behind the training stream's
pending work). The prior comment claimed single-stream eval was needed
"for cuBLAS determinism" — incorrect: cuBLAS bit-wise reproducibility
caveats apply per-handle, not across the process, and each stream
already owns its own PerStreamCublasHandles. Costs ~30-40s/epoch on L40S.
Plan A:
- Route eval through self.validation_stream (already forked at
constructor.rs:547 but unused on the eval path) and add a fresh
cuda_stream-recorded train_done_event so validation_stream +
evaluator's internal forked stream both wait on it before launching
eval kernels. GPU-side cuStreamWaitEvent — no CPU sync.
- Replace metrics_buf and plan_diag_buf in GpuBacktestEvaluator from
CudaSlice<f32> to MappedF32Buffer (cuMemHostAlloc DEVICEMAP). Kernels
write through the buffer's dev_ptr; host reads via volatile host_ptr
after a single eval_stream.synchronize() per evaluation. Replaces
the forbidden clone_dtoh / memcpy_dtoh path
per feedback_no_htod_htoh_only_mapped_pinned.md. The single sync
blocks only the eval stream — training rolls on uninterrupted.
- Update the misleading "single-stream cuBLAS determinism" comment
to explain the actual per-stream-handle architecture.
Per feedback_no_partial_refactor: every consumer of the
metrics_buf / plan_diag_buf contract migrated in lockstep (struct
fields, alloc, kernel-arg passing, host readout). Per-stream cuBLAS
handle parity with training preserved (val TLOB still uses
PerStreamCublasHandles::new(&eval_stream); evaluator forks its own
internal cuBLAS state from parent_stream).
Verified: cargo check -p ml --lib clean (13 warnings, workspace
baseline). cargo test -p ml --lib --no-run clean. No fingerprint
change — mapped-pinned is allocation-method orthogonal to kernel
buffer layout.
Companion commit lands Plan B (async best-checkpoint serialize).
Wire-up audit entry covers Plan A here; will be extended with Plan B
edit sites in the companion commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previous fixes (PopArt carry-forward + iqr floor in d710f9d50) reduced
fold-1 ep-1 train loss 16× but ep-2/ep-3 still oscillated 10k-200k
while val Sharpe stayed healthy at 76 — confirming the model itself
trains fine and the inflated train_loss is a pure measurement artefact.
Root cause in mse_loss_kernel.cu:457: the Bellman target
target_q = reward + γ·(1-done)·target_eq
is NOT clamped to the C51 atom support [v_min, v_max], whereas
`online_eq` is naturally bounded (it's an expected value of softmax
probabilities over atoms in that support). When PopArt-normalised
reward lands outside support — most acutely at fold boundaries when
cached robust stats lag the new fold's distribution by one epoch —
td = online_eq - target_q
grows to O(10^3–10^4), and
mse = 0.5·td²
inflates to 10^5–10^6 per-sample, dwarfing the (correctly bounded)
C51 distributional loss in the blended warmup total
(1-α)·MSE + α·C51
C51 already handles this in c51_loss_kernel.cu:247
t_z = fminf(fmaxf(t_z, v_min), v_max)
because the categorical projection requires it. MSE was missing the
matching clamp.
Fix: one-line `target_q = fminf(fmaxf(target_q, v_min), v_max)` in
mse_loss_kernel.cu between the ensemble-disagreement adjustment and
the td compute. Mirrors C51's clamp exactly.
No information lost: anything outside [v_min, v_max] is unrepresentable
in the categorical distribution anyway, so MSE warmup tracking targets
that the network categorically CANNOT learn produces a meaningless
reading. With the clamp, train loss stays bounded by support × atoms,
matching healthy fold-0 readings.
Per pearl_blend_formulas_must_have_permanent_floor.md (numerical-
stability bound carve-out under Invariant 1).
L40S 15-epoch repro #2 (train-multi-seed-kdkdv) revealed train_loss
exploded 1000-2000× in fold 1 (3.1 → 318k-694k) while val Sharpe
stayed healthy and CLIMBING (73 → 114). Train/val disconnect = pure
measurement bug, not real instability.
Mechanism: reset_for_fold() zeroed cached_iqr/cached_median at every
fold boundary. The `if cached_iqr > 0.0` guard at fused_training.rs:1220
then forced fold N+1's first epoch to the Welford GPU path. Welford
running stats inherited from fold N, combined with fold N+1's slightly
different reward distribution post-S&P + adversarial regime, produced
normalized rewards far outside C51 atom support [-50, 50] — categorical
loss readings 10^5x inflated. On rare timing-sensitive paths the
near-zero divide overflowed to Inf → NaN (the run-1 flagged=[2=on_b_logits,
3=mse_loss, 6=grad_buf, 7=save_current_lp, 8=save_projected]
diagnostic — what we caught was downstream of THIS root cause).
The diagnostic infrastructure from 756b1ef31 + 32e5375ac worked perfectly:
its precise signal of "on_v_logits clean, on_b_logits NaN, but loss is
also NaN, params clean pre-forward" surfaced the train/val disconnect
that pointed at the reward normalization bug.
Fix A (carry-forward, fused_training.rs:919-922):
Stop resetting cached_iqr / cached_median at fold boundary. Carry
fold N's final-epoch median/IQR forward as fold N+1's epoch-1
default. Same instrument, similar reward distribution between
adjacent walk-forward folds, so the carry is safe and gets replaced
by fresh stats at the end of fold N+1's epoch 1.
prev_popart_var still resets (sole consumer is tau-change detection;
a fresh fold counts as a change point regardless).
Fix B (permanent floor, dqn_utility_kernels.cu:1657):
Raise iqr fmaxf floor 1e-6 → 1e-4. With iqr=1e-6 a trade-exit
reward of 5.0 normalizes to 5e6 (vs post-fix 5e4) — much harder to
hit fp32 overflow. Defensive bound for genuinely-pathological iqr
paths (e.g. genuinely degenerate data quantiles), not a tuned
knob — Invariant 1 carve-out for numerical-stability bounds.
Per pearl_blend_formulas_must_have_permanent_floor.md (the same
recipe that resolved Kelly cap warmup + var_scale collapse before).
Resolves task #84 ("Fold-boundary state reset gap causes fold 1 grad
explosion").
Previous L40S 15-epoch repro fired the wired NaN diagnostic with
flagged=[2=on_b_logits, 3=mse_loss_scalar, 6=grad_buf, 7=save_current_lp,
8=save_projected] while value stream (flag 1) and pre-forward params
(flags 4-5) stayed clean. That isolates the corruption to the branch
advantage GEMM/activation but leaves one open question: is the GEMM
input (h_s2, shared between value and branch heads) finite or already
NaN?
Flag 12 = save_h_s2 covers the post-trunk activation. Outcomes:
- flag 12 clean + flag 2 NaN → branch GEMM overflowed (advantage-side
fp32 overflow under post-S&P + adversarial reward stress).
- flag 12 NaN → trunk encoder corrupted upstream of both heads.
Mechanically: one new check_nan_f32 call against save_h_s2 in
run_nan_checks_post_forward, names array slot 12 updated rsv12 →
save_h_s2 in training_loop.rs error message. Same ~5µs cost as the
other 12 checks. Slots 13-15 still reserved.
Per `feedback_no_partial_refactor.md` (extending the NaN diagnostic
contract atomically) and `feedback_no_stubs.md` (rsv12 was a real
reservation; now consumed).
Migrates the remaining 4 forward sites:
- VSN Linear_1 (x6 groups): adds 6 per-group RELU_BIAS shapes to the
relu_bias cache; migrates sgemm_f32_ldb + launch_add_bias_relu_f32_raw
pairs to sgemm_f32_fused_relu_bias.
- VSN Linear_2 (x6 groups, single shape (1, B, VSN_HIDDEN, VSN_HIDDEN)):
migrates to sgemm_f32_fused_bias.
- GLU value head + gate (launch_vsn_glu_branch Steps 3-4): migrates to
sgemm_f32_fused_bias with per-stream workspace selection.
Also fixes a partial-refactor leak in forward_online_raw — the
sequential branch fallback (distinct_branches=false) was still on the
unfused sgemm_f32 + launch_add_bias_relu_f32_raw pair while its
multi-stream sibling already used the fused epilogue. Now both code
paths take the fused-first contract.
Layout fingerprint unchanged (no params buffer changes).
Migrates 6 branch FC adv_logits sites (decoder_forward_only,
forward_online_raw multi-stream + sequential, forward_online_f32
multi-stream + sequential, forward_target_raw multi-stream) from
sgemm_f32(_branch) + launch_add_bias_f32_raw pairs to
sgemm_f32_fused_bias.
The branch FC output projects [B, adv_h] -> [B, branch_size_d * num_atoms]
across 4 distinct branch shapes, all pre-cached. Multi-stream sites use
the per-branch workspace (branch_workspace_ptrs[d]) to avoid contention
with concurrent branch streams. Drops up to 4 add_bias_f32 launches per
training step (one per branch direction) on each of {online, target,
f32-collector} forwards.
Migrates 4 v_logits sites (forward_online_raw, forward_online_f32,
forward_value_head ensemble heads, forward_target_raw) from
sgemm_f32 + launch_add_bias_f32_raw pairs to the new
sgemm_f32_fused_bias helper. The value-head output projects
[B, value_h] -> [B, num_atoms] (C51 atom logits), no ReLU on output.
Drops 4 standalone add_bias_f32 kernel launches per training step.
Migrates 8 trunk sites (4 online + 4 target) from
sgemm_f32 + launch_add_bias_f32_raw pairs to the new
sgemm_f32_fused_bias helper. Covers all four GRN Linear layers per
encoder: h_s1 Linear_a, h_s1 Linear_b, h_s2 Linear_a, h_s2 Linear_b.
Linear_residual has no bias term and stays a plain sgemm_f32_ldb.
Drops 8 standalone add_bias_f32 kernel launches + 8 memory
round-trips per (online + target) forward step.
Adds gemm_cache_bias field, create_cached_fwd_gemm_desc_bias builder,
and sgemm_f32_fused_bias helper. Pre-creates descriptors for every
BIAS-only output shape used in the forward pass. No call sites
migrated yet — subsequent commits flip individual sites to use the
fused helper.
Mirrors the established RELU_BIAS infrastructure pattern: deterministic
algorithm selection, per-call bias-pointer wiring, Err-on-missing-cache
contract for graceful fall-back.
Fold 1 of train-multi-seed-72fl6 hit NaN at step 5 with `flagged=[]` —
diagnostic told us nothing because:
1. The 8 NaN-check kernels in `run_nan_checks_*_forward` were never
invoked from production training. Allocated-zero flags read back
as zeros; halt-on-NaN reported "no buffer flagged" while loss was
demonstrably NaN via host-side pinned readback.
2. The error message labels (training_loop.rs:1853) referenced
`bf16_params` — dead code from before the bf16→TF32 switch — and
the format string did not match the actual kernel index→buffer
mapping.
Wired the existing NaN check kernels into `submit_post_aux_ops` so
they run every step inside the captured parent graph (post_aux child).
Flags persist across steps within a fold so the FIRST buffer to go
NaN remains visible at host-readback time; reset is host-side at fold
boundary in `reset_for_fold`.
Extended coverage beyond the original 8 buffers (states, on_v_logits,
on_b_logits, mse_loss, params, grad_buf, save_current_lp) with 4
loss-component buffers most likely to break under post-S&P + adversarial
regime stress:
Flag 8 save_projected C51 target distribution
Flag 9 moe_gate_softmax MoE gate over 8 experts
Flag 10 aux_nb_loss_scalar aux next-bar MSE
Flag 11 aux_rg_loss_scalar aux regime CE
Flag buffer grown 8 → 16; slots 12-15 reserved for future expansion
(CQL / IQN / per-branch backward).
Updated training_loop.rs error message: 16 named slots matching actual
kernel layout, per-flag indexed name in flagged list, dropped stale
`bf16_params` / `_bf16` suffixes, explicit hint when `flagged=[]`.
Cost per step: ~12 single-block reductions × ~5µs = ~60µs. <0.1%
overhead at the 258ms/step measured on L40S — pinpoints source
buffer on next NaN halt.
Per `feedback_no_legacy_aliases.md`, `feedback_no_stubs.md`,
`feedback_trust_code_not_docs.md`.
TLOB attention GEMMs (M=TLOB_OUT=16, K=TLOB_IN=32, N=batch) are too
narrow for cuBLAS-Lt's heuristic search; cublasLtMatmulAlgoGetHeuristic
returns "no algo found" → GpuTlob::new fails → graceful-degrade hides
the bug per feedback_no_hiding.md. The previous symbol-rename fix
(7208836d1) merely surfaced this deeper API-choice problem.
Per NVIDIA best practice, cuBLAS-Lt is for tensor-core-optimised GEMMs
at scale (M,K,N ≥ 64-128); classic cublasSgemm_v2 is for general-
purpose any-shape GEMMs. TLOB's tiny attention dims belong in the
second category. Migrating all 8 TLOB GEMM call sites:
- 3× fwd Q/K/V proj (M=16, N=batch, K=32)
- 1× fwd O proj (M=16, N=batch, K=16)
- 3× bwd dW_QKV (M=16, N=32, K=batch)
- 1× bwd dW_O (M=16, N=16, K=batch)
- 1× bwd dX_O (M=16, N=batch, K=16)
Single API, single code path, no try/catch. Removed graceful-degrade
wrap in fused_training.rs and trainer/metrics.rs — TLOB is no longer
optional. Field type changed from Option<GpuTlob> to GpuTlob; all five
consumer sites (forward/backward in fused_training, mean_max in
training_loop, val-side init+sync in metrics) migrated together per
feedback_no_partial_refactor.md.
PerStreamCublasHandles registry gained a classic_handle field +
classic_for(stream) accessor mirroring the existing lt_for(stream) API.
Both handles share the same 32 MB workspace, both bound to their
stream + workspace at creation time. Default-stream classic handle is
created in PerStreamCublasHandles::new; side-stream handles are
provisioned lazily alongside their cuBLAS-Lt sibling inside
classic_for/lt_for/pre_register_stream. Classic handles default to
CUBLAS_TF32_TENSOR_OP_MATH (matching gpu_curiosity_trainer's existing
convention; same TF32 path as the cuBLAS-Lt CUBLAS_COMPUTE_32F_FAST_TF32
compute type).
Verification:
- SQLX_OFFLINE=true cargo check --workspace — clean
- cargo test -p ml --lib --release gpu_tlob — new
tlob_sgemm_parity_with_cpu_reference test passes; verifies all 8
GEMMs (forward Q/K/V/O, backward dX_O, dW_O, dW_Q/K/V) match a CPU
sgemm reference within 2e-3 absolute on batch_size=64 synthetic
OFI input
- dqn-wire-up-audit.md updated; new pearl
pearl_cublas_lt_vs_classic_sgemm.md captures the rule
The cuBLAS-Lt path remains in use for the larger gpu_attention module
(trunk attention, dims 128+) where tensor-core throughput pays off.
This is the standard split: Lt for big, classic for small.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
gpu_tlob.rs:232 was loading symbol `attn_adam_update` from the
attention_backward_kernel cubin, but the kernel is defined as
`attn_adam_kernel` (attention_backward_kernel.cu:70); gpu_attention.rs:281
already loads the same symbol with the correct name. Mismatch caused
GpuTlob::new to fail with "named symbol not found" on every workflow
init, suppressed by the surrounding "training continues without TLOB"
warning. Per feedback_no_hiding.md, graceful-degraded modules ARE the
ghost-feature pattern.
With the symbol now loading, a separate cuBLAS-Lt heuristic failure
surfaces ("tlob heuristic (fwd_q): no algo found") — the TLOB GEMM
shape M=TLOB_OUT=16, K=TLOB_IN=32, N=batch is too narrow for cuBLAS-Lt
heuristics to find an algo. That's a real follow-up (pad dims, classic
sgemm fallback, or custom small-M kernel) but is independent of and
strictly upstream of this commit's symbol-name fix. The previous symbol
error was hiding the heuristic error.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Val backtest subsamples every Nth bar at metrics.rs:517-518 but
annualization_factor at gpu_backtest_evaluator.rs:674 was still computed
from the full bars_per_day. Per audit
`docs/lookahead-bias-audit-2026-04-28.md` §5, this inflates reported
sharpe_annualised by sqrt(N) — for N=4, factor of 2.
Fix: thread the subsample stride through `GpuBacktestConfig` as a new
`val_subsample_stride: u32` field (default 1 = no subsampling). The
constructor divides `bars_per_day` by the sanitized stride before
sqrt, so reported Sharpe / Sortino / Calmar all reflect the effective
sampling cadence.
The val call site at metrics.rs uses a single `VAL_SUBSAMPLE_STRIDE`
const that drives both the `if i % stride != 0 { continue; }` predicate
AND the config field — single source of truth means future cadence
changes automatically keep the annualization in sync. Per-bar Sharpe
trajectory is unchanged.
Other GpuBacktestConfig consumers (hyperopt adapters,
evaluate_baseline, gpu_backtest_validation tests) all use
`..Default::default()` and inherit `val_subsample_stride: 1`.
Regression test added:
test_annualization_factor_compensates_for_subsample_stride asserts the
ratio f1/f4 = 2.0 for stride=4, plus stride=0 sanitization.
dqn-wire-up-audit.md updated to reflect the new contract:
metrics.rs::VAL_SUBSAMPLE_STRIDE is the single source of truth paired
with GpuBacktestConfig::val_subsample_stride.
NOTE TO OPERATORS: Sharpe values reported in HEALTH_DIAG val [...]
lines post-merge will be ~2× LOWER than pre-merge for the same model.
This is the corrected number, not a regression.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
