Bug audit Pattern 2 — 6 SP15 launchers in gpu_dqn_trainer.rs were doing
load_cubin + load_function PER CALL inside the launcher body, called
from gpu_experience_collector.rs's per-rollout-step body (~thousands
of times per epoch x 4096 envs x 1000 timesteps).
Same architectural bug class as commits 5d63762ab (bn_tanh_concat_dd)
and 1396b62ec (sp15_baseline + cost_net). load_cubin/load_function
are host-side driver API calls — not capturable inside graph capture,
and prone to CUDA_ERROR_ILLEGAL_ADDRESS in subprocess child contexts
(documented failure mode in 1396b62ec).
Migration:
- launch_sp15_dd_state, launch_sp15_dd_state_reduce,
launch_sp15_dd_trajectory_decreasing, launch_sp15_alpha_split_producer,
launch_sp15_final_reward, launch_sp15_plasticity_injection —
signatures changed to accept &CudaFunction parameter.
- Collector struct gains 6 new pre-loaded CudaFunction fields,
populated once in collector::new() (mirrors SP14 EGF kernel loading
added in commit 09202aa99).
- All 6 hot-path call sites updated to pass &self.exp_<kernel>_kernel
reference (5 in collect_experiences_gpu, 1 in plasticity-injection
trigger path).
- Oracle tests in sp15_phase1_oracle_tests.rs pass through a small
load_sp15_kernel test helper that inlines the per-call load (graph
capture isn't a concern in oracle scaffolds; production callers use
struct-cached handles via the collector).
- docs/dqn-gpu-hot-path-audit.md gains Fix 29 documenting the
pattern, the 6 migrated sites, deferred evaluator launchers, and
out-of-scope orphan launchers.
Per feedback_no_partial_refactor: all 6 launchers + their callers
migrate atomically. Per pearl_no_host_branches_in_captured_graph:
zero load_cubin in collector per-rollout-step body after this commit.
Deferred (separate scope): launch_sp15_sharpe_per_bar and
launch_sp15_position_history_derivation are evaluator-path callers
(GpuBacktestEvaluator) — fixing them requires adding fields to a
different struct, deferred to a separate atomic commit. Orphan
launchers (regret_signal, cooldown) have no production callers; left
untouched.
Cubin static visibility: all 6 affected SP15 cubins were already
pub static (collector imports work without visibility promotion).
Verification: cargo check clean (only pre-existing warnings); 7/7
sp14_oracle_tests pass; 36/36 sp15_phase1_oracle_tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Production training run train-multi-seed-bn42w showed label_scale=5481
(raw_close magnitude) at epoch 4 vs smoke's ~25 at the same commit
29b1d34c6. The 5-layer data-loading defense + DBN spread filter + target
stride/column fix all in place — yet column 0 of next_states_buf still
sees raw-price-magnitude values in production but not in the
multi_fold_convergence test path.
This one-shot diagnostic fires once at the first training batch and
prints:
- aux_nb_label_buf stats (the kernel input; should be ~0.8 z-norm)
- next_states_buf col 0 sample (what strided_gather reads)
- features_raw_cuda col 0 sample (the source feature buffer)
- targets_raw_cuda col 2 sample (raw_close — the suspected leak)
- Verdict line interpreting the 4 stats
Three numbers will pin the source. If aux_buf ~5500: raw_close leaked
into next_states. If features_raw_cuda col 0 ~5500: the writer didn't
normalize. If targets_raw_cuda col 2 ~5500 but aux_buf ~0.8: only
production reads raw_close, smoke doesn't.
One-shot via static AtomicBool. Pre-graph-capture, no perf impact.
Removal gate: delete once the leak source is identified and fixed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Direct inspection of ES.FUT_2024-Q1.dbn.zst (databento python client, offline
kubectl-cp from PVC) pinpoints the source of the 8,799 corrupt bars that
sanitize_bars (Fix 25) caught at the bar-level gate.
Q1 file content:
Outrights (correct): ESH4/ESM4/ESU4/ESZ4/ESH5 — 43,353 records (76.87%)
price range $5,063–$5,478
Spreads (poison): ESH4-ESM4/ESM4-ESU4/... — 13,048 records (23.13%)
price range $47.30–$219.70
Calendar spreads trade at the price DIFFERENCE between adjacent contracts
(~$60-150 cost-of-carry roll basis). Databento's stype_in=parent resolution
for ES.FUT returns BOTH outrights AND every spread combination in the same
DBN stream. The legacy decoder keyed only by ts_event + dedup-by-volume;
during low-volume overnight windows + active rollover periods, spread bars
beat the outright on volume and survived the dedup. 780 spread bars
survived in Q1 alone; ~8,800 across 2024-2026.
Fix: build dbn::TsSymbolMap from metadata once, resolve each record's
instrument_id → symbol, skip any symbol containing `-` (spread separator).
Same-ts dedup-by-volume continues to handle legitimate front/back-month
overlap among outrights.
Adds `time = "0.3"` to ml/Cargo.toml (dbn::TsSymbolMap uses time::Date).
Why complementary to the Fix 25 sanitize_bars gate:
- Spread filter (this commit) catches the cause precisely by symbol pattern,
but only for instruments where parent-symbol resolution is the source.
- sanitize_bars catches the symptom universally by close-ratio bound, but
can't distinguish a low-basis spread from a fast-moving outright in
extreme cases.
Both layers active: spread filter dispatches at parser level, sanitize as
defense-in-depth backstop for unknown future contamination shapes.
Validation: `cargo check -p ml --example train_baseline_rl` clean.
Refs: Bug 2 chain (#191, #194, label_scale=5443 leaks), today's
sanitize_bars find (Fix 25). Closes the contamination-source investigation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Stops chasing one-off corruption bugs. Three+ historical fixes patched
specific writers (#191 fxcache column-0, Bug 1 target schema,
label_scale=5443 leaks, today's state[0] heavy-tail). Each new
corruption shape found the next hole. This installs a structural
defense so corruption is REJECTED at the data-loading boundary
regardless of source.
Five independent layers, each mandatory:
1. Bar-level sanity (baseline_common::sanitize_bars):
drop bars with non-positive OHLC, high<low, non-finite values, or
close/prev_close outside [0.5, 2.0]. Catches DBN parse glitches,
broker tick errors, near-zero-open bars at source.
2. safe_log_return result clamp (extraction.rs:1246):
ratio.ln().clamp(±0.1). Real-market 1-bar log returns rarely
exceed ±0.05; ±0.1 traps every legitimate move while rejecting
the corruption shape (corrupt bar with bar.open ≈ 0 → ln = -30
→ previously normalized to -30000-magnitude state[0] outliers).
3. validate_features pre-norm bound (extraction.rs:538):
|val| ≤ 5.0 post-extraction. Pre-norm features come from
safe_normalize ([0,1]/[-1,1]), safe_clip (max ±3), or clamped
log-returns (±0.1); ±5 catches extractor invariant breaks.
4. NormStats::normalize post-norm clamp (walk_forward.rs:688):
((val - mean) / std).clamp(±20.0). Even if upstream produces
outliers, every value uploaded to GPU is bounded.
5. Shared validate_normalized_features gate (walk_forward.rs):
single source-of-truth invariant enforced at THREE sites:
- fxcache fast path (after discover_and_load)
- DBN fallback (after normalize_batch)
- precompute writer (before fxcache write — never persist
a poisoned cache)
Removed: DIAG_BUG2 + DIAG_BUG2_v2 one-shot diagnostics
(~125 lines of host-side download + outlier scan in
training_loop.rs). Replaced by structural defense — instrumentation
isn't needed when corruption can't reach state[0].
FEATURE_SCHEMA_HASH auto-bumps via build.rs FNV-1a hash over
SCHEMA_FILES (extraction.rs included). All pre-fix .fxcache files
on PVC are invalidated at load time; ensure-fxcache regen produces
clean cache with new clamps applied.
Why this finally closes the chapter: per-writer fixes are reactive
(land after corruption hits prod). Boundary validation is
proactive — every future regression to extraction or normalization
trips the gate at load, not at epoch 5 of a 50-epoch run. The 5
layers are independent: a bug in any one leaves the others as
backstop.
Validation: cargo check -p ml --all-targets --offline clean.
NormStats unit tests (walk_forward.rs:706+) still pass — clamp +
validate are additive; existing test inputs are well within bounds.
Refs: SP5 Bug 2 (state[0] std=570 outliers on smoke-test-xb78r),
historical #191#210#214#193#195 chains.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Augments the existing DIAG_BUG2 one-shot block with two narrower scans
that disambiguate where state[0] heavy-tail (mean=6e-3, std=570,
mean_abs=20 over 3200 samples on smoke-test-xb78r) actually originates.
H1 (data-source corruption): scan features_buf.host_ptr for bars where
|feat[bar*42 + 0]| > 100. host_ptr aliases the same physical memory
the kernel reads (mapped-pinned) — direct read of the data
state_gather sees. Reports up to 10 outlier bar indices + mean/std/
max_abs over up to 2M bars.
H2 (kernel/gather bug): scan gpu_batch.states/next_states (already
DtoH-downloaded) for sample indices where |state[i*sd + 0]| > 100.
Reports up to 10 outlier sample indices.
Verdict line logged: feat-outliers nonempty → H1; state-outliers
nonempty AND feat-outliers empty → H2.
Context: smoke-test-xb78r at HEAD cba9f25ed went through DBN fallback
(train_baseline_rl.rs:611, "Loaded N bars (ts to ts)" message) — no
fxcache file exists on training-data PVC. DBN-fallback applies z-score
normalization (line 633) which has small stddev for log-return columns;
a single bar with bar.open ≈ 0 produces ln(ratio) = -30 → normalized
= -30000. Few bars in 700K could carry this signature.
Triggers once via static AtomicBool. Pre-graph-capture, no perf impact.
Cleanup gate: remove DIAG_BUG2 block once Bug 2 root cause is fixed
(likely a tighter clamp inside safe_log_return for log-return columns).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two related blocks landing together (Bug 2 diagnostic forces the structural
conversion of touched files; pre-commit guard at 5275932f4 enforces zero
*_via_pinned helpers per `feedback_no_hiding`).
== Bug 2 instrumentation (training_loop.rs after collect_experiences_gpu) ==
One-shot AtomicBool-gated DIAG_BUG2 dump of first 5 samples × 5 cols of
both gpu_batch.states and gpu_batch.next_states + col-0 mean/std/mean_abs
across the full batch. Prints once at first rollout — pre-graph-capture so
no hot-path impact. Interpretation key embedded:
mean_abs ~0.001 = normalized log_return ✓
mean_abs ~5000 = raw price ✗ (Bug 2 confirmed)
Resolves the smoke-vs-production state[0] divergence question that pure
static code reading couldn't pin: production aux_label_scale=5300 traces
back through gather kernel to "feat[0] of next_states_buf" but the fxcache
shows feat[0] stddev=1.0 (z-normalized log_return). Either next_states_buf
is populated from a different source than fxcache feat[0], or some kernel
mutates state[0] post-gather. The diagnostic prints both states (post-gather)
and next_states (post-shift) to disambiguate.
== Structural conversion: targets_raw_cuda + features_raw_cuda + 10 sites ==
Field types (DQNTrainer in trainer/mod.rs:621/624):
Option<CudaSlice<f32>> → Option<MappedF32Buffer>
init_gpu_raw_buffers_from_slices (training_loop.rs): clone_to_device_f32_via_pinned
calls (lines 1308/1312) replaced with `MappedF32Buffer::new + write_from_slice`.
Consumer signatures (3 functions across 2 files):
- gpu_experience_collector::collect_experiences_gpu(market_features_buf,
targets_buf): &CudaSlice<f32> → &MappedF32Buffer (each)
- gpu_experience_collector::launch_timestep_loop: same
- gpu_experience_collector::compute_difficulty_scores(targets): same
- decision_transformer::build_dt_trajectories(features_gpu, targets_gpu): same
Launch sites: `.arg(buf)` → `.arg(&buf.dev_ptr)` at all 5 launch_builder
invocations in gpu_experience_collector.rs and the 2 in build_dt_trajectories.
== Cold-path init conversions (forced by guard touching shared files) ==
gpu_dqn_trainer.rs (9 sites → 4 buffer-migration groups):
- spec_u_s1/v_s1/u_s2/v_s2 (4 buffers, explicit)
- spec_u/v macro pairs (alloc_spec_pair! body, expands to ~22 buffers)
- graph_params (cross-branch graph message-passing, 60 floats)
- denoise_params (diffusion Q-refinement MLP, 1800 floats)
- qlstm_weights (xLSTM mLSTM-cell, 528 floats)
gpu_experience_collector.rs (1 site):
- upload_ofi_features → ofi_gpu field type Option<CudaSlice<f32>> → MappedF32Buffer
Inherited from prior worktree-agent attempts (compile clean, included here):
- sel_clip_buf in gpu_dqn_trainer.rs
- RmsNormWeightSet γ buffers in gpu_weights.rs
PPO trainer (trainers/ppo.rs) deferred — not on eval-collapse hot path.
== Validation ==
cargo check -p ml --offline: clean
pre-commit hook (check_no_dtod_via_pinned + Invariant 7 + GPU hot-path guard): pass
Bug 2 diagnostic will fire on next L40S run and print state[0] stats for
first batch. If mean_abs ~0.001 → state[0] is correctly normalized and Bug 2
is elsewhere (maybe label_scale_ema initialization). If mean_abs ~5000 →
state[0] really is raw price; the rollout state-builder is the bug.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bug 1 of the eval-Hold-collapse diagnosis. The fxcache target schema
documented in experience_kernels.cu:1556 + cuda_pipeline/mod.rs:508 specifies:
target[0] = preproc_close — log-return-normalized close (network input)
target[1] = preproc_next — log-return-normalized next close
target[2] = raw_close — raw price for portfolio simulation
target[3] = raw_next — raw price
target[4] = raw_open — raw price
target[5] = mid_price_open — MBP-10 midpoint (fallback raw_open)
Both writers — `precompute_features.rs:360` and `data_loading.rs:510` —
violated the contract by storing raw OHLCV close prices in slots [0:1].
Empirical fxcache inspection: target[0..4] mean=$5967, stddev=$582 (raw
prices throughout). The raw-price values at target[0:1] were never directly
consumed by training (production aux head reads next_states[i][0] = MARKET
feat[0] = log_return), but they corrupted any code reading targets per the
documented contract.
Both writers now compute (raw_curr / prev_close).ln() and (raw_next /
raw_curr).ln() for the preproc columns. FXCACHE_VERSION bumped 7→8 to
invalidate existing caches and trigger ensure-fxcache regen.
A second bug — eval label_scale=5300 (raw price magnitude) at production
binary despite source state[0] tracing back to z-normalized log_return —
remains unresolved. Bug 2 instrumentation lands in the next commit; that
runtime trace will pin which production-binary code path injects raw_close
into state[0] post-gather.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two related changes installing the structural guard against the SP6 Pearl 5
IQN τ failure mode (root cause fixed at facbf76eb for that one site) and
removing the only remaining orphan callers of the broken pattern.
The bug class. The mapped_pinned::{upload,clone_to_device}_{f32,i32}_via_pinned
helpers are named to suggest "no HtoD per feedback_no_htod_htoh_only_mapped_pinned"
but their bodies do MappedXBuffer::new() + memcpy_dtod_async() +
stream.synchronize(). The DtoD copy and synchronize are both forbidden
inside CUDA Graph capture (CUDA_ERROR_STREAM_CAPTURE_INVALIDATED) and add a
host stall otherwise. The canonical pattern is MappedXBuffer stored directly
+ write_from_slice + kernel reads via .dev_ptr, used by SP4 portfolio_state
and SP6 IQN τ at facbf76eb.
Guard. New check_no_dtod_via_pinned in pre-commit-hook.sh rejects any
staged .rs file calling upload_(f32|i32)_via_pinned or
clone_to_device_(f32|i32)_via_pinned, except mapped_pinned.rs itself. Per
feedback_no_hiding: no suppression marker. Also fixes a pre-existing
silent-skip bug: the gpu-hotpath-guard.sh invocation used
$(cd "$(dirname "$0")" && pwd) which resolved to .git/hooks/ (the symlink's
directory) instead of scripts/, so the guard never ran. Replaced with
readlink -f "$0" + an explicit "guard missing" error branch — silent skip
is worse than no guard.
Orphan deletion. gpu_her.rs carried legacy relabel_batch, generate_random_donors
(CPU), HerBatch, slice_clone_f32, slice_clone_i32 — zero production callers
(verified via grep). Production uses relabel_batch_with_strategy +
generate_random_donors_gpu. The orphan held the only upload_i32_via_pinned
callers in the codebase; per feedback_no_hiding the right fix is delete.
Scope. Eliminates 2 of 47 production *_via_pinned call sites. Remaining 45
across 14 files are cold-path init — graph-capture-fragile and host-stalling
but not breaking operationally. Guard enforces no new calls; existing 45
migrate in subsequent atomic per-buffer commits. After all 45 are converted,
the four helpers themselves get deleted from mapped_pinned.rs.
Validation. Smoke smoke-test-82fjk at facbf76eb succeeded — magnitude
differentiation restored (q_full=0.462 > q_half=0.409 > q_quarter=0.350 vs
baseline frozen Pascal-triangle 0.225/0.280/0.495), eval distribution
unfrozen (eq=0.596, eh=0.404, ef=0.000 vs baseline single-action collapse).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pearl 5's online_taus/target_taus/cos_features were declared as
CudaSlice<f32> (device-only), populated via upload_f32_via_pinned
which does a DtoD copy from a separate mapped-pinned staging buffer.
The DtoD inside CUDA Graph capture triggers
CUDA_ERROR_STREAM_CAPTURE_INVALIDATED and the 'continuing ungraphed'
fallback observed in smoke-test-hhr5q.
This violates feedback_no_htod_htoh_only_mapped_pinned: the rule is
mapped-pinned (cuMemHostAlloc DEVICEMAP) for ALL CPU↔GPU paths. No
DtoD copies, no HtoD copies, no exceptions.
Fix: convert all 3 buffers (online_taus, target_taus, cos_features)
to MappedF32Buffer per-branch [MappedF32Buffer; 4] arrays. Host writes
go directly to host_ptr; IQN kernel reads dev_ptr of the same memory
— no copy step at all. The mem::swap pattern is replaced with pure
selection: activate_branch_taus sets active_branch_idx; kernel launch
sites index online_taus_per_branch[active_branch_idx].dev_ptr.
Eliminates upload_f32_via_pinned calls for these buffers entirely.
Refresh becomes a host write to mapped-pinned host_ptr at fold
boundary; subsequent kernel launches see the write through the
mapped-pinned coherence guarantee after stream sync.
cargo check + cargo build --release + cargo test --lib (sp4 sp5
state_reset_registry: 13/13) all clean. Sanity grep for
upload_f32_via_pinned in gpu_iqn_head.rs returns zero.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
nsys profile of multi_fold_convergence on L40S identified compute_expected_q
as the #3 GPU consumer at 12.9% (207ms / 1382 calls — ~150 µs per call,
compute-bound at typical batch=8192 × num_atoms=51). The kernel computed each
per-action softmax over atoms in 3 passes (max → sum_exp → normalize+expected
+var+entropy+util), re-reading v_row[z]+adv_a[z] 3 times per atom for each of
the 4 × ~3.25 = 13 actions per sample.
Replaced with online softmax (running-max + running-sum) so expected_q,
sum_z_sq, and entropy all accumulate in a single forward pass over atoms.
Atom utilisation still needs a 2nd pass because it depends on the final
S = sum exp(logit-M) being known to compute prob_z = exp(logit_z-M)/S for
the threshold test.
Inner loop reduction: 3 → 2 passes, ~33% fewer global loads of branch
advantage logits (which dominate because v_row[z] is reused across 13
(action, branch) pairs and may stay in L1, while adv_a[z] flips per
action and is pure global).
Online accumulation pattern (Page-Olshen):
M, S, TZ, TZ², TLM = -inf, 0, 0, 0, 0
for z in 0..num_atoms:
logit = v_row[z] + adv_a[z]
if logit > M:
scale = exp(M - logit) # ≤ 1, no overflow
S, TZ, TZ², TLM *= scale
M = logit
w = exp(logit - M)
S += w
TZ += w * z_val
TZ² += w * z_val²
TLM += w * (logit - M)
expected_q = TZ / S
sum_z_sq = TZ² / S
entropy = log(S) - TLM / S # analytical: -sum(p log p)
Correctness: bit-stable when atoms processed in fixed order
(z = 0..num_atoms-1). The analytical entropy form is mathematically
identical to -sum(p log p):
-sum(p log p) = log(S) - (1/S) * sum(exp(logit-M) * (logit-M))
= log(S) - TLM / S
The previous code's `if (prob > 1e-10f)` underflow guard is no longer needed:
exp(logit - M) underflows cleanly to 0 when logit << M, and the analytical
form does not multiply tiny probs by very negative logs. Bit-stable per
feedback_stop_on_anomaly.md — no fuzzy tolerance change.
Atom-stat block-sum reduction unchanged (warp shfl + shared-mem bank,
deterministic).
Expected wall-clock saving: ~33% of 207ms = ~70ms across the smoke run; on
production batch=8192 × 1382 calls per fold, roughly proportional. Lower-bound
estimate — at low batch (smoke) the kernel may be launch-bound rather than
load-bound. nsys re-profile after deployment will quantify.
ABI unchanged; no Rust caller changes required. Per Invariant 7 the audit
doc dqn-gpu-hot-path-audit.md is updated with Fix 19 entry.
Build: SQLX_OFFLINE=true cargo check -p ml --lib clean (12 warnings, baseline).
Tests: cargo test -p ml --lib --no-run clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
nsys profile of multi_fold_convergence on L40S identified backtest_state_gather as
the #1 GPU consumer at 37.2% (596 ms / 60,588 calls — kernel launch latency
dominated compute) and the per-step TLOB cuBLAS gemvx calls as #2 at 25%
(242K calls). Both share the same per-step amplification: chunk_len=512 separate
gather launches + 512 separate TLOB.forward calls (4 SGEMMs each) per chunk
before any Q-values can be computed.
This commit replaces the per-step gather + DtoD pattern with a single batched
launch, and reuses the same chunked buffer for a single chunk-wide TLOB forward.
Per-chunk launch reduction: from 2*chunk_len + chunk_len*7 to 1 + 7 for the
gather+TLOB phase (4608 -> 8 with chunk_len=512, a 576x reduction).
New kernel `backtest_state_gather_chunk` (experience_kernels.cu):
- Writes [chunk_len, N, padded_sd] directly into chunked_states_buf
- chunk_len * N threads, 1 thread per output row
- Mathematically identical to per-step gather: portfolio_buf and plan_isv_buf
are CONSTANT within a chunk (env_step + plan_state_isv update at chunk
boundary only). Each thread reads independent feature offsets, no atomics,
no reordering.
Chunked val TLOB (gpu_backtest_evaluator.rs + metrics.rs):
- Val TLOB instance now sized to DQN_BACKTEST_CHUNK_SIZE * n_windows via new
GpuBacktestEvaluator::val_tlob_batch_size() helper.
- submit_dqn_step_loop_cublas calls tlob.forward(chunked_states, batch) ONCE
per chunk on the chunk-wide buffer instead of chunk_len times on states_buf.
- Partial last chunks reuse the same buffers (forward(b) accepts any
b <= construction_batch).
Borrow restructure:
- Removed top-of-function `let ch_states = self.chunked_states_buf.as_ref()?`
binding (TLOB needs &mut). Replaced with per-chunk ch_states_base raw u64
device pointer extracted in tight scope, reused by Phase 1 (gather) and
Phase 2+3 (compute_q_values_to + last_step_states_ptr). The pointer is
stable across the chunk because the Option<CudaSlice<f32>> does not
reallocate.
Per-step gather kernel `backtest_state_gather` retained unchanged for
evaluate() / evaluate_ppo() / evaluate_supervised() paths that still need
a single-step writer (closure-based callers with no chunked buffer).
Audit doc dqn-gpu-hot-path-audit.md updated with Fix 18 entry per Invariant 7.
Build: SQLX_OFFLINE=true cargo check -p ml --lib clean (12 warnings, baseline).
Tests: cargo test -p ml --lib --no-run clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After Fix 1..16 migrated all 80+ production callers off
`super::htod_f32` and `super::clone_htod_f32`, the helper bodies in
`cuda_pipeline/mod.rs:129-145` had zero non-test consumers. Deleted
both function definitions per `feedback_no_legacy_aliases.md` (no
deprecated wrappers).
Per `feedback_no_partial_refactor.md` (when a shared contract is
deleted, every consumer migrates together — including tests), the
two surviving test-block callers in `gpu_tlob.rs::tests` (lines
1017 and 1132) are migrated to `mapped_pinned::upload_f32_via_pinned`
in the same commit. The other test-only callers in
`signal_adapter.rs::tests`, `gpu_action_selector.rs::tests`, and
`cuda_pipeline/mod.rs::tests` use bare `stream.memcpy_htod` /
`stream.memcpy_stod` against the cudarc handle directly (not the
deleted helpers) — no change needed.
A docstring was added at the deletion site recording when and why
the helpers were removed, pointing future readers at the canonical
replacements `mapped_pinned::clone_to_device_f32_via_pinned` and
`mapped_pinned::upload_f32_via_pinned`.
Final state of the HtoD migration sequence:
- production callers of `stream.memcpy_htod` / `memcpy_stod`: 0
- production callers of `htod_f32` / `clone_htod_f32`: 0
- helper definitions: removed from `mod.rs`
docs/dqn-gpu-hot-path-audit.md updated with Fix 17 entry.
cargo check -p ml --lib clean at 12 warnings.
cargo check -p ml --tests clean at 23 warnings (12 lib duplicates +
11 test-specific, baseline unchanged).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The two `crate::cuda_pipeline::clone_htod_f32` callsites in
`training_loop.rs::set_raw_market_data` (lines 1250 and 1254) are
rewritten to `mapped_pinned::clone_to_device_f32_via_pinned` per
`feedback_no_htod_htoh_only_mapped_pinned.md`.
These run once per fold immediately before the step loop begins; the
`_raw` suffix indicates the un-normalized arrays kept on-GPU for the
eval-time critic.
Error-type continuity: the helper already returns `Result<_, String>`,
which feeds `anyhow::anyhow!("…raw upload: {e}")` directly — the prior
.map_err wrapper is preserved verbatim. 1:1 path swap.
docs/dqn-gpu-hot-path-audit.md updated with Fix 16 entry.
cargo check -p ml --lib clean at 12 warnings.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The remaining `super::clone_htod_f32` call in `upload_ofi_features`
(`:4258`) is rewritten to `mapped_pinned::clone_to_device_f32_via_pinned`
per `feedback_no_htod_htoh_only_mapped_pinned.md`. Called from
training_loop's data-load phase before each fold's training run, so
the OFI tensor (4M bars × 32 dims) now stages through mapped-pinned
+ DtoD just like all other large data uploads.
Error-type shift: `mapped_pinned::clone_to_device_f32_via_pinned`
returns `Result<CudaSlice<f32>, String>` whereas `clone_htod_f32`
returned `Result<CudaSlice<f32>, MLError>`. Wrapped with
`.map_err(|e| MLError::ModelError(format!("upload_ofi_features: {e}")))`
to preserve the call-site label in error messages.
docs/dqn-gpu-hot-path-audit.md updated with Fix 15 entry.
cargo check -p ml --lib clean at 12 warnings.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Residual COLD ctor sites missed by the original audit are now migrated
off explicit HtoD per `feedback_no_htod_htoh_only_mapped_pinned.md`:
- `:9922` sel_clip_buf 1-element init (sigmoid head clip-norm seed)
- `:10075-10078` spectral norm init_u_s1/v_s1/u_s2/v_s2 (4 calls)
- `:10095-10096` `alloc_spec_pair!` macro body — both u/v init uploads
inside the macro now go through `upload_f32_via_pinned`; the macro's
`$lbl_u`/`$lbl_v` are reused in the error-message format so per-pair
failures stay diagnostically distinct
- `:11276` graph_params_host (60 floats: cross-branch graph message
passing weights)
- `:11330` denoise_params_host (1800 floats: 2-step diffusion Q-refinement)
- `:11476` qlstm_weights_host (528 floats: QLSTM Xavier init)
All sites use `mapped_pinned::upload_f32_via_pinned` (the canonical
mapped-pinned + DtoD staging helper). The helper returns
`Result<_, String>` whereas this constructor returns
`Result<_, MLError>`, so each site wraps the error via
`.map_err(|e| MLError::ModelError(format!("<site> upload via pinned: {e}")))`.
Site labels preserved so backtraces remain readable.
docs/dqn-gpu-hot-path-audit.md updated with Fix 14 entry.
cargo check -p ml --lib clean at 12 warnings.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Agent 1 — bulk COLD/WARM migration across init/loader files.
Sites: 39 production HtoD calls migrated to MappedF32Buffer / MappedI32Buffer /
new MappedU32Buffer. Cross-file infrastructure includes new staging helpers
(clone_to_device_{f32,i32}_via_pinned, upload_{f32,i32,u32}_via_pinned),
MappedU32Buffer type, and host_slice_mut accessor on MappedF32/U32Buffer.
Per feedback_no_htod_htoh_only_mapped_pinned.md — first of 3 parallel batches.
# Conflicts:
# docs/dqn-gpu-hot-path-audit.md
HOT sites:
- L2992 hindsight bar_indices: was `alloc + memcpy_htod(total i32s)` on
every collect_experiences_gpu when hindsight_fraction > 0. Added
persistent `bar_indices_pinned: MappedI32Buffer` (capacity =
alloc_episodes * alloc_timesteps * 2) allocated at constructor.
Per-call host writes go through host_ptr; relabel kernel reads via
dev_ptr u64.
- L3186 feature_mask: was alloc + memcpy_htod each epoch at t==0.
Promoted `feature_mask_buf` to `Option<MappedF32Buffer>`. Reallocates
only when mask size changes; CPU writes via host_ptr, state_gather
reads via dev_ptr.
- L3816 update_per_sample_support: per_sample_support_buf is read-only
by the kernel — promoted to MappedF32Buffer (kernel args now
dev_ptr u64). per-epoch tile fill becomes a direct host_ptr write.
WARM sites:
- L3097 episode_starts upload: episode_starts_buf is GPU-mutated by
domain_rand_episode_starts so must remain CudaSlice. Stage via
MappedI32Buffer + memcpy_dtod_async through new helper
`upload_host_to_cuda_i32_via_pinned`.
- L2009 upload_expert_actions: promoted expert_actions_gpu from
Option<CudaSlice<i32>> to Option<MappedI32Buffer>. Direct host_ptr
write replaces alloc + memcpy_htod.
COLD sites:
- L1076,3848 portfolio_states init/reset (GPU-mutated): use
upload_host_to_cuda_f32_via_pinned (mapped-pinned staging + DtoD).
- L1093 epoch_state init (GPU-mutated): replace clone_htod_f32 with
explicit alloc_zeros + upload_host_to_cuda_f32_via_pinned.
- L1341 saboteur_base init (GPU-mutated): use the same helper.
- L2784 trade_stats_buf pre-reduction zero: replaced htod_f32 of an
all-zeros vec with stream.memset_zeros — fully GPU-side, no PCIe.
Imports: added DevicePtrMut for memcpy_dtod_async pointer extraction.
Per `feedback_no_partial_refactor.md`, every consumer of the migrated
buffers is updated in this commit:
- per_sample_support_buf kernel args at L3426 (compute_expected_q) and
L3546 (quantile_q_select) now consume dev_ptr u64.
- feature_mask_buf accessor at L3195 reads via .dev_ptr.
- expert_actions_gpu field type change is contained (no external
consumers in this crate).
cargo check -p ml --lib: 11 warnings (unchanged from prior commit on
this branch; 2 below 13-warning baseline because two unrelated trivial
warnings disappeared as a side-effect of the refactor).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cold-path constructor sites (2):
- portfolio_states init: htod_f32 → upload_f32_via_pinned
- rng_states init: memcpy_htod → upload_u32_via_pinned
Warm per-collect sites (2):
- episode_starts_buf: memcpy_htod → upload_i32_via_pinned
- barrier_config: htod_f32 → upload_f32_via_pinned
Warm per-epoch reset (2 sites + persistent staging refactor):
The Vec<f32>/Vec<u32> host staging fields (portfolio_init_staging,
rng_seed_staging) replaced with persistent MappedF32Buffer /
MappedU32Buffer allocated once at construction. Resets fill via
host_slice_mut() (zero-allocation, zero-HtoD); a single async DtoD
seeds the persistent device-resident portfolio_states / rng_states
(which the kernel mutates each step and must remain in VRAM).
Adds host_slice_mut() accessor on MappedF32Buffer and MappedU32Buffer
to support structured per-element writes against the mapped pages.
Audit row appended (Fix 11) in docs/dqn-gpu-hot-path-audit.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- 1 constructor site (rng_states u32 init) rewritten to
mapped_pinned::upload_u32_via_pinned.
- 2 warm-path sites (source_indices, donor_indices in relabel_batch)
rewritten to mapped_pinned::upload_i32_via_pinned. The warm sites
re-allocate pinned staging per relabel call — acceptable on this
cold path (HER relabel runs once per epoch).
Adds MappedU32Buffer type to mapped_pinned.rs mirroring MappedI32Buffer,
plus upload_u32_via_pinned helper. Manual Debug impl so warning count
stays at 13.
Audit row appended (Fix 10) in docs/dqn-gpu-hot-path-audit.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
HOT site: `train_step_gpu` was uploading the 4-byte Adam step counter via
`stream.memcpy_htod(&[step], &mut scratch.adam_t)` on every backward pass.
Promoted `DtScratch.adam_t` from `CudaSlice<i32>[1]` to `MappedI32Buffer(1)`
allocated once in `alloc_dt_scratch`. Per-step CPU writes go through
`host_ptr`; the Adam kernel reads via `dev_ptr` u64. Trivial fix, massive
frequency — runs once per gradient step.
Other sites:
- L478 ctor params init: `params` is mutated by Adam (must stay CudaSlice).
Upload via mapped-pinned staging + `memcpy_dtod_async` through new
helper `upload_host_to_cuda_f32`.
- L501 ctor wd_mask init: wd_mask is read-only by Adam. Promoted to
`MappedF32Buffer`; CPU writes 1.0s through host_ptr, kernel reads dev_ptr.
Caller switched from `wd_mask.raw_ptr()` to `wd_mask.dev_ptr`.
- L1105 trajectory build episode_starts: one-time, replaced `clone_htod`
with `MappedI32Buffer` (direct host write, kernel read via dev_ptr).
- L1147 trajectory build ep_rewards: one-time, replaced `htod_f32` with
`MappedF32Buffer` (direct host write, return_to_go reads via dev_ptr).
Also added `#[allow(missing_debug_implementations)]` to MappedI32Buffer
and MappedF32Buffer in `mapped_pinned.rs` — raw pointers + CUdeviceptr
have no useful Debug impl, and the workspace `missing_debug_implementations`
lint now reaches them through the public DT type that holds them.
Audit doc: docs/dqn-gpu-hot-path-audit.md updated.
cargo check -p ml --lib: 11 warnings (down from 13 baseline; net-zero
regression, 2 minor unrelated warnings vanished as a side-effect).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- 4 stream.clone_htod sites in IQN constructor: cos_features precompute,
online_taus + target_taus tile broadcast, init_iqn_xavier_weights
upload — all rewritten to mapped_pinned::clone_to_device_f32_via_pinned.
- clone_cuda_slice helper rewritten to a single DtoD copy via the
existing super::clone_cuda_slice_f32 helper. Previously did
device→host→device, double-violating both no-DtoH and no-HtoD rules.
Sole callsite is constructor seeding of target params from online.
Audit row appended (Fix 9) in docs/dqn-gpu-hot-path-audit.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- trainers/ppo.rs: 2 sites (features, targets in set_raw_market_data)
- hyperopt/adapters/ppo.rs: 3 sites (features, targets in upload path;
action_indices in run_gpu_backtest forward closure)
- hyperopt/adapters/mamba2.rs: 1 site (gpu_to_stream)
All clone_htod_f32 and bare stream.clone_htod calls rewritten to
mapped_pinned::clone_to_device_{f32,i32}_via_pinned.
Audit row appended (Fix 7) in docs/dqn-gpu-hot-path-audit.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
HOT site at simulate_batch_gpu — `actions: &[i32]` was uploaded
per-batch via stream.memcpy_htod into a persistent CudaSlice. Promoted
the field to `MappedI32Buffer` allocated once at constructor; per-call
host writes now go through `host_ptr` (no HtoD copy). Kernel arg switched
to passing `dev_ptr` as a u64.
The two COLD sites (ctor init_state, reset init_state) writing into the
GPU-mutated `portfolio_state_buf` cannot replace the destination buffer
type (env_step kernel writes back to it). Use a mapped-pinned staging
buffer + memcpy_dtod_async via a new `upload_portfolio_state` helper.
Retains zero direct HtoD copies on the only path the violation rule
permits per `feedback_no_htod_htoh_only_mapped_pinned.md`.
Sites migrated:
- L122 ctor init_state (htod_f32 -> staging+DtoD via upload_portfolio_state)
- L174 simulate_batch_gpu actions upload (HOT: persistent MappedI32Buffer)
- L296 reset init_state (htod_f32 -> staging+DtoD)
Audit doc: docs/dqn-gpu-hot-path-audit.md updated with new MIGRATED entries.
cargo check -p ml --lib: 13 warnings, baseline preserved.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Migrates the 3 clone_htod_f32 calls in walk-forward GPU data upload
(features, targets, OFI) to mapped_pinned::clone_to_device_f32_via_pinned.
Audit row appended (Fix 6) in docs/dqn-gpu-hot-path-audit.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Migrates the 7 in-file callers of clone_htod_f32 in DqnGpuData/PpoGpuData
constructors and helpers to mapped_pinned::clone_to_device_f32_via_pinned.
Sites: features+targets+ofi upload (DqnGpuData::upload_slices,
::upload_ofi), portfolio scratch (build_batch_states), htod_copy_into,
PpoGpuData::upload.
Helper bodies htod_f32 / clone_htod_f32 left intact at lines 129-145 —
other parallel agents are still migrating files that call them.
Audit row appended (Fix 5) in docs/dqn-gpu-hot-path-audit.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds clone_to_device_{f32,i32}_via_pinned and upload_{f32,i32}_via_pinned
in `mapped_pinned.rs`. They allocate a temporary MappedXxxBuffer
(cuMemHostAlloc DEVICEMAP), write the payload via host_ptr, then async
DtoD into a target CudaSlice<T>. Used to migrate cold-path uploads off
explicit cudaMemcpy HtoD per `feedback_no_htod_htoh_only_mapped_pinned.md`
without changing struct field types where the kernel mutates the buffer
every step (e.g. portfolio_buf, weight buffers consumed by Adam).
Audit table updated in docs/dqn-gpu-hot-path-audit.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The async-diag merge (3c0d26292, building on d9cb14f1b + 673b04a8d) wired the
GPU side of cross-stream eval pipelining correctly (training stream `cuda_stream`,
eval stream `validation_stream`, `cuStreamWaitEvent` barrier on `train_done_event`)
but left a host-side `stream.synchronize()` inside
`gpu_backtest_evaluator::launch_metrics_and_download` that blocked the CPU thread
for the FULL eval drain (~25-30s/epoch on L40S). The host thread is the same one
that submits the next epoch's training kernels via `run_full_step` — so until eval
drained, training submission was gated on it, defeating the dedicated
`validation_stream`.
Fix: split metrics readback into record/consume halves at the buffer-readback layer:
* `launch_metrics_and_record_event` — submits the metrics kernel + DtoD-async
copies of `actions_history_buf` / `intent_mag_buf` / `picked_action_history_buf`
into mapped-pinned mirrors, then records `eval_done_event`. Returns immediately.
* `consume_metrics_after_event` — `event.synchronize()` (the SOLE host wait per
epoch boundary), then `read_volatile`s mapped-pinned `metrics_buf`. After the
event syncs, the four action-distribution helpers read directly from the
now-coherent mapped-pinned mirrors — eliminating four synchronous DtoH copies
that violated `feedback_no_htod_htoh_only_mapped_pinned.md`.
Caller migration (per `feedback_no_partial_refactor.md`):
* `evaluate_dqn_graphed` (existing public API) becomes a thin sync wrapper:
launch_async → consume. ABI unchanged.
* New `evaluate_dqn_graphed_async` for the pipelined path.
* `evaluate` / `evaluate_ppo` / `evaluate_supervised` migrated inline to
launch+consume (no caller pipelines them).
* `compute_validation_loss` replaced by `launch_validation_loss` +
`consume_validation_loss`. Training loop now consumes pending at epoch start
(cached_async_val_loss = epoch N-1's val_sharpe) and launches at epoch end
(sentinel-only return). Final post-loop consume drains the LAST epoch's launch
so smoke tests / hyperopt that read `last_eval_direction_dist` see the most
recent epoch's data.
Mathematical identity preserved: every val_sharpe consumed is bit-identical to
what the prior synchronous flow would have produced for the same epoch — only
the timing of the host parse differs (one epoch later, matching the existing
`cached_async_val_loss` lag semantics; HEALTH_DIAG `val [...]` / `val_dir_dist` /
`val_picked_dir_dist` emit moves with the consume).
Audit: docs/dqn-gpu-hot-path-audit.md Fix 4. Out of scope: periodic chunk-level
`stream.synchronize` calls in `submit_dqn_step_loop_cublas` (lines ~1596 + ~1863)
for kernel-error detection — host-blocking but smaller in aggregate; future work.
Workspace baseline preserved: 13 warnings.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
atoms_update_kernel.cu: 4-block kernel (grid=(4,1,1), block=(256,1,1)) replaces
the CPU loop that launched adaptive_atom_positions 4× per branch. Each block reads
ISV v-range slots [23..31) and spacing_raw params[52..56); computes softmax +
cumsum; writes atom_positions_buf[branch * num_atoms]. Same formula as the
existing adaptive_atom_positions kernel in experience_kernels.cu.
Deleted recompute_atom_positions and warm_start_atom_positions from GpuDqnTrainer
and their fused_training.rs delegates. step_atom_positions now calls
launch_atoms_update. training_loop.rs: recompute_atom_positions → launch_atoms_update;
warm_start_atom_positions + compute_reward_quantiles block removed.
AtomsMonitor reads ISV[V_CENTER_DIR_INDEX=23] as representative scalar;
diagnose() exposes all 8 v-range slots [23..31). state_reset_registry.rs
"follow-up task" descriptions updated for all 4 adaptive slots.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
kelly_cap_update_kernel.cu: single-thread cold-path kernel aggregates Kelly
win/loss stats across all n_envs from portfolio_states[n_envs, PS_STRIDE].
Computes mean half-Kelly × health-coupled safety multiplier (0.5+0.5*health)
and writes ISV[KELLY_CAP_EFF_INDEX=44]. Matching Bayesian priors from
trade_physics.cuh (prior_wins=2, prior_losses=2) prevent cold-start collapse.
GpuExperienceCollector::portfolio_states_dev_ptr() added — exposes device
pointer for the portfolio_states buffer without a GPU→CPU roundtrip.
GpuDqnTrainer::launch_kelly_cap_update takes ps_dev_ptr + n_envs.
FusedTrainingCtx delegates to trainer. training_loop.rs launches at epoch
boundary after gamma_update (requires both fused_ctx and gpu_experience_collector).
KellyCapMonitor reads ISV[44] for HEALTH_DIAG via AdaptiveMonitor trait.
state_reset_registry.rs description updated; audit docs updated (Invariant 7).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
tau_update kernel computes Polyak-EMA tau from ISV[EPOCH_IDX=39,
TOTAL_EPOCHS=40, LEARNING_HEALTH=12] with cosine schedule + health-coupled
floor (rises to 0.01 at full collapse). Single-thread cold-path kernel
writes ISV[TAU_EFF_INDEX=42].
TauMonitor is a read-only observer exposing tau_eff, epoch_idx, total_ep,
health, progress, fire_rate in DiagSnapshot.
Wires EPOCH_IDX_INDEX CPU-born input write at epoch start via existing
write_isv_signal_at accessor (pinned-memory zero-copy). Also launches
tau_update kernel at epoch boundary before any tau consumer.
Adds read_isv_signal_at() to GpuDqnTrainer (symmetric counterpart to
write_isv_signal_at).
Deleted: compute_cosine_annealed_tau free function (fused_training.rs),
apply_health_coupled_tau_floor method (GpuDqnTrainer), last_tau_eff
cached field + last_tau_eff() delegate. 4 consumer sites in
fused_training.rs now read ISV[TAU_EFF_INDEX] directly.
Tests: 3 monitor unit tests pass. cargo check -p ml at 8-warning baseline.
Plan 1 Task 13. Spec §4.C.6 (2026-04-24 revision).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Enumerated every DtoH/HtoD/memcpy call in the DQN hot path
(run_full_step → child graphs). Classified 55 call sites:
31 OK-pinned, 20 COLD-PATH, 4 MIGRATED.
Fix 1 (gpu_dqn_trainer.rs): removed dead cuMemcpyDtoHAsync_v2 in
run_causal_intervention_unconditional — result (causal_mean_scratch)
was never consumed; now stays on device.
Fix 2 (fused_training.rs + training_loop.rs): compute_iqr() was
called inside submit_aux_ops (captured aux_child graph). Its sync
cuMemcpyDtoH_v2 cannot be graph-captured — silently ran only during
capture, then HtoD replayed stale IQR data on every step. Removed
from submit_aux_ops; added refresh_iqn_iqr() called once per epoch
in process_epoch_boundary.
Fix 3 (gpu_iqn_head.rs): tau_buf (CudaSlice<f32>) + tau_host (f32) +
cuMemcpyHtoDAsync_v2 each step replaced by tau_pinned (*mut f32) +
tau_dev_ptr (u64) via cuMemAllocHost_v2(DEVICEMAP) +
cuMemHostGetDevicePointer_v2. CPU writes *tau_pinned = tau; EMA
kernel reads via tau_dev_ptr — zero PCIe overhead. Drop updated.
Audit table populated in docs/dqn-gpu-hot-path-audit.md.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Plan 1 Task 1. Creates the five audit docs plus config/metric-bands.toml
that track Invariants 2, 7, 8 per the DQN v2 spec, and extends the
pre-commit hook with two checks:
- component-adding commits must touch an audit doc (Invariant 7)
- added code may not contain TODO/FIXME/XXX/HACK/TBD/unimplemented!/
todo! markers (Invariant 9)
Tests: manually verified by staging a TODO-marked file; commit
rejected with the correct error message.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
