The L40S smoke template uses CARGO_TARGET_DIR=/cargo-target on a
persistent PVC, so every smoke probe builds incrementally on top of
artefacts left by prior probes. File deletions (e.g., regime_conditional.rs
in ff00af68a) can leave dangling rmeta/object references that perturb
downstream codegen between bisect runs — making the bisect result
contingent on which order probes were submitted in, not on the source.
Adds an optional `clean-cache` parameter (default `false`) which runs
`cargo clean -p ml -p ml-dqn --release` before the compile step. Other
crate artefacts (ml-core, ml-supervised, etc.) stay cached so the wipe
is bounded — fresh ml/ml-dqn compile in ~3-5 min vs ~30+ min full clean.
Use case: re-running a52d99613 + ff00af68a with `--clean-cache` to
verify the bisect under controlled build conditions. If the
broken/clean status flips with clean cache, the regression is
build-state-dependent rather than source-line; if it reproduces, the
source regression is real and bisect is definitive.
scripts/argo-smoke.sh exposes `--clean-cache` flag passing through
to the workflow parameter.
Adds a no-wrapper L40S smoke runner alongside the nsys/sanitizer
templates. Same compile + checkout + data-mount layout, no profiler
binary, no sanitizer instrumentation, no MinIO artefact upload — just
a plain test execution against the full training-data PVC.
Use case: validate a smoke test passes on the real 27-month dataset
when local data is too short (laptop only has 1 quarter of MBP-10/
trades, fxcache truncates below the 10-month minimum).
- infra/k8s/argo/smoke-test-template.yaml: WorkflowTemplate
smoke-test, entrypoint smoke-run, default test
multi_fold_convergence::test_multi_fold_convergence.
- infra/k8s/argo/kustomization.yaml: register the new template.
- infra/k8s/argo/argo-workflow-netpol.yaml: extend the
sanitizer/nsys NetworkPolicy podSelector to include smoke-test
(identical egress requirements: git fetch, ci-builder pull,
training-data PVC).
- scripts/argo-smoke.sh: thin wrapper mirroring argo-nsys.sh /
argo-sanitizer.sh (--multi-fold default, --test, --ref, --watch).
Verified: kustomize dry-run clean, kubectl apply -k creates the
template + reconfigures the netpol live in foxhunt namespace.
22 of 147 snapshot fields populated (ISV-mirror copies, no reductions
or masking). Foundation for Phases 2B-2E (4 more kernels) with stable
type contract: HealthDiagSnapshot 147-word layout + GpuHealthDiag
orchestrator with launch_isv_mirror() entrypoint.
* health_diag_kernel.cu (NEW): inline WORD-index table mirrors Rust
struct field order; static_assert(WORD_TOTAL == 147) keeps offset
map locked to Rust struct size.
* gpu_health_diag.rs (NEW): GpuHealthDiag orchestrator owns the
MappedHealthDiagSnapshot.
* GpuDqnTrainer integration: launch_health_diag_isv_mirror() +
health_diag_snapshot() accessor.
Subsequent kernels need parallel-shadow validation against existing
CPU paths because masking semantics in 2B-2E are non-trivial.
Phase 2A of the HEALTH_DIAG GPU port. Lands the simplest member of the
kernel family (single-block single-thread ISV scalar copies) plus the
orchestrator scaffolding (cubin loader, mapped-pinned snapshot owner,
launcher) that subsequent phases will reuse.
Why simplest first: 5 kernels each with different masking semantics
landing in one commit risks silent numerical mismatches that downstream
parsers (aggregate-multi-seed-metrics.py, smoke summarisers) would
mis-parse. Per feedback_no_quickfixes.md, kernel-by-kernel with
parallel-shadow validation. Phase 4 deletes CPU paths in one commit
once all kernels are bit-identical.
What this kernel does: copies 22 ISV signal-bus slots into the matching
HealthDiagSnapshot fields — reward-component EMAs (slots 63-68), VSN
attention focus EMAs (87, 88), target-drift EMAs (92, 93), aux-head
loss EMAs (113, 114, 117), MoE expert utilisation (118-126), gate
entropy (126), and λ_eff (128). 22 of 147 snapshot words populated
(reward_split[6] + noisy_vsn[2] + noisy_drift[2] + aux[3] + aux_moe[10]).
Producer-only — no consumer reads health_diag.snapshot() yet; Phase 3
wires the call site, Phase 4 makes the snapshot the sole emit source.
Architecture rules upheld:
- No HtoD / DtoH (writes through mapped-pinned device pointer).
- No atomicAdd (single-thread kernel; family-wide rule for 2B-2E too).
- No DtoD memcpy on the diag path (kernel pointer-load + store).
- Kernel WORD-index table inline; static_assert pins WORD_TOTAL=147 in
lockstep with snapshot_size_is_stable test.
- ISV indices passed as kernel args (not #define'd) so the kernel
never embeds the numbering — single source of truth lives in
gpu_dqn_trainer.rs constants.
Changes:
- new crates/ml/src/cuda_pipeline/health_diag_kernel.cu (311 lines).
- new crates/ml/src/cuda_pipeline/gpu_health_diag.rs (190 lines)
holds GpuHealthDiag orchestrator (cubin module + isv_mirror_kernel
handle + MappedHealthDiagSnapshot).
- mod.rs: pub mod gpu_health_diag.
- build.rs: registers health_diag_kernel.cu in kernels_with_common.
- gpu_dqn_trainer.rs:
- imports super::gpu_health_diag::GpuHealthDiag;
- adds health_diag: Option<GpuHealthDiag> field next to aux_heads_fwd;
- constructor calls GpuHealthDiag::new() after the cubin loads;
- adds pub fn launch_health_diag_isv_mirror() (uses compile-time
ISV index constants, single source of truth);
- adds pub fn health_diag_snapshot() accessor.
- docs/health_diag_inventory.md: Phase 2A entry + Phase 4 deletion
list pinned per Invariant 7.
- docs/dqn-wire-up-audit.md: new row for health_diag_kernel.cu under
"CUDA Kernels (.cu files)" — classified Pending wire-up (Phase 3).
Verification:
- cargo check -p ml: 0 new warnings (12 lib warnings pre-existing,
none touch health_diag/gpu_health_diag).
- cargo test -p ml --no-run: clean.
- cargo test -p ml --lib cuda_pipeline::health_diag::tests: 3/3 pass
(snapshot_size_is_stable, default_is_zeroed, alignment_is_4_bytes
— the WORD_TOTAL=147 static_assert in the kernel matches the host
size_of test bit-for-bit).
Scope discipline (per prompt's "if kernel #3 takes too long, stop"):
This commit lands the orchestrator scaffolding + simplest kernel only.
Phases 2B-2E (eval-histogram, q-mag-reduce, per-sample-reduce,
finalise) intentionally deferred — masking semantics for q_mag_reduce
in particular need careful per-CPU-getter inspection before kernel-side
implementation. Producer-only by design; no production caller of
launch_health_diag_isv_mirror in this commit (matches AuxHeadsForwardOps
landing pattern from Plan 4 Task 6 Commit A).
Phase 0 (commit 333ea7184): exhaustive inventory of every numeric value
emitted across the 7 HEALTH_DIAG sites in training_loop.rs and metrics.rs.
Classified each as GPU-already / CPU-bound / Mixed / Host-state /
ISV-already. Aggregate cost: ~50-150 MB/epoch DtoH + ~70 sec/epoch CPU
compute (the original observation that motivated the port). Inventory
lives at docs/health_diag_inventory.md.
Phase 1 (commit 60fd7de96): foundation only — no kernel, no producer
launch, no CPU code deletion yet.
* HealthDiagSnapshot #[repr(C)] POD with 147 fields (all f32 or u32)
covering every numeric value in the HEALTH_DIAG line family. Field
order matches the existing log format strings (downstream parsers
like aggregate-multi-seed-metrics.py depend on this).
* Controller fire-bools promoted from u8 to u32 to avoid #[repr(C)]
padding mismatch when followed by f32 fields. ~24-byte cost.
* MappedHealthDiagSnapshot wrapper around cuMemHostAlloc(DEVICEMAP|
PORTABLE) + cuMemHostGetDevicePointer_v2, mirroring the
MappedF32Buffer pattern.
* Default impl via MaybeUninit::zeroed() (allocator-free, valid
bit-pattern).
* 3 unit tests pinning size (588 bytes), alignment (4 bytes),
zeroed-default — all passing on local CPU host.
Phases 2-5 deferred (kernel family + wiring + CPU emit rewrite + old
code deletion). The 5 kernels each read different device buffers with
different masking semantics — getting any one wrong would silently
produce numerically-different HEALTH_DIAG output that downstream
parsers would mis-parse. Per the prompt's 'don't wedge yourself'
guidance and feedback_no_quickfixes.md, the safer move was to land
the typed wrapper cleanly so subsequent commits can iterate kernel
by kernel against the actual buffer-by-buffer reduction.
L40S nsys profile (commit 0d630c799) identified three hot kernels:
- backtest_state_gather: 37.2% / 60k calls
- cuBLAS gemvx: 25.0% / 242k calls (val TLOB SGEMMs)
- compute_expected_q: 12.9% / 1.4k calls
Kernels #1 + cuBLAS #2 (folded together because most gemvx calls were
val TLOB SGEMMs) — commit 072ca4568:
* Added backtest_state_gather_chunk kernel that writes [chunk_len,
n_windows, padded_sd] directly into chunked_states_buf in a single
launch with chunk_len*n_windows threads (1 thread per output row).
* Refactored submit_dqn_step_loop_cublas Phase 1: per-step launch +
DtoD copy loop replaced by a single batched call. Per-chunk launch
count: 2*chunk_len -> 1 (1024x reduction at chunk_len=512).
* Val TLOB now sized to DQN_BACKTEST_CHUNK_SIZE * n_windows so it
runs once per chunk on chunked_states_buf at batch=chunk_len*n_windows
instead of chunk_len separate calls. Per-chunk cuBLAS reduction:
chunk_len*4=2048 -> 4 (~500x).
* Single-step backtest_state_gather retained for evaluate() /
evaluate_ppo() / evaluate_supervised() paths.
Kernel #3 (compute_expected_q) — commit 5f26c5751:
* Replaced 3-pass softmax (max -> sum_exp -> normalize+expected+entropy)
with online softmax (Page-Olshen running-max + running-sum). Inner
atom loop: 3 -> 2 passes, ~33% fewer global loads of adv_a[z] per
(action, branch) pair.
* Bit-stable when atoms processed in fixed order. Underflow safety
preserved analytically (no prob*log(prob) edge case).
* ABI unchanged; no Rust caller migration needed.
Deferred: Q/K/V SGEMM fusion (~50% of remaining ~500 val + ~300k
training calls) gated by tlob_sdp_forward kernel ABI changes
(stride parameter or unpacking step). ~200 LOC follow-up; recommend
re-profiling post-deploy to confirm training-side TLOB still hot.
nsys profile of multi_fold_convergence on L40S identified compute_expected_q
as the #3 GPU consumer at 12.9% (207ms / 1382 calls — ~150 µs per call,
compute-bound at typical batch=8192 × num_atoms=51). The kernel computed each
per-action softmax over atoms in 3 passes (max → sum_exp → normalize+expected
+var+entropy+util), re-reading v_row[z]+adv_a[z] 3 times per atom for each of
the 4 × ~3.25 = 13 actions per sample.
Replaced with online softmax (running-max + running-sum) so expected_q,
sum_z_sq, and entropy all accumulate in a single forward pass over atoms.
Atom utilisation still needs a 2nd pass because it depends on the final
S = sum exp(logit-M) being known to compute prob_z = exp(logit_z-M)/S for
the threshold test.
Inner loop reduction: 3 → 2 passes, ~33% fewer global loads of branch
advantage logits (which dominate because v_row[z] is reused across 13
(action, branch) pairs and may stay in L1, while adv_a[z] flips per
action and is pure global).
Online accumulation pattern (Page-Olshen):
M, S, TZ, TZ², TLM = -inf, 0, 0, 0, 0
for z in 0..num_atoms:
logit = v_row[z] + adv_a[z]
if logit > M:
scale = exp(M - logit) # ≤ 1, no overflow
S, TZ, TZ², TLM *= scale
M = logit
w = exp(logit - M)
S += w
TZ += w * z_val
TZ² += w * z_val²
TLM += w * (logit - M)
expected_q = TZ / S
sum_z_sq = TZ² / S
entropy = log(S) - TLM / S # analytical: -sum(p log p)
Correctness: bit-stable when atoms processed in fixed order
(z = 0..num_atoms-1). The analytical entropy form is mathematically
identical to -sum(p log p):
-sum(p log p) = log(S) - (1/S) * sum(exp(logit-M) * (logit-M))
= log(S) - TLM / S
The previous code's `if (prob > 1e-10f)` underflow guard is no longer needed:
exp(logit - M) underflows cleanly to 0 when logit << M, and the analytical
form does not multiply tiny probs by very negative logs. Bit-stable per
feedback_stop_on_anomaly.md — no fuzzy tolerance change.
Atom-stat block-sum reduction unchanged (warp shfl + shared-mem bank,
deterministic).
Expected wall-clock saving: ~33% of 207ms = ~70ms across the smoke run; on
production batch=8192 × 1382 calls per fold, roughly proportional. Lower-bound
estimate — at low batch (smoke) the kernel may be launch-bound rather than
load-bound. nsys re-profile after deployment will quantify.
ABI unchanged; no Rust caller changes required. Per Invariant 7 the audit
doc dqn-gpu-hot-path-audit.md is updated with Fix 19 entry.
Build: SQLX_OFFLINE=true cargo check -p ml --lib clean (12 warnings, baseline).
Tests: cargo test -p ml --lib --no-run clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 1 of the HEALTH_DIAG GPU port (Phase 0 inventory landed in
333ea7184). Lands the typed snapshot struct and the mapped-pinned
allocation that subsequent phases write into and read from — no kernel
and no caller wiring yet, deliberately so the type can stabilise before
Phase 2 invests in CUDA shmem reductions against its byte layout.
`HealthDiagSnapshot` (#[repr(C)] POD, 147×4 = 588 bytes) holds every
numeric field emitted across the 7 HEALTH_DIAG log sites. All fields
are `f32` or `u32` so CPU and GPU agree byte-for-byte with no padding
to reason about. Controller fire booleans (`fire_lr`/`fire_tau`/etc.)
are promoted from u8 → u32 per the Phase 0 design review's open-
question #3 — a `[u8; 6]` followed by `f32` would risk implementation-
defined padding mismatch between Rust's #[repr(C)] and CUDA's struct
layout rules; the +24 bytes of cost is acceptable.
`MappedHealthDiagSnapshot` wraps `cuMemHostAlloc(DEVICEMAP|PORTABLE)`
of `sizeof(HealthDiagSnapshot)` plus `cuMemHostGetDevicePointer_v2` —
mirrors the existing `MappedF32Buffer` allocator pattern but typed.
The kernel chain in Phase 2 will write through `dev_ptr()`; the host
emit code in Phase 4 reads through `host_ref()`. No `memcpy_dtoh`,
no `Vec` allocator on the path — consistent with
`feedback_no_htod_htoh_only_mapped_pinned.md`.
Three unit tests pin the layout (`snapshot_size_is_stable`,
`alignment_is_4_bytes`, `default_is_zeroed`) so any future field
add/reorder requires an explicit test update — guards the kernel-side
offset table from silent drift between commits. The size assertion
records the field count breakdown line by line in a comment so the
next maintainer can audit the math without re-deriving it.
Touched: cuda_pipeline/health_diag.rs (+339 LOC new),
cuda_pipeline/mod.rs (+8 re-exports). cargo check clean at 12 warnings
(workspace baseline). Three new unit tests pass on local CPU host (no
GPU required). Audit doc note added per Invariant 7 — explicitly notes
producer kernel + caller migrations land in Phase 2 / 4 commits.
No fingerprint change — separate mapped-pinned alloc, not part of the
param-tensor layout the fingerprint guards.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
nsys profile of multi_fold_convergence on L40S identified backtest_state_gather as
the #1 GPU consumer at 37.2% (596 ms / 60,588 calls — kernel launch latency
dominated compute) and the per-step TLOB cuBLAS gemvx calls as #2 at 25%
(242K calls). Both share the same per-step amplification: chunk_len=512 separate
gather launches + 512 separate TLOB.forward calls (4 SGEMMs each) per chunk
before any Q-values can be computed.
This commit replaces the per-step gather + DtoD pattern with a single batched
launch, and reuses the same chunked buffer for a single chunk-wide TLOB forward.
Per-chunk launch reduction: from 2*chunk_len + chunk_len*7 to 1 + 7 for the
gather+TLOB phase (4608 -> 8 with chunk_len=512, a 576x reduction).
New kernel `backtest_state_gather_chunk` (experience_kernels.cu):
- Writes [chunk_len, N, padded_sd] directly into chunked_states_buf
- chunk_len * N threads, 1 thread per output row
- Mathematically identical to per-step gather: portfolio_buf and plan_isv_buf
are CONSTANT within a chunk (env_step + plan_state_isv update at chunk
boundary only). Each thread reads independent feature offsets, no atomics,
no reordering.
Chunked val TLOB (gpu_backtest_evaluator.rs + metrics.rs):
- Val TLOB instance now sized to DQN_BACKTEST_CHUNK_SIZE * n_windows via new
GpuBacktestEvaluator::val_tlob_batch_size() helper.
- submit_dqn_step_loop_cublas calls tlob.forward(chunked_states, batch) ONCE
per chunk on the chunk-wide buffer instead of chunk_len times on states_buf.
- Partial last chunks reuse the same buffers (forward(b) accepts any
b <= construction_batch).
Borrow restructure:
- Removed top-of-function `let ch_states = self.chunked_states_buf.as_ref()?`
binding (TLOB needs &mut). Replaced with per-chunk ch_states_base raw u64
device pointer extracted in tight scope, reused by Phase 1 (gather) and
Phase 2+3 (compute_q_values_to + last_step_states_ptr). The pointer is
stable across the chunk because the Option<CudaSlice<f32>> does not
reallocate.
Per-step gather kernel `backtest_state_gather` retained unchanged for
evaluate() / evaluate_ppo() / evaluate_supervised() paths that still need
a single-step writer (closure-based callers with no chunked buffer).
Audit doc dqn-gpu-hot-path-audit.md updated with Fix 18 entry per Invariant 7.
Build: SQLX_OFFLINE=true cargo check -p ml --lib clean (12 warnings, baseline).
Tests: cargo test -p ml --lib --no-run clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Catalogue every numeric field in the HEALTH_DIAG family of log lines
(7 emit sites across training_loop.rs + metrics.rs) and classify each
source as GPU-already / CPU-bound / Mixed / Host-state / ISV-already.
Identifies the dominant cost contributors driving the observed
~70 s/epoch HEALTH_DIAG overhead on L40S after the eval async-split
(commit f815f7239) — they are all CPU-side reductions over multi-million
element per-sample buffers fetched via memcpy_dtoh:
- update_q_mag_means_cached (~786 KB DtoH + B×total_actions loop)
- var_scale_epoch_mean (N-element DtoH + conditional avg)
- trail_fire_and_hold_per_mag (4× N-element DtoH + per-mag loop)
- per_magnitude_winrate_and_variance (4× DtoH + sumsq loop)
- reward_contrib_fractions (6× large-buffer DtoH + 6 passes)
- read_eval_action_distribution_* (4 sites; CPU iter on pinned)
- branch_noisy_sigma_mean (NoisyNet param DtoH per branch)
Aggregate per-emit DtoH ≈ 50-150 MB depending on alloc_episodes ×
alloc_timesteps; fully consistent with the 70 s/epoch observation.
All inputs are already device-resident, so the migration is "stop
reducing on the host" plus a small kernel family writing into a single
mapped-pinned HealthDiagSnapshot struct.
Phase 0 deliverable per the dispatching prompt; subsequent commits
(struct + mapped wrapper, kernel family, CPU emit rewrite, audit-doc
updates) follow the phased plan documented at the bottom of the file
and gate-review on this inventory before touching code.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The original target 'foxhunt-training-artifacts' (cloned from
train-multi-seed-template) does not exist on the cluster MinIO.
Available production buckets: foxhunt-models, foxhunt-training-data,
foxhunt-training-results, foxhunt-binaries, foxhunt-backups,
foxhunt-gitlab-{artifacts,packages,registry}.
foxhunt-training-results is the right home for profiling artefacts
(model evaluation outputs already land there). Upload path is now
foxhunt-training-results/profiles/smoke/<short-sha>/profile-<pod>.nsys-rep.
Note: train-multi-seed-template has the same bucket-name typo —
production --profile uploads have been silently failing. Out of scope
for this commit, separate fix needed there.
NVIDIA's GPU metrics counters require elevated container perms
(/proc/sys/kernel/perf_event_paranoid + nvidia.com/gpu performance
counter device unlock). Containers in our Kapsule pods don't have
those — nsys exits exit-1 immediately with NVGPUCTRPERM.
Default to CUDA/NVTX/osrt traces only — kernel timeline + per-kernel
duration + call-graph context, which is what the bottleneck hunts
actually need. Counters can be re-enabled via --extra-args when
running on a relaxed node.
The test-data-pvc only had 3-4 months of MBP-10 (Q1 2024 only), causing
walk-forward fold generation to fail with 'Need at least 10 months of data'.
Production training uses training-data-pvc mounted at /data with subdirs
futures-baseline / futures-baseline-mbp10 / futures-baseline-trades —
that's the full 27-month dataset.
Switching all three env vars (FOXHUNT_TEST_DATA, FOXHUNT_MBP10_DATA,
FOXHUNT_TRADES_DATA) and the volume/mount to point at the production
data PVC. multi_fold_convergence's 6/2/2 walk-forward (10 months min)
now has ample data.
`grep -q` exits early on first match, closing stdin to upstream
`$TEST_BIN --list`. With pipefail enabled, that SIGPIPE on the upstream
process bubbles up as exit 141 and kills the workflow before the
sanitizer/nsys can launch.
Capture --list output once into a variable, then grep the captured
string (which doesn't pipe to a process). Same pattern for the
already-broken-but-defensive head -30 in the error branch.
Per feedback_mbp10_mandatory.md, MBP-10 + trades are mandatory inputs.
multi_fold_convergence now reads FOXHUNT_MBP10_DATA / FOXHUNT_TRADES_DATA
env vars, forwards them to train_baseline_rl as --mbp10-data-dir /
--trades-data-dir, and hard-fails if either path is missing.
Argo sanitizer-test + nsys-test templates set the env vars to PVC paths
/data/test-data/{mbp10,trades} and FOXHUNT_TEST_DATA points at
/data/test-data/ohlcv (PVC layout has lost+found/trades/mbp10/ohlcv as
top-level subdirs; the symbol ES.FUT lives under ohlcv/).
Without this, the smoke would silently fall back to tick-rule proxy
classification and validate a degraded model variant (no real OFI),
defeating the purpose of L40S validation. Audit doc updated per
Invariant 7.
`set -o pipefail` + `head -1` closing stdin produces SIGPIPE (exit 141)
from upstream `ls`/`grep`. The subshell + || true contains the failure
to the inner pipeline so the script's pipefail doesn't propagate it.
Sanitizer fired exit 141 immediately after the cargo build; nsys
reproduced the same exit on its parallel run, confirming the diagnosis.
The workflow sets CARGO_TARGET_DIR=/cargo-target so cargo writes binaries
to /cargo-target/release/deps/ml-*, not target/release/deps/ml-*. Both
templates were globbing the wrong path and failed with exit 3 (now
caught by the explicit error message).
Compile finished cleanly in 4m14s (sccache cache hit on first run after
re-checkout); test binary lookup then failed and the script exited at
the 'could not locate compiled test binary' check.
Both new workflow components were missing egress permissions and the
default-deny in the foxhunt namespace blocked their git fetch SSH to
gitlab-shell.foxhunt.svc.cluster.local:2222 — caused exit code 128
"Could not read from remote repository" before any GPU work started.
New `argo-sanitizer-nsys-workflow` policy mirrors the ci-pipeline
egress (DNS, K8s API, gitlab SSH on port 2222, MinIO 9000 for log
archive + nsys-rep upload, gitlab registry 5000 for ci-builder image
pull, webservice 8181 for registry token auth, external 80/443 for
mc binary download from dl.min.io).
train-template.yaml had a stray `value: "5"` line directly after
`value: "1.0"` under `- name: spread-ticks`. Stricter YAML parsers
(Go go-yaml used by kustomize) reject duplicate mapping keys and bail
out of the entire kustomize bundle with line-52 errors that mislead
diagnosis to unrelated files.
Permissive parsers (kubectl apply -f's path) silently take the last
value, which made the bug invisible until `kubectl apply -k` was
attempted.
The orphan was a leftover from an earlier edit (likely a max-folds=5
or similar parameter that was renamed but not fully removed).
The unquoted test name string contains '::' (Rust path separator). Some
strict YAML implementations (Go go-yaml when used by certain kustomize
paths) interpret it ambiguously. Quoting makes it always a string scalar.
Templates were applied directly via 'kubectl apply -f' which uses a
permissive parser; the unquoted form worked. Quoting makes them
also kustomize-compatible.
Two new one-shot WorkflowTemplates for validating smoke tests under
GPU-instrumentation tools that don't fit in the laptop's 4 GB VRAM:
- `sanitizer-test`: wraps cargo test smoke under `compute-sanitizer
--tool=memcheck|racecheck|synccheck|initcheck` with --target-processes all
so multi_fold_convergence's spawned `train_baseline_rl` subprocess is also
instrumented. Pre-builds train_baseline_rl example to avoid sanitizer
instrumenting cargo/rustc on the inner spawn. Triages internal-vs-real
errors and exits non-zero only on real bugs.
- `nsys-test`: wraps cargo test smoke under `nsys profile`. Captures CUDA +
NVTX + osrt traces with GPU metrics (ga10x set). Uploads .nsys-rep to
MinIO at foxhunt-training-artifacts/profiles/smoke/<short-sha>/, mirroring
the existing Plan 5 Task 3 production-training profile pattern in
train-multi-seed-template.
Wrapper scripts:
- scripts/argo-sanitizer.sh — `argo submit` wrapper, supports --multi-fold
shortcut for fold-boundary code paths (IQN sync, aux Adam reset,
iqn_readiness reset, MSE clamp) that single-fold tests cannot exercise.
- scripts/argo-nsys.sh — same shape as argo-sanitizer.sh, default test is
performance::test_real_data_single_epoch for broad coverage.
Why L40S: laptop RTX 3050 Ti's 4 GB cannot fit compute-sanitizer's
instrumentation metadata (~2-3x app VRAM) — sanitizer falls back to "didn't
track the launch" with 60k+ internal-allocation errors. L40S 48 GB has
ample headroom for both memcheck and nsys overhead.
Both templates compile cargo test --release --lib --no-run plus cargo build
--release --example train_baseline_rl on the cargo-target PVC. Compile time
dominated by sccache hit rate (production training image: 100% C/C++ cache,
~75% Rust cache after warmup).
Templates registered in kustomization.yaml — apply with `kubectl apply -k
infra/k8s/argo` before first submit.
Final cleanup pass closing out the HtoD migration started by the 3-agent
parallel batch:
- 24afb2baf gpu_dqn_trainer.rs residual COLD ctor sites (8 sites: sel_clip_buf,
spectral norm init_u/v_s1/s2, OFI weight macro, graph/denoise/qlstm params)
- 25b777199 gpu_experience_collector.rs::upload_ofi_features (1 WARM site)
- 840408ceb training_loop.rs raw features/targets upload (2 sites)
- d3762e725 delete orphan htod_f32 + clone_htod_f32 from cuda_pipeline/mod.rs;
paired migration of gpu_tlob.rs:1017/1132 test-block callers per
feedback_no_partial_refactor.md (delete contract → migrate every consumer)
Production HtoD count post-merge: 0. Test-only callsites (signal_adapter,
gpu_action_selector, mod.rs:1116) use cudarc stream API directly and are
excluded by feedback_no_htod_htoh_only_mapped_pinned.md scope.
Total HtoD→mapped-pinned migration across all 4 batches: 91 production sites.
After Fix 1..16 migrated all 80+ production callers off
`super::htod_f32` and `super::clone_htod_f32`, the helper bodies in
`cuda_pipeline/mod.rs:129-145` had zero non-test consumers. Deleted
both function definitions per `feedback_no_legacy_aliases.md` (no
deprecated wrappers).
Per `feedback_no_partial_refactor.md` (when a shared contract is
deleted, every consumer migrates together — including tests), the
two surviving test-block callers in `gpu_tlob.rs::tests` (lines
1017 and 1132) are migrated to `mapped_pinned::upload_f32_via_pinned`
in the same commit. The other test-only callers in
`signal_adapter.rs::tests`, `gpu_action_selector.rs::tests`, and
`cuda_pipeline/mod.rs::tests` use bare `stream.memcpy_htod` /
`stream.memcpy_stod` against the cudarc handle directly (not the
deleted helpers) — no change needed.
A docstring was added at the deletion site recording when and why
the helpers were removed, pointing future readers at the canonical
replacements `mapped_pinned::clone_to_device_f32_via_pinned` and
`mapped_pinned::upload_f32_via_pinned`.
Final state of the HtoD migration sequence:
- production callers of `stream.memcpy_htod` / `memcpy_stod`: 0
- production callers of `htod_f32` / `clone_htod_f32`: 0
- helper definitions: removed from `mod.rs`
docs/dqn-gpu-hot-path-audit.md updated with Fix 17 entry.
cargo check -p ml --lib clean at 12 warnings.
cargo check -p ml --tests clean at 23 warnings (12 lib duplicates +
11 test-specific, baseline unchanged).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The two `crate::cuda_pipeline::clone_htod_f32` callsites in
`training_loop.rs::set_raw_market_data` (lines 1250 and 1254) are
rewritten to `mapped_pinned::clone_to_device_f32_via_pinned` per
`feedback_no_htod_htoh_only_mapped_pinned.md`.
These run once per fold immediately before the step loop begins; the
`_raw` suffix indicates the un-normalized arrays kept on-GPU for the
eval-time critic.
Error-type continuity: the helper already returns `Result<_, String>`,
which feeds `anyhow::anyhow!("…raw upload: {e}")` directly — the prior
.map_err wrapper is preserved verbatim. 1:1 path swap.
docs/dqn-gpu-hot-path-audit.md updated with Fix 16 entry.
cargo check -p ml --lib clean at 12 warnings.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The remaining `super::clone_htod_f32` call in `upload_ofi_features`
(`:4258`) is rewritten to `mapped_pinned::clone_to_device_f32_via_pinned`
per `feedback_no_htod_htoh_only_mapped_pinned.md`. Called from
training_loop's data-load phase before each fold's training run, so
the OFI tensor (4M bars × 32 dims) now stages through mapped-pinned
+ DtoD just like all other large data uploads.
Error-type shift: `mapped_pinned::clone_to_device_f32_via_pinned`
returns `Result<CudaSlice<f32>, String>` whereas `clone_htod_f32`
returned `Result<CudaSlice<f32>, MLError>`. Wrapped with
`.map_err(|e| MLError::ModelError(format!("upload_ofi_features: {e}")))`
to preserve the call-site label in error messages.
docs/dqn-gpu-hot-path-audit.md updated with Fix 15 entry.
cargo check -p ml --lib clean at 12 warnings.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Residual COLD ctor sites missed by the original audit are now migrated
off explicit HtoD per `feedback_no_htod_htoh_only_mapped_pinned.md`:
- `:9922` sel_clip_buf 1-element init (sigmoid head clip-norm seed)
- `:10075-10078` spectral norm init_u_s1/v_s1/u_s2/v_s2 (4 calls)
- `:10095-10096` `alloc_spec_pair!` macro body — both u/v init uploads
inside the macro now go through `upload_f32_via_pinned`; the macro's
`$lbl_u`/`$lbl_v` are reused in the error-message format so per-pair
failures stay diagnostically distinct
- `:11276` graph_params_host (60 floats: cross-branch graph message
passing weights)
- `:11330` denoise_params_host (1800 floats: 2-step diffusion Q-refinement)
- `:11476` qlstm_weights_host (528 floats: QLSTM Xavier init)
All sites use `mapped_pinned::upload_f32_via_pinned` (the canonical
mapped-pinned + DtoD staging helper). The helper returns
`Result<_, String>` whereas this constructor returns
`Result<_, MLError>`, so each site wraps the error via
`.map_err(|e| MLError::ModelError(format!("<site> upload via pinned: {e}")))`.
Site labels preserved so backtraces remain readable.
docs/dqn-gpu-hot-path-audit.md updated with Fix 14 entry.
cargo check -p ml --lib clean at 12 warnings.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Agent 1 — bulk COLD/WARM migration across init/loader files.
Sites: 39 production HtoD calls migrated to MappedF32Buffer / MappedI32Buffer /
new MappedU32Buffer. Cross-file infrastructure includes new staging helpers
(clone_to_device_{f32,i32}_via_pinned, upload_{f32,i32,u32}_via_pinned),
MappedU32Buffer type, and host_slice_mut accessor on MappedF32/U32Buffer.
Per feedback_no_htod_htoh_only_mapped_pinned.md — first of 3 parallel batches.
# Conflicts:
# docs/dqn-gpu-hot-path-audit.md
HOT sites:
- L2992 hindsight bar_indices: was `alloc + memcpy_htod(total i32s)` on
every collect_experiences_gpu when hindsight_fraction > 0. Added
persistent `bar_indices_pinned: MappedI32Buffer` (capacity =
alloc_episodes * alloc_timesteps * 2) allocated at constructor.
Per-call host writes go through host_ptr; relabel kernel reads via
dev_ptr u64.
- L3186 feature_mask: was alloc + memcpy_htod each epoch at t==0.
Promoted `feature_mask_buf` to `Option<MappedF32Buffer>`. Reallocates
only when mask size changes; CPU writes via host_ptr, state_gather
reads via dev_ptr.
- L3816 update_per_sample_support: per_sample_support_buf is read-only
by the kernel — promoted to MappedF32Buffer (kernel args now
dev_ptr u64). per-epoch tile fill becomes a direct host_ptr write.
WARM sites:
- L3097 episode_starts upload: episode_starts_buf is GPU-mutated by
domain_rand_episode_starts so must remain CudaSlice. Stage via
MappedI32Buffer + memcpy_dtod_async through new helper
`upload_host_to_cuda_i32_via_pinned`.
- L2009 upload_expert_actions: promoted expert_actions_gpu from
Option<CudaSlice<i32>> to Option<MappedI32Buffer>. Direct host_ptr
write replaces alloc + memcpy_htod.
COLD sites:
- L1076,3848 portfolio_states init/reset (GPU-mutated): use
upload_host_to_cuda_f32_via_pinned (mapped-pinned staging + DtoD).
- L1093 epoch_state init (GPU-mutated): replace clone_htod_f32 with
explicit alloc_zeros + upload_host_to_cuda_f32_via_pinned.
- L1341 saboteur_base init (GPU-mutated): use the same helper.
- L2784 trade_stats_buf pre-reduction zero: replaced htod_f32 of an
all-zeros vec with stream.memset_zeros — fully GPU-side, no PCIe.
Imports: added DevicePtrMut for memcpy_dtod_async pointer extraction.
Per `feedback_no_partial_refactor.md`, every consumer of the migrated
buffers is updated in this commit:
- per_sample_support_buf kernel args at L3426 (compute_expected_q) and
L3546 (quantile_q_select) now consume dev_ptr u64.
- feature_mask_buf accessor at L3195 reads via .dev_ptr.
- expert_actions_gpu field type change is contained (no external
consumers in this crate).
cargo check -p ml --lib: 11 warnings (unchanged from prior commit on
this branch; 2 below 13-warning baseline because two unrelated trivial
warnings disappeared as a side-effect of the refactor).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per `feedback_no_htod_htoh_only_mapped_pinned.md`, mapped pinned
(cuMemHostAlloc DEVICEMAP) is the only allowed CPU↔GPU path.
Migrates 3 WARM HtoD sites:
- `fused_training.rs:505` HER source_indices Vec<i32> upload
- `fused_training.rs:511` HER reward_ones Vec<f32> upload
- `training_loop.rs:470` curriculum episode_starts Vec<i32> upload
For `fused_training.rs` adds module-level `staging_upload_{i32,f32}`
helpers. For `training_loop.rs` uses an inline staging+DtoD block
because the destination is reached via the out-of-scope accessor
`collector.episode_starts_buf_mut() -> &mut CudaSlice<i32>`.
All destination buffer types remain `CudaSlice<T>` so no downstream
consumer (incl. `gpu_her::relabel_batch_with_strategy` and any
`episode_starts_buf` reader) needs to change.
Note: if Agent 2's `gpu_experience_collector` merge changes
`episode_starts_buf_mut`'s return type to a mapped pinned buffer,
the inline staging block in `training_loop.rs:470` can be simplified
to a direct `write_from_slice` (follow-up).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per `feedback_no_htod_htoh_only_mapped_pinned.md`, mapped pinned
(cuMemHostAlloc DEVICEMAP) is the only allowed CPU↔GPU path.
Both `upload_params` and `upload_target_params` previously did
`stream.memcpy_htod` of TOTAL_PARAMS f32 (~MB) at fold boundaries /
external weight loads. Now route through the helper
`update_via_mapped_f32` introduced in the previous commit — stages
CPU bytes through a transient mapped pinned buffer and DtoD-copies
into the existing `params_buf` / `target_params_buf` `CudaSlice<f32>`.
No public API change. The destination buffer types remain
`CudaSlice<f32>` so every downstream kernel-arg consumer is untouched.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cold-path constructor sites (2):
- portfolio_states init: htod_f32 → upload_f32_via_pinned
- rng_states init: memcpy_htod → upload_u32_via_pinned
Warm per-collect sites (2):
- episode_starts_buf: memcpy_htod → upload_i32_via_pinned
- barrier_config: htod_f32 → upload_f32_via_pinned
Warm per-epoch reset (2 sites + persistent staging refactor):
The Vec<f32>/Vec<u32> host staging fields (portfolio_init_staging,
rng_seed_staging) replaced with persistent MappedF32Buffer /
MappedU32Buffer allocated once at construction. Resets fill via
host_slice_mut() (zero-allocation, zero-HtoD); a single async DtoD
seeds the persistent device-resident portfolio_states / rng_states
(which the kernel mutates each step and must remain in VRAM).
Adds host_slice_mut() accessor on MappedF32Buffer and MappedU32Buffer
to support structured per-element writes against the mapped pages.
Audit row appended (Fix 11) in docs/dqn-gpu-hot-path-audit.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per `feedback_no_htod_htoh_only_mapped_pinned.md`, mapped pinned
(cuMemHostAlloc DEVICEMAP) is the only allowed CPU↔GPU path.
Adds module-level `upload_via_mapped_{f32,i32,u32,u64}` helpers and an
in-place `update_via_mapped_f32`. Each stages CPU data through a
transient mapped pinned buffer and DtoD-copies into the destination
`CudaSlice<T>`, then stream-syncs so the staging buffer is safe to
drop. Destination buffer types remain `CudaSlice<T>` so every existing
consumer (raw_ptr / kernel arg) is untouched, satisfying
`feedback_no_partial_refactor.md` for these one-shot init paths.
Migrated COLD sites in `GpuDqnTrainer::new`:
- weight_decay_mask (TOTAL_PARAMS f32)
- branch_slice_starts_dev, branch_slice_lens_dev ([i32; 4])
- branch_grad_scales_dev ([f32; 4])
- per_branch_gamma_base/max_dev ([f32; 4] each)
- q_quantile_branch_offsets/sizes_dev ([i32; 4] each)
- spectral_norm_descriptors_dev ([u64; 78])
- stochastic_depth_scale_buf ([f32; 3])
- stochastic_depth_rng_state ([u32; 1])
- vsn_group_begins_buf, vsn_group_ends_buf ([i32; num_groups] each)
- mamba2_params Xavier init (mamba2_param_count f32)
11 COLD HtoD sites eliminated. cargo check clean (15 warnings; +2 over
baseline 13 are the new MappedU32/U64 struct visibility warnings,
matching the existing MappedF32/I32 pattern).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- 1 constructor site (rng_states u32 init) rewritten to
mapped_pinned::upload_u32_via_pinned.
- 2 warm-path sites (source_indices, donor_indices in relabel_batch)
rewritten to mapped_pinned::upload_i32_via_pinned. The warm sites
re-allocate pinned staging per relabel call — acceptable on this
cold path (HER relabel runs once per epoch).
Adds MappedU32Buffer type to mapped_pinned.rs mirroring MappedI32Buffer,
plus upload_u32_via_pinned helper. Manual Debug impl so warning count
stays at 13.
Audit row appended (Fix 10) in docs/dqn-gpu-hot-path-audit.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
HOT site: `train_step_gpu` was uploading the 4-byte Adam step counter via
`stream.memcpy_htod(&[step], &mut scratch.adam_t)` on every backward pass.
Promoted `DtScratch.adam_t` from `CudaSlice<i32>[1]` to `MappedI32Buffer(1)`
allocated once in `alloc_dt_scratch`. Per-step CPU writes go through
`host_ptr`; the Adam kernel reads via `dev_ptr` u64. Trivial fix, massive
frequency — runs once per gradient step.
Other sites:
- L478 ctor params init: `params` is mutated by Adam (must stay CudaSlice).
Upload via mapped-pinned staging + `memcpy_dtod_async` through new
helper `upload_host_to_cuda_f32`.
- L501 ctor wd_mask init: wd_mask is read-only by Adam. Promoted to
`MappedF32Buffer`; CPU writes 1.0s through host_ptr, kernel reads dev_ptr.
Caller switched from `wd_mask.raw_ptr()` to `wd_mask.dev_ptr`.
- L1105 trajectory build episode_starts: one-time, replaced `clone_htod`
with `MappedI32Buffer` (direct host write, kernel read via dev_ptr).
- L1147 trajectory build ep_rewards: one-time, replaced `htod_f32` with
`MappedF32Buffer` (direct host write, return_to_go reads via dev_ptr).
Also added `#[allow(missing_debug_implementations)]` to MappedI32Buffer
and MappedF32Buffer in `mapped_pinned.rs` — raw pointers + CUdeviceptr
have no useful Debug impl, and the workspace `missing_debug_implementations`
lint now reaches them through the public DT type that holds them.
Audit doc: docs/dqn-gpu-hot-path-audit.md updated.
cargo check -p ml --lib: 11 warnings (down from 13 baseline; net-zero
regression, 2 minor unrelated warnings vanished as a side-effect).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- 4 stream.clone_htod sites in IQN constructor: cos_features precompute,
online_taus + target_taus tile broadcast, init_iqn_xavier_weights
upload — all rewritten to mapped_pinned::clone_to_device_f32_via_pinned.
- clone_cuda_slice helper rewritten to a single DtoD copy via the
existing super::clone_cuda_slice_f32 helper. Previously did
device→host→device, double-violating both no-DtoH and no-HtoD rules.
Sole callsite is constructor seeding of target params from online.
Audit row appended (Fix 9) in docs/dqn-gpu-hot-path-audit.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per `feedback_no_htod_htoh_only_mapped_pinned.md`, mapped pinned
(cuMemHostAlloc DEVICEMAP) is the only allowed CPU↔GPU path.
`GpuActionSelector::new` previously did one HtoD memcpy to upload
RNG seeds. Now allocates `MappedU32Buffer`, writes seeds via
host_ptr, and passes `dev_ptr` (CUdeviceptr) to the three kernels
that consume rng_states (epsilon_greedy, epsilon_greedy_routed,
branching_action_select).
Adds `MappedU32Buffer` and `MappedU64Buffer` to `mapped_pinned.rs`
mirroring the existing `MappedF32Buffer`/`MappedI32Buffer` API
(new/write_from_slice/read_all/Drop). The U64 variant is staged
for the upcoming spectral-norm host_desc[78] descriptor table
migration in `gpu_dqn_trainer::new`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- trainers/ppo.rs: 2 sites (features, targets in set_raw_market_data)
- hyperopt/adapters/ppo.rs: 3 sites (features, targets in upload path;
action_indices in run_gpu_backtest forward closure)
- hyperopt/adapters/mamba2.rs: 1 site (gpu_to_stream)
All clone_htod_f32 and bare stream.clone_htod calls rewritten to
mapped_pinned::clone_to_device_{f32,i32}_via_pinned.
Audit row appended (Fix 7) in docs/dqn-gpu-hot-path-audit.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
HOT site at simulate_batch_gpu — `actions: &[i32]` was uploaded
per-batch via stream.memcpy_htod into a persistent CudaSlice. Promoted
the field to `MappedI32Buffer` allocated once at constructor; per-call
host writes now go through `host_ptr` (no HtoD copy). Kernel arg switched
to passing `dev_ptr` as a u64.
The two COLD sites (ctor init_state, reset init_state) writing into the
GPU-mutated `portfolio_state_buf` cannot replace the destination buffer
type (env_step kernel writes back to it). Use a mapped-pinned staging
buffer + memcpy_dtod_async via a new `upload_portfolio_state` helper.
Retains zero direct HtoD copies on the only path the violation rule
permits per `feedback_no_htod_htoh_only_mapped_pinned.md`.
Sites migrated:
- L122 ctor init_state (htod_f32 -> staging+DtoD via upload_portfolio_state)
- L174 simulate_batch_gpu actions upload (HOT: persistent MappedI32Buffer)
- L296 reset init_state (htod_f32 -> staging+DtoD)
Audit doc: docs/dqn-gpu-hot-path-audit.md updated with new MIGRATED entries.
cargo check -p ml --lib: 13 warnings, baseline preserved.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Migrates the 3 clone_htod_f32 calls in walk-forward GPU data upload
(features, targets, OFI) to mapped_pinned::clone_to_device_f32_via_pinned.
Audit row appended (Fix 6) in docs/dqn-gpu-hot-path-audit.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per `feedback_no_htod_htoh_only_mapped_pinned.md`, mapped pinned
(cuMemHostAlloc DEVICEMAP) is the only allowed CPU↔GPU path.
`set_count_bonuses` is called per action-selection — converting the
three bonus buffers from CudaSlice<f32> (HtoD memcpy) to
MappedF32Buffer eliminates 3-6 HtoD memcpys per call.
Before: 3 reused-buffer htod_f32 (lines 92/96/100) + 3 first-call
clone_htod_f32 (lines 93/97/101) on every call = 3 HtoD steady-state.
After: zero HtoD — direct host_ptr writes via write_from_slice.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Migrates the 7 in-file callers of clone_htod_f32 in DqnGpuData/PpoGpuData
constructors and helpers to mapped_pinned::clone_to_device_f32_via_pinned.
Sites: features+targets+ofi upload (DqnGpuData::upload_slices,
::upload_ofi), portfolio scratch (build_batch_states), htod_copy_into,
PpoGpuData::upload.
Helper bodies htod_f32 / clone_htod_f32 left intact at lines 129-145 —
other parallel agents are still migrating files that call them.
Audit row appended (Fix 5) in docs/dqn-gpu-hot-path-audit.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds clone_to_device_{f32,i32}_via_pinned and upload_{f32,i32}_via_pinned
in `mapped_pinned.rs`. They allocate a temporary MappedXxxBuffer
(cuMemHostAlloc DEVICEMAP), write the payload via host_ptr, then async
DtoD into a target CudaSlice<T>. Used to migrate cold-path uploads off
explicit cudaMemcpy HtoD per `feedback_no_htod_htoh_only_mapped_pinned.md`
without changing struct field types where the kernel mutates the buffer
every step (e.g. portfolio_buf, weight buffers consumed by Adam).
Audit table updated in docs/dqn-gpu-hot-path-audit.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- 660f02ff4 split compute_validation_loss into launch_validation_loss + consume_validation_loss; mirror split on GpuBacktestEvaluator (launch_metrics_and_record_event + consume_metrics_after_event); evaluate_dqn_graphed_async pipelined path; mapped-pinned mirrors for actions_history/intent_mag/picked_action; lazy eval_done_event re-recorded each epoch; post-loop drain so smoke tests see fresh data
- fe2b77388 adaptive kill threshold from ISV |Q| reference (no tuned constant)
- f2335b3e5 real GPU runtime test for sync_target_from_online (replaces static test)
The async-diag merge (3c0d26292, building on d9cb14f1b + 673b04a8d) wired the
GPU side of cross-stream eval pipelining correctly (training stream `cuda_stream`,
eval stream `validation_stream`, `cuStreamWaitEvent` barrier on `train_done_event`)
but left a host-side `stream.synchronize()` inside
`gpu_backtest_evaluator::launch_metrics_and_download` that blocked the CPU thread
for the FULL eval drain (~25-30s/epoch on L40S). The host thread is the same one
that submits the next epoch's training kernels via `run_full_step` — so until eval
drained, training submission was gated on it, defeating the dedicated
`validation_stream`.
Fix: split metrics readback into record/consume halves at the buffer-readback layer:
* `launch_metrics_and_record_event` — submits the metrics kernel + DtoD-async
copies of `actions_history_buf` / `intent_mag_buf` / `picked_action_history_buf`
into mapped-pinned mirrors, then records `eval_done_event`. Returns immediately.
* `consume_metrics_after_event` — `event.synchronize()` (the SOLE host wait per
epoch boundary), then `read_volatile`s mapped-pinned `metrics_buf`. After the
event syncs, the four action-distribution helpers read directly from the
now-coherent mapped-pinned mirrors — eliminating four synchronous DtoH copies
that violated `feedback_no_htod_htoh_only_mapped_pinned.md`.
Caller migration (per `feedback_no_partial_refactor.md`):
* `evaluate_dqn_graphed` (existing public API) becomes a thin sync wrapper:
launch_async → consume. ABI unchanged.
* New `evaluate_dqn_graphed_async` for the pipelined path.
* `evaluate` / `evaluate_ppo` / `evaluate_supervised` migrated inline to
launch+consume (no caller pipelines them).
* `compute_validation_loss` replaced by `launch_validation_loss` +
`consume_validation_loss`. Training loop now consumes pending at epoch start
(cached_async_val_loss = epoch N-1's val_sharpe) and launches at epoch end
(sentinel-only return). Final post-loop consume drains the LAST epoch's launch
so smoke tests / hyperopt that read `last_eval_direction_dist` see the most
recent epoch's data.
Mathematical identity preserved: every val_sharpe consumed is bit-identical to
what the prior synchronous flow would have produced for the same epoch — only
the timing of the host parse differs (one epoch later, matching the existing
`cached_async_val_loss` lag semantics; HEALTH_DIAG `val [...]` / `val_dir_dist` /
`val_picked_dir_dist` emit moves with the consume).
Audit: docs/dqn-gpu-hot-path-audit.md Fix 4. Out of scope: periodic chunk-level
`stream.synchronize` calls in `submit_dqn_step_loop_cublas` (lines ~1596 + ~1863)
for kernel-error detection — host-blocking but smaller in aggregate; future work.
Workspace baseline preserved: 13 warnings.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the proposed static `include_str!` regression guard for the
fold-boundary IQN target hard-sync (issue #84, root-cause fix in commit
`7c19b5903`) with a real GPU runtime test that exercises the contract
end-to-end. The static guard only caught literal deletion of the call
line — a stub body returning `Ok(())`, a copy against the wrong buffer,
the wrong copy direction, or a queue against the wrong stream all pass
the textual assertion silently.
Test
(`cuda_pipeline::gpu_iqn_head::tests::iqn_sync_target_from_online_makes_target_equal_online`):
1. Construct a `GpuIqnHead` with default `GpuIqnConfig` on the
default CUDA stream.
2. Fill `online_params` ← 0.42 and `target_params` ← 0.99 via a
single mapped-pinned staging buffer + `cuMemcpyDtoDAsync`. No
HtoD copy is issued; the host write through `MappedF32Buffer::host_ptr`
reaches the GPU through the device-mapped alias and the DtoD
copies the staged values into each parameter buffer. The
witnesses 0.42 / 0.99 are arbitrary distinct fp32 constants —
the contract asserted is buffer equality, independent of magnitude.
3. Sanity: read both buffers back via fresh mapped-pinned destinations
+ DtoD, assert they differ pointwise.
4. Call `iqn.sync_target_from_online()`.
5. `stream.synchronize()` so the queued DtoD has retired.
6. Read both buffers back and assert bit-for-bit equality across all
`total_params` slots using `f32::to_bits` (so any future NaN-bearing
implementation also fails loud).
Per `feedback_no_htod_htoh_only_mapped_pinned.md`, all CPU↔GPU
communication routes through `cuMemHostAlloc(DEVICEMAP|PORTABLE)`
mapped pinned memory. Tests are not exempt — fills and read-backs
both use `MappedF32Buffer` + `cuMemcpyDtoDAsync`.
Buffer access exposed via four new `#[cfg(test)] pub(crate)` accessors
on `GpuIqnHead` (`online_params_slice`, `target_params_slice`,
`total_params_for_test`, `stream_for_test`) so the public API is not
widened.
Test carries `#[ignore = "gpu"]` matching the smoke-test convention
already used in `regression_detection.rs`. `cargo test -p ml --lib`
on a CPU-only host (the worktree environment) skips it cleanly; the
L40S smoke validation pool runs it via `--ignored`.
Paired with a strengthened doc-block at the call site in
`fused_training.rs::reset_for_fold` (boxed `DO NOT DELETE` warning +
reference to the new test name and issue #84) so anyone touching the
line sees the regression context inline before deleting.
Touched: `gpu_iqn_head.rs` (4 cfg(test) accessors + tests mod with
helpers + the runtime test, +217 LOC), `fused_training.rs` (boxed
comment + test reference, +16 LOC, no behaviour change),
`docs/dqn-wire-up-audit.md` (audit entry replacing the
static-test entry from the previous proposal, +33 LOC).
Verified:
* `cargo check -p ml --lib` — clean at 13 warnings (workspace
baseline).
* `cargo test -p ml --lib --no-run` — clean at 24 warnings (test
profile baseline).
* `cargo test -p ml --lib state_reset_registry` — 3/3 existing
tests pass (no 4th static-source test added).
* `cargo test -p ml --lib gpu_iqn_head` — 1 test discovered,
correctly reports `ignored, gpu` on this CPU-only worktree.
Local run not attempted — worktree environment lacks a GPU. The test
runs as part of L40S smoke validation via `--ignored`.
No fingerprint change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
