Surfaced the SP7 T7 dispatch-arm bug at runtime (unknown-name panic at
fold boundary). This test enumerates RegistryEntry names from
StateResetRegistry::new() and asserts each has a match arm in
reset_named_state, catching the bug at `cargo test -p ml --lib`
instead of mid-training.
Source-introspection design — no production-code change. Test fails
fast if a future contributor adds a registry entry without the
corresponding dispatch arm.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The pearl_loss_balance_controller memory pearl is at
~/.claude/projects/.../memory/pearl_loss_balance_controller.md (not in
git — user-memory subsystem). MEMORY.md index entry updated.
This audit-doc commit records the T8 landing for the Fix 31 chain.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pre-existing test bug: line 164 asserted `ef >= 0.05` (post-Kelly-cap
eval_dist), which gates on Kelly cold-start warmup completing within
the smoke's training horizon — not on whether the Q-head learned to
prefer Full magnitude. On the local-laptop smoke (1 quarter MBP-10),
Kelly warmup never completes, pinning eval_dist[Full] = 0 even when
Q(Full) > Q(Half) clearly.
Per `project_magnitude_eval_collapse_kelly_capped`, the diagnostic
split landed in #212: intent_dist measures policy learning, eval_dist
measures policy + Kelly-cap. Tests asserting on Q-learning success
must use intent_dist; the test was never updated.
Smoke now PASSES with intent_full=0.057 (intent_full=0.866 in the
prior re-run — both above the 0.05 threshold; Q(Full) is clearly
preferred when Kelly cap doesn't suppress it).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
T2 (commit aa2854017) registered 4 SP7 ResetEntries:
sp7_lb_diff_var_cql → ISV[297..301)
sp7_lb_sample_var_cql → ISV[301..305)
sp7_lb_diff_var_c51 → ISV[305..309)
sp7_lb_sample_var_c51 → ISV[309..313)
But never added their dispatch arms in `reset_named_state`. At fold
boundary, the dispatcher panics with "unknown name 'sp7_lb_diff_var_cql'"
because every FoldReset entry must have a matching match arm.
Same shape as bug #281 (SP5 Layer A bug-fix). Surfaced by the SP7 T7
local sanity smoke (test_magnitude_distribution).
The 4 new arms mirror the existing `sp5_budget_cql` / `sp5_budget_c51`
template — `for b in 0..4 { write_isv_signal_at(BASE + b, 0.0); }`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three compile errors in crates/ml/tests/smoke_test_real_data.rs at lines 568,
644, and 715 — all three were calls to `collect_experiences_gpu(&market_buf,
&target_buf, ...)` where `market_buf` and `target_buf` were `CudaSlice<f32>`
allocated via `stream.alloc_zeros + memcpy_htod`.
The production API changed in cba9f25ed (Bug-1 close-out): both parameters now
require `&MappedF32Buffer` (mapped pinned DEVICEMAP, no HtoD copy). The test
helper `real_market_data` was never updated.
Migration applied to smoke_test_real_data.rs:
- Added `use ml::cuda_pipeline::mapped_pinned::MappedF32Buffer` import
- Removed now-unused `type CudaSlice<T>` alias
- Changed `real_market_data` return type from `(CudaSlice<f32>, CudaSlice<f32>, usize)`
to `(MappedF32Buffer, MappedF32Buffer, usize)`
- Replaced `stream.alloc_zeros + memcpy_htod` with
`unsafe { MappedF32Buffer::new(len) } + write_from_slice` at lines 511-517
- Parameter renamed `stream` → `_stream` since it is no longer used by the buffer
construction (stream is still used by callers via `GpuExperienceCollector::new`)
Pattern mirrors production caller in
crates/ml/src/trainers/dqn/trainer/training_loop.rs:1322-1337.
The two `unsafe` block warnings in the test are expected (same `warn(unsafe_code)`
lint applied project-wide; production code carries the same warnings).
mamba2_hyperopt_p0_p1_fixes.rs was also reported as failed but its failure was
purely a cascading linker OOM kill from the smoke_test_real_data compile error,
not an independent type error — confirmed by `cargo build --test
mamba2_hyperopt_p0_p1_fixes` succeeding independently.
NOTE: The SP7 T7 magnitude_distribution smoke does NOT pass — see report in
commit message body. Root cause: four sp7_lb_* entries were registered in
state_reset_registry.rs (commit aa2854017) but their dispatch arms in
DQNTrainer::reset_named_state (training_loop.rs) were never wired. Error at
runtime: "unknown name 'sp7_lb_diff_var_cql'". This is a T7 wire-up regression
separate from the test-compile fixes in this commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two clarity-only edits surfaced by code review:
- M1: compute_adaptive_budgets header docstring claimed floors of
"0.05 C51, 0.05 IQN, 0.02 CQL, 0.02 Ens" — stale post-T7 (IQN is now
0.11, others are sentinel-aware bootstrap). Rewritten to reflect the
SP7 contract: IQN keeps BASE_IQN=0.11 structural floor; C51/CQL/ENS
use sentinel-aware bootstrap.
- M2: last_ens_budget_eff docstring misattributed ENS to "SP5 Pearl 2
(flatness-modulated)" — Pearl 2 stopped writing BUDGET_ENS_BASE in
T6. Rewritten to state ENS has no active controller driver in SP7;
bootstrap-only.
Comment-only — zero runtime effect. cargo check clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Atomic per feedback_no_partial_refactor — launch site + consumer floor
change + stale doc are three halves of the same contract:
(a) launch_loss_balance_controller wired into the per-step pipeline
after launch_sp5_pearl_2_budget (FLATNESS_BASE populated) and after
backward (grad_decomp pinned slots populated).
(b) compute_adaptive_budgets replaces hard floors (0.02/0.05) with
sentinel-aware bootstrap. The controller may now drive budgets near
zero when the structural target says so. IQN keeps BASE_IQN=0.11
structural floor (reference, can't be 0).
(c) All 4 stale "B4/G5" budget-eff docstrings rewritten to reflect
actual ownership: last_iqn_budget_eff (Pearl 2 flatness),
last_cql_budget_eff (SP7 controller), last_c51_budget_eff (SP7
controller), last_ens_budget_eff (Pearl 2 flatness +
sentinel-aware bootstrap). The previous claims (e.g.,
"0.10×(1−regime)×health") were never implemented; per
feedback_trust_code_not_docs, code wins over docs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four comment/docstring sites still described the pre-SP7 Pearl 2 as
writing "20 floats / 5 outputs / c51/cql/ens" after the T6 kernel-
signature shrink. Updated to reflect the current 6-arg kernel writing
8 floats (IQN + flatness only) into scratch[115..119, 127..131).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Kernel signature reduced from 9 to 6 args; pearl_2_budget_update now
writes only budget_iqn[4] and flatness[4]. Launcher migrated atomically:
arg list shrinks, apply_pearls smoothing loop shrinks from 5 slot-blocks
to 2.
SCRATCH_PEARL_2_C51, SCRATCH_PEARL_2_CQL, SCRATCH_PEARL_2_ENS deleted as
orphan constants per feedback_wire_everything_up. The corresponding
producer scratch slots (111..115, 119..127) become reserved-for-future
inside the buffer.
CQL/C51/ENS budget ownership now lives in the SP7 loss-balance
controller (loaded in T5; wired in T7).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two clarity-only edits surfaced by code review:
- Rename `cql_dev` → `cql_sx_dev` in `launch_loss_balance_controller`.
The launcher uses `grad_decomp_result_dev_ptr + 6 * f32_size` which
is the cql_sx (post-budget) slot, not cql (pre-budget at offset 3).
The variable name now reflects that.
- Add `// 213` inline annotation on `base_wiener_offset` to match the
pattern used by all four sibling launchers (lines 11062, 11194, 11303).
No semantic impact — pointer arithmetic and parameter order unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
LOSS_BALANCE_CONTROLLER_CUBIN static, kernel slot in GpuDqnTrainer,
6 SCRATCH_LB_* index constants (218..242), pub(crate) launch_loss_balance_controller
that runs the producer + 24 apply_pearls_ad smoothing launches.
SP5_SCRATCH_TOTAL bumped 218→242 with updated docblock and allocation
comment to match the 24 new slots.
Component pointers into grad_decomp_result_pinned: IQN at offset 0,
CQL_SX at offset 6, C51 at offset 9 (3-float layout per
launch_grad_decomp docstring).
Producer-only — call site in training_loop.rs lands at T7 atomically
with the consumer floor change and stale-doc deletion per
feedback_no_partial_refactor.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the kernel to the build.rs manifest after pearl_1_ext_num_atoms_kernel
and before the SP5 Layer D producers (matching the chronological SP5/SP7
producer ordering). nvcc-compiled cubin lands under
$OUT_DIR/loss_balance_controller_kernel.cubin and is loaded by
gpu_dqn_trainer.rs in T5.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three comment-only edits surfaced by code review:
- M1: Annotate `tid >= 8` guard with `// 8 = 2 heads × 4 branches`.
- M2: Header pseudocode `actual_ratio = h_norm / iqn_norm` clarified —
the implementation correctly omits the inline `fmaxf` because the
outer guard already enforces `iqn_n >= EPS_DIV`. Comment now states
the invariant explicitly.
- M3: Header "No atomicAdd" line now cites `feedback_no_atomicadd` for
consistency with sibling pearl kernels.
Comment-only — zero runtime effect. cargo check clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Single-block 8-thread kernel: 2 heads (CQL, C51) × 4 branches. Reads
3-float views into grad_decomp_result_pinned for IQN/CQL_SX/C51, plus
FLATNESS_BASE for the per-branch target modulator. Writes new budgets +
raw Wiener observations (diff² and sample²) to producer scratch.
Per-branch flatness-modulated target ratio (CQL → ANCHOR×(1-flatness),
C51 → ANCHOR×flatness). Wiener-optimal α from per-branch EMA state
slots, clamped to [1e-4, 0.5]. Cold-start seeds at consumer-side
bootstrap (0.02 CQL, 0.05 C51) on first observation; sentinel-aware.
Producer-only commit; build.rs entry, kernel slot, launch site land in
subsequent tasks.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rewrite the three sp5_budget_c51/cql/ens registry descriptions to be
accurate at HEAD aa2854017: replace present-tense "driven by"/"no longer
writes" with future-tense "will be driven by"/"will stop writing", replace
non-existent C51_BOOTSTRAP_BUDGET/CQL_BOOTSTRAP_BUDGET/ENS_BOOTSTRAP_BUDGET
named constants with their actual anonymous .max() literal line references,
and apply the "(Pearl 2 will stop writing...)" parenthetical uniformly to all
three slots (cql was missing it). Audit doc T2 bullet updated accordingly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
4 new ResetEntries × 4 slots each cover ISV[297..313). Sentinel 0
triggers Pearl A first-observation bootstrap on fold boundary, matching
the existing Pearl 2/3/4/5/6/8 pattern.
Also: corrected the stale sp5_budget_cql description claiming a
non-existent "regime_stability allocator" (per
feedback_trust_code_not_docs); clarified sp5_budget_c51 / sp5_budget_ens
to reflect SP7 ownership.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Minor item 1: Remove extra column-alignment spaces from the 4 SP7
constants (LB_DIFF_VAR_CQL_BASE, LB_SAMPLE_VAR_CQL_BASE,
LB_DIFF_VAR_C51_BASE, LB_SAMPLE_VAR_C51_BASE) so they match the
no-alignment style of surrounding SP5 constants (BUDGET_C51_BASE etc.).
Constant values (297, 301, 305, 309) and inline comments are unchanged.
Minor item 2: Soften the Fix 31 T7 stub in dqn-wire-up-audit.md from
"sentinel-aware consumer" to "sentinel-aware bootstrap with bootstrap
constants matching the kernel's cold-start basis (defined in T7)"
so the audit entry does not forward-reference specific constant names
before they land.
Cargo check: clean (zero warnings, zero errors).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Update stale (118) → 137 unique-slot count and [174..294) = 120 → [174..313) = 139
range in the SP5_PRODUCER_COUNT docstring. Update inline comment unique-slot count
121 → 137 with SP7 T1 breakdown entry. Append SP7 Task 1 history paragraph matching
the established Layer D D3 paragraph style.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
LB_DIFF_VAR_CQL_BASE=297, LB_SAMPLE_VAR_CQL_BASE=301,
LB_DIFF_VAR_C51_BASE=305, LB_SAMPLE_VAR_C51_BASE=309. Bumps SP5_SLOT_END
297 → 313 and SP5_PRODUCER_COUNT 123 → 139. Helper fns mirror the
existing budget_*/flatness/q_var_per_branch pattern. Layout fingerprint
extended. slot_layout_no_overlaps_and_total_correct asserts the new
total.
Audit doc Fix 31 stub added; will be extended commit-by-commit through
Tasks 2-9 of the SP7 plan.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Critical re-review of v1 caught:
1. Task 6 retained Pearl 2 kernel signature with (void) no-op args
"to save 3-site cascade" — that's the partial-refactor anti-pattern
feedback_no_partial_refactor explicitly forbids. v2 does the full
contract change: signature shrinks 9→6 args, launcher migrates
atomically.
2. SCRATCH_PEARL_2_C51/_CQL/_ENS would become orphan constants —
feedback_wire_everything_up says delete or wire. v2 deletes them in
T6.
3. Audit-doc updates were separated into a single late commit — would
fail check_audit_doc_updates pre-commit gate on T1-T6. v2 folds Fix
31 stub into T1 and extends it commit-by-commit.
4. T5 referenced grad_decomp_*_result_dev_ptr field names that don't
exist — actual layout is one shared 27-float pinned buffer with
per-component byte offsets (iqn=0, cql_sx=24, c51=36). v2 uses
pointer arithmetic from grad_decomp_result_dev_ptr.
5. Tasks 1, 2, 4 had "if file shape X then Y, otherwise Z" placeholder
branches — writing-plans skill forbids these. v2 has concrete patches
based on reading the actual files.
Also corrected: T2 fixes the stale "regime_stability allocator"
description in state_reset_registry.rs alongside the new entries.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Outcome-driven controller replacing Pearl 2's hardcoded-0 CQL budget and
floor-pinned C51 budget with multiplicative ratio adaptation. Targets
`grad_split_bwd cql/iqn = 2.0` and `c51/iqn = 1.0` per slice (trunk, dir,
mag). Slow EMA (α=0.01) consistent with existing controller pearls; Pearl
A sentinel-bootstrap on fold boundary; Pearl D Wiener-optimal smoothing.
Replaces ghost docstring at fused_training.rs:3409 ("0.10×(1−regime)×health"
formula was never implemented). Pearl 2 keeps owning IQN budget (reference
denominator) and FLATNESS_BASE (consumed by NoisyNet σ pearl); stops
writing CQL/C51/ENS slots so the new controller is the sole driver.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The smoke template already compiled `train_baseline_rl`. Now also build
`evaluate_baseline` + `precompute_features` and copy + strip all 3 to
`/data/bin/$SHORT_SHA/` after PASS — exactly the layout that
train-multi-seed-template.yaml's `ensure-binary` cache check
(lines 235-246) looks up by SHA. A smoke-then-train sequence at the same
SHA now hits the cache and skips the ~6-min compile, exiting in the
~1-min pod-startup baseline.
- Drop readOnly:true on training-data PVC mount (cargo writes binaries
into /data/bin/$SHORT_SHA; read-side fxcache + market data unchanged)
- Build all 3 example binaries in a single cargo invocation (sccache-warm)
- Idempotent SHA-keyed dest dir; PASS-only so broken bins never cache
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes Fix 29 audit row #13 — the last ⚠ Stale row from the Bug-1
contract drift triage. Pre-fix, `backtest_plan_state_isv` extracted
raw_close as `features[bar*feat_dim + 0]`. Post Bug-1 (commit
`5a5dd0fed`) `features[..+0]` is z-normed log-return, NOT raw_close.
The resulting `equity = cash + position*raw_close` and
`unrealized = position*(raw_close - entry_price)` formulas mixed
z-normed-log-return as a dollar price, corrupting val plan_isv
slots [PNL_VS_TARGET] (slot 1) and [PNL_VS_STOP] (slot 2). Other
plan_isv slots (progress, conviction, drift, regime, remaining)
don't depend on raw_close and were correct pre-fix.
Resolution: route raw_close from the upload-once `prices` buffer
(layout `[n*max_len*4]` raw OHLC). Close is at column index 3 — the
same index `backtest_env_kernel.cu` already reads from for portfolio
mark-to-market (line 64 of that kernel; OHLC layout is canonical
across the val backtest path). Reading from `prices` aligns the val
plan_isv path with env_step's source-of-truth, eliminating the
mixed-units pathology end-to-end.
Sites fixed:
- crates/ml/src/cuda_pipeline/backtest_plan_kernel.cu:74-100
Kernel signature: `const float* features` and `int feat_dim`
parameters dropped, replaced by `const float* prices` (the
[n*max_len*4] raw OHLC buffer). Raw_close read becomes
`prices[(w*max_len + current_step)*4 + 3]`. Multi-line comment
block documents the Bug-1 origin and the env_step parity
reference. The `bool have_close` / `prices != nullptr` guard
semantics preserved — callers without OHLC data fall back to
`unrealized = 0` exactly as pre-fix.
- crates/ml/src/cuda_pipeline/gpu_backtest_evaluator.rs:~1956
Single launcher (`evaluate_dqn_graphed` chunk loop, the only
invocation site of `plan_state_isv_kernel`) updated to pass
`&self.prices_buf.dev_ptr` instead of `&self.features_buf.dev_ptr`
and to drop the now-unused `feat_dim_i32` local + `.arg()` call.
Inline comment explains the Bug-1 origin.
- docs/dqn-wire-up-audit.md
Stale-B row appended to Fix 30's table. The standalone Stale-B
DEFERRED paragraph at the bottom replaced by the commit summary
+ the Fix 30 closure note (all 4 ⚠ Stale and 1 ❓ Ambiguous rows
from Fix 29's deferred follow-ups now resolved).
Migration scope per `feedback_no_partial_refactor`: kernel signature
changed → every consumer migrates in the same commit. Single launcher;
verified via `grep -rn backtest_plan_state_isv` (only the kernel
definition + the gpu_backtest_evaluator launcher + load site appear).
Verification:
- SQLX_OFFLINE=true cargo check -p ml --offline (43.68s) clean.
- SQLX_OFFLINE=true cargo build -p ml --release --offline
--features cuda (1m 30s) clean; cubin recompiled via nvcc.
- Pre-commit DtoD-via-pinned guard passes (the prereq commit
`4d966e62f` migrated `gpu_backtest_evaluator.rs`'s buffers to
MappedF32Buffer, eliminating the 5 `_via_pinned` callers that
had blocked any prior staging of this file).
Refs Fix 29 row #13. `feedback_no_partial_refactor` (single launcher
migrated alongside kernel signature change in one commit),
`feedback_no_functionality_removal` (PNL_VS_TARGET / PNL_VS_STOP
slots preserved — only their data source corrected; the audit's
"drop the slots" alternative explicitly rejected),
`feedback_no_hiding` (no fallback to z-normed reads remaining;
kernel either gets real raw_close or falls through to
`have_close=false` with `unrealized=0`, identical to pre-fix
smoke-test semantics where prices==NULL),
`feedback_no_cpu_compute_strict` n/a (zero new host-side compute),
`feedback_no_htod_htoh_only_mapped_pinned` already satisfied
(`prices_buf` is `MappedF32Buffer` post the prereq migration),
`feedback_trust_code_not_docs` (the kernel comment said
"raw_close from features buffer" for months — accurate-when-written
pre-Bug-1, stale-after-Bug-1; verify-against-code disambiguates).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Prerequisite for the deferred Fix 30 Stale-B kernel-side fix
(`backtest_plan_kernel.cu` raw_close source). The DtoD-via-pinned
pre-commit guard (`scripts/pre-commit-hook.sh::check_no_dtod_via_pinned`,
commit `5275932f4`) blocks any commit that stages `gpu_backtest_evaluator.rs`
against the file's 5 pre-existing `clone_to_device_*_via_pinned` callers
that landed before the guard. Stale-B has to stage this file (to thread
`prices_buf` into the `backtest_plan_state_isv` launcher), so the
migration must land first as its own atomic commit per
`feedback_no_partial_refactor`.
Migration scope. Single file, four buffer fields:
- prices_buf CudaSlice<f32> → MappedF32Buffer [n*max_len*4]
- features_buf CudaSlice<f32> → MappedF32Buffer [n*max_len*feat_dim]
- portfolio_buf CudaSlice<f32> → MappedF32Buffer [n*8] (kernel-mutated)
- window_lens_buf CudaSlice<i32> → MappedI32Buffer [n]
Init sites (lines ~651-664 post-edit). 4 `clone_to_device_*_via_pinned`
calls replaced with `MappedF32Buffer::new(host.len())` +
`write_from_slice(host)`. No memcpy_dtod_async + stream.synchronize()
pair at construction — mapped-pinned coherence makes the kernel see
host writes after the next stream-sync barrier.
Reset site (`reset_evaluation_state`, line ~1485 post-edit). In-place
`self.portfolio_buf.write_from_slice(&portfolio_init)` replaces the
prior buffer-replacement
`self.portfolio_buf = clone_to_device_f32_via_pinned(...)`. No alloc
churn per epoch; the device pointer stays stable across resets which
matches MappedF32Buffer's intended use.
Consumer sites (17 kernel arg passes). `arg(&self.X_buf)` →
`arg(&self.X_buf.dev_ptr)` so the launcher receives the device pointer
the kernel expects. Sites: launch_gather (×2), launch_gather_chunk,
launch_env_step, env_batch_kernel chunked path, plan_state_isv kernel,
metrics_kernel. CUdeviceptr (u64) is passed by reference exactly as
metrics_dev_ptr already does at the metrics launch site (~line 2497).
Kernel-mutated MappedF32Buffer precedent. portfolio_buf is mutated by
the env_step kernel every step. The same file's `plan_diag_buf` (a
MappedF32Buffer) is also kernel-written via dev_ptr (lines ~1230-1256)
and host-read via host_ptr — direct precedent. The IQN τ migration
(commit `facbf76eb`) confirms cuMemHostAlloc DEVICEMAP for kernel
reads. The mapped_pinned.rs module docstring states explicitly:
"Kernels write through dev_ptr (with __threadfence_system())".
What this change touches:
- crates/ml/src/cuda_pipeline/gpu_backtest_evaluator.rs
Field types (lines 299-321), init block (lines 641-665), reset
(lines 1462-1471), 17 kernel arg passes across 6 launchers.
- docs/dqn-wire-up-audit.md
Fix 30 Stale-B paragraph extended with the prereq commit summary.
Stale-B itself remains DEFERRED (kernel-side fix lands next).
Verification:
- SQLX_OFFLINE=true cargo check -p ml --offline (44.38s) clean.
- SQLX_OFFLINE=true cargo build -p ml --release --offline
--features cuda (53.78s) clean.
- grep -n "clone_to_device_.*_via_pinned\|upload_.*_via_pinned"
crates/ml/src/cuda_pipeline/gpu_backtest_evaluator.rs returns
zero hits.
- grep -n "self\.\(prices\|features\|window_lens\|portfolio\)_buf"
shows every kernel-arg site followed by `.dev_ptr`. The only
non-`.dev_ptr` references are the field declarations, the
constructor moves into `Self { ... }`, and the
`write_from_slice` calls in `reset_evaluation_state`.
- Pre-commit DtoD-via-pinned guard now passes on staging this file.
Eliminates the last 5 `_via_pinned` callers in
`gpu_backtest_evaluator.rs`. Per
`feedback_no_htod_htoh_only_mapped_pinned`, `feedback_no_partial_refactor`
(every consumer of a field-type change migrates in one commit),
`feedback_no_hiding` (no `--no-verify`; the migration IS the fix the
guard is asking for).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes Fix 29 audit row #18. The synthetic-data overlay kernel
`phantom_liquidity_gbm` in `dqn_utility_kernels.cu:1050-1096` was
flagged ❓ Ambiguous because its writer wrote `log_ret` to
`market_features[bar*market_dim + 0]` and `+3` — the WRITER's
downstream consumer contract was unverifiable post Bug 1 (commit
`5a5dd0fed`) which redefined feature index 0 from raw log-return to
z-normed log-return.
Investigation result: zero callers in the entire codebase. The audit
grep `grep -rn "phantom_liquidity_gbm" crates/ml/src/ services/
crates/ bin/` returns ONLY the kernel definition itself — no
`load_function`, no `launch_builder`, no Rust-side launcher. The
kernel has been compiled but never invoked since at least early 2026.
Resolution: delete the kernel definition and replace with an
explanatory comment block. Per `feedback_no_hiding` an orphan kernel
with an ambiguous post-Bug-1 contract is exactly the failure mode the
rule warns against — a future re-wirer would have written
synthetic z-normed log-returns into a slot the production pipeline
expects raw log-returns at, re-introducing the very Bug-1 drift the
audit was triaging. Per `feedback_wire_everything_up` a kernel that
compiles but is unconsumed is either wired in the same commit or
deleted; this kernel was never wired since its writer existed.
The replacement comment block (23 lines) documents the design intent
(GBM-based synthetic-data augmentation) so any future re-introduction
has the spec available, plus the re-introduction conditions:
caller wiring must land in the same commit, and the writer/consumer
contract for `market_features[..+0]` (z-normed vs raw log-return)
must be explicit at the kernel boundary.
What this change touches:
- crates/ml/src/cuda_pipeline/dqn_utility_kernels.cu:1037-1096
47-line kernel definition + ══ comment header deleted; 23-line
DELETED notice replaces it. Net -24 lines.
- docs/dqn-wire-up-audit.md
Fix 30 Ambiguous-A row appended with the dead-code-deleted
resolution. Stale-B deferral note also added (separate paragraph)
documenting the pre-existing `_via_pinned` guard block requires a
dedicated `gpu_backtest_evaluator.rs` migration commit before the
Stale-B kernel-side fix can land.
Verification:
- SQLX_OFFLINE=true cargo build -p ml --release --offline
--features cuda (1m 30s) clean; cubin recompiled via nvcc.
- Pre-commit guards pass.
- Final grep for `phantom_liquidity_gbm` returns only the comment
block in the kernel file.
Refs Fix 29 row #18 deferred follow-up. `feedback_no_hiding`
(delete orphans, do not leave them as ambiguous landmines),
`feedback_wire_everything_up` (every module/feature/kernel built
must be wired to a production path or removed),
`feedback_no_functionality_removal` does NOT apply: a kernel with
zero callers is not a functional feature, and the audit doc retains
the design intent so any future re-introduction can rebuild the
mechanism with the correct post-Bug-1 contract.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes Fix 29 audit row #12. Pre-fix, `scripted_policy_select` read
`state[MARKET_START]` as `close_now` and computed
`recent_ret = (close_now - prev_close) / prev_close`. Post Bug-1
(commit `5a5dd0fed`) `state[MARKET_START]` is z-normed log-return at
the WRITER, not raw_close, so the formula mixed two unrelated signals
(z-normed return vs dollar price). The seed-phase MOMENTUM /
MEAN_REV / VWAP_DEV branches degenerated to noise-floor below the
±0.0001f cutoffs, leaving 60% of seed-phase episodes (the non-UNIFORM
20%+20%+20%) effectively in DIR_HOLD instead of expressing the
intended scripted-policy diversity.
Resolution: route raw_close from the same fxcache target buffer the
env_step kernel reads from. Kernel signature gains
`const float* targets`, `const int* episode_starts`, `int t`,
`int total_bars` parameters. Per-thread `bar_idx = episode_starts[i]
+ t` (matching `experience_kernels.cu:1769`'s indexing convention),
then `close_now = targets[bar_idx*6 + TARGET_RAW_CLOSE]`. Bounds-clamp
to `[0, total_bars-1]` mirrors env_step's out-of-bounds early-return.
Sites fixed:
- crates/ml/src/cuda_pipeline/scripted_policy_kernel.cu
Kernel signature extended; close_now read source switched;
`FXCACHE_TARGET_STRIDE` / `FXCACHE_TARGET_RAW_CLOSE` mirrored as
#defines (kernels can't import Rust constants; co-locating the
literals in a comment-block keeps the cross-language contract
visible at the call site for any future TARGET_DIM bump). The
previous `bar_idx` parameter renamed `t`; the LCG seed now mixes
per-thread `bar_idx = episode_starts[i] + t` instead of the
per-step `t` — stronger entropy across episodes, no behavioural
regression (UNIFORM policy still produces 4-direction uniform).
- crates/ml/src/cuda_pipeline/gpu_experience_collector.rs:~3735
Single launcher updated to pass `&targets_buf.dev_ptr`,
`&self.episode_starts_buf`, `t as i32`, and `total_bars` (already
in scope from line 3301). Inline comment explains Bug-1 origin.
Migration scope per `feedback_no_partial_refactor`: single launcher;
no other call sites. Verified via
`grep -rn scripted_policy_select crates/ml/src/`. NO new `PS_*` slot
or `PORTFOLIO_STRIDE` bump required — the targets buffer is the
canonical raw_close source and routing it directly avoids cascading
through the ~12 PORTFOLIO_STRIDE consumers (kelly_cap_update_kernel,
trade_stats_kernel, gpu_experience_collector, ml-core mirror, etc.).
This is the "route raw_close directly" branch of the audit's contract
decision; the alternative "add an extra state slot" branch was
explicitly rejected as cascade-heavy for a seed-phase-only signal.
Verification:
- SQLX_OFFLINE=true cargo check -p ml --offline (43.87s) clean.
- SQLX_OFFLINE=true cargo build -p ml --release --offline
--features cuda (1m 30s) clean; cubin recompiled via nvcc.
- No host-side compute added; all reads on GPU.
- `targets_buf` is mapped pinned by upstream caller (per
`feedback_no_htod_htoh_only_mapped_pinned`).
Refs Fix 29 row #12. `feedback_no_partial_refactor` (single launcher
migrated in same commit), `feedback_no_functionality_removal`
(`recent_ret` signal preserved — only its data source is fixed; the
CUSUM-substitution alternative path explicitly rejected because the
recent_ret signal IS the scripted-policy contract, not an
implementation accident), `feedback_no_hiding` (no fallback to
z-normed-log-return reads; kernel either gets real raw_close or
clamps to total_bars-1 boundary), `feedback_no_cpu_compute_strict`
n/a (zero new host-side compute), `feedback_trust_code_not_docs` (the
kernel comment said `state[MARKET_START + 0] = close_now` for months
— accurate-when-written, stale-after-Bug-1; verify-against-code
disambiguates).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes Fix 29 audit rows #14 and #15 (Bug-1 contract drift in val/HPO
close-price extraction). Both sites read `target[0]` thinking it is
raw_close, but post Bug-1 (commit `5a5dd0fed`) `target[0]` is
preproc_close (z-normed log-return). The `fv[3]` fallback path is
unreachable on every production code path because
`set_val_data_from_slices` (in `dqn/trainer/mod.rs:1704`) always yields
`Vec<f64>` of length 6 from `[f64; 6]` slices post `TARGET_DIM=6` bump
(commit `063fd2716`), so `target.len() >= 2` is an always-true guard.
Per `feedback_no_hiding` the dead fallback is removed in the same edit
rather than left as a silent wrong-units path.
Sites fixed:
- crates/ml/src/trainers/dqn/trainer/metrics.rs:576
(val window_prices for GpuBacktestEvaluator)
- crates/ml/src/hyperopt/adapters/dqn.rs:2492
(HPO adapter val_close_prices for window-aggregated backtest)
Both now read `target[TARGET_RAW_CLOSE]` (col 2). Both import the named
constant from `crate::fxcache` so a future column rename moves the call
site with the writer (Fix 27 prevention pattern).
Verification:
- SQLX_OFFLINE=true cargo check -p ml --offline (8.03s) clean.
- No GPU code changed; cubin not affected.
Affects val Sharpe annualization + window equity curves on training
metrics; affects HPO val score on every trial. Pre-fix would yield
"prices" of magnitude ~stddev(log_return) (~7e-5 for ES 1-min) feeding
into PnL math that expects dollar prices, producing degenerate
backtest output. Post-fix prices are real raw_close values.
References Fix 27 Bug B (host-side Welford) — same kind of bug surfaced
at val/HPO consumer rather than training Welford. References
feedback_no_hiding (delete unreachable fallbacks rather than leaving
silent wrong-units fall-through), feedback_no_partial_refactor (both
consumers of the same (fv, target) tuple convention migrate together),
feedback_trust_code_not_docs (the `target.len() >= 2` guard read as
defensive but was masking a contract drift).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The kernel block at experience_kernels.cu:698-704 divided
market_features[0..3] by vol_normalizer at runtime — designed for the
pre-Bug-1 pipeline where features arrived as RAW log returns (~±0.001).
After Bug 1 fix (commit 5a5dd0fed) moved z-normalization to the WRITER
(precompute_features.rs::NormStats::normalize_batch before fxcache write,
plus data_loading.rs DBN-fallback path applying the same op), features
arrive already-z-normalized. The runtime division then created a
1000-13000× DOUBLE NORMALIZATION inflating column 0 of next_states to
raw-price magnitude.
DIAG_AUX_LABEL diagnostic ground truth (production train-multi-seed-bn42w):
- features_raw_cuda col 0 mean_abs = 0.443 (clean z-norm at SOURCE)
- aux_nb_label_buf mean_abs = 5398 (= 0.443 × inv_vol ~ 13245)
- ratio matches 1 / vol_normalizer for ES 1-min realised vol ~7.5e-5
This bug caused label_scale=5481 in 50-epoch validation (cancelled
train-multi-seed-bn42w) while smoke ran with smaller window producing
smaller inv_vol → label_scale ~25-432. Both wrong, just different
inflation factors.
What changed:
- experience_kernels.cu:698-704 block deleted; replaced with header
comment explaining why. Kernel parameter `vol_normalizer` retained
in signature with `(void)vol_normalizer;` to silence the
unused-warning — removing the param would cascade through
gpu_experience_collector.rs config struct + launcher + per-epoch
Welford in training_loop.rs (60 lines). Bounded scope: leave the
pipe wired, gut the consumer.
- DIAG_AUX_LABEL diagnostic removed per its Fix 28 removal gate
(gpu_dqn_trainer.rs ~12981 diagnostic block ~165 lines + the
DIAG_AUX_LABEL_SOURCE_PTRS OnceLock static ~22 lines +
training_loop.rs populate site ~22 lines).
- Audit doc Fix 29 entry with audit results + open follow-ups.
Bug-1 contract audit (this commit's exhaustive re-check, 18 sites
classified):
- 1 ⚠→✅: experience_kernels.cu:698-704 (this fix)
- 13 ✅: env-step kernel body, mirror universe, feature mask/noise,
target reads at col 2 raw_close (Fix-27 already correct), DT
rewards kernel, curriculum/hindsight/portfolio_sim kernel target
reads, kernel header docs, PS_PREV_CLOSE state-layout slot
- 4 ⚠ Stale (deferred — separate triage commits):
* scripted_policy_kernel.cu:59-65 — seed-phase momentum reads
state[MARKET_START] as raw_close, but post-Bug-1 it is z-normed
log-return. Affects seed-phase scripted policy quality only.
* backtest_plan_kernel.cu:77-100 — val plan_isv reads
features[bar*feat_dim+0] as raw_close. Corrupts val plan_isv
slots [PNL_VS_TARGET]/[PNL_VS_STOP].
* metrics.rs:576 — val_data → window_prices reads target[0] as
close (clones Fix-27 Bug B at host-side; affects val Sharpe).
* hyperopt/adapters/dqn.rs:2492 — same target[0] pattern in HPO
val_close_prices.
- 1 ❓ Ambiguous: dqn_utility_kernels.cu:1089-1093 synthetic feature
overlay; needs downstream-consumer contract verification.
Verification:
- SQLX_OFFLINE=true cargo check -p ml --offline (47.87s) clean.
- SQLX_OFFLINE=true cargo build -p ml --release --offline
--features cuda (1m 30s) clean; cubin recompiled via nvcc.
- Next L40S production run should show label_scale ~0.8 (matching
kernel docstring expectation).
Refs: DIAG_AUX_LABEL diagnostic from Fix 28 in dqn-gpu-hot-path-audit.md
(commit 2683d4637) which captured the ground truth that pinned this
bug. Cancelled validation runs train-multi-seed-p5qzw (label_scale=808)
and train-multi-seed-bn42w (label_scale=5481) both blocked on this —
Fix 27 cleared the host-side variant (Welford reading target[0] as
raw_close); this Fix 29 closes the kernel-side variant. Bug 1 chain
(label_scale=5443 from 4-month-old #193) closes here.
feedback_trust_code_not_docs (the kernel comment said `#13 Vol
normalization` for months — accurate-when-written, stale-after-Bug-1).
feedback_no_partial_refactor does not apply because the kernel
parameter is retained as a no-op, deliberately leaving the
launcher/config-struct contract intact while the consumer is gutted.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Production training run train-multi-seed-bn42w showed label_scale=5481
(raw_close magnitude) at epoch 4 vs smoke's ~25 at the same commit
29b1d34c6. The 5-layer data-loading defense + DBN spread filter + target
stride/column fix all in place — yet column 0 of next_states_buf still
sees raw-price-magnitude values in production but not in the
multi_fold_convergence test path.
This one-shot diagnostic fires once at the first training batch and
prints:
- aux_nb_label_buf stats (the kernel input; should be ~0.8 z-norm)
- next_states_buf col 0 sample (what strided_gather reads)
- features_raw_cuda col 0 sample (the source feature buffer)
- targets_raw_cuda col 2 sample (raw_close — the suspected leak)
- Verdict line interpreting the 4 stats
Three numbers will pin the source. If aux_buf ~5500: raw_close leaked
into next_states. If features_raw_cuda col 0 ~5500: the writer didn't
normalize. If targets_raw_cuda col 2 ~5500 but aux_buf ~0.8: only
production reads raw_close, smoke doesn't.
One-shot via static AtomicBool. Pre-graph-capture, no perf impact.
Removal gate: delete once the leak source is identified and fixed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two latent bugs converged in the cancelled 50-epoch run
(train-multi-seed-p5qzw at 96769d171, label_scale=808 vs smoke
baseline 22-28):
Bug A (latent since 063fd2716, 2026-04-19): TARGET_DIM was bumped 4→6
in fxcache (added raw_open at col 4 + mid_price_open at col 5), but
N consumer kernels in experience_kernels.cu + dt_kernels.cu hardcoded
stride 4 when indexing targets[bar*4+col]. Reading at the old stride
against the stride-6 buffer slid every lookup into the wrong bar's
data. Smoke harness paths (multi_fold_convergence) didn't exercise
expert_action_override / compute_difficulty_scores / hindsight_relabel
heavily; production 368k-bar run surfaced the drift via the
compounded label_scale = 30× expected magnitude.
Bug B (introduced today at 5a5dd0fed, 2026-05-02): the Bug 1 fix
changed targets[0] from raw_close to log-return-normalized
preproc_close. training_loop.rs::epoch_vol_normalizer's Welford pass
still read targets[0] expecting raw_close → output was
ln(log_return/log_return) ~ stddev 0.6 → triggered sanity-band warning
+ default fallback (5e-4); but downstream label_scale consumers fed
the corrupt value through.
Sites fixed:
HOST (1 site):
- training_loop.rs:570-573, :603 — w[i].1[0] → w[i].1[TARGET_RAW_CLOSE]
+ warning message updated
GPU (5 sites in experience_kernels.cu):
- line 3768 — kernel param doc OHLCV → fxcache TARGET_DIM=6
- lines 3806-3808 — bar*4+3 → bar*6+2 (raw_close column)
- lines 3974-3975 — i*4+2 → i*6+2 (stride-only)
- lines 4015, 4020 — bar*4+2 → bar*6+2 (stride-only)
- lines 3400-3404 — t_offset = global_idx*4 → *6 (stride-only)
- lines 1553-1561 — kernel header doc updated to 6-column layout
GPU (1 site in dt_kernels.cu):
- line 793 — kernel param doc OHLC → fxcache TARGET_DIM=6
- lines 803, 807 — i*4+3 → i*6+2 (raw_close col 2)
HOST docstring (decision_transformer.rs:1049) — [num_bars, 4]
→ TARGET_DIM=6 with reference to fxcache constant module.
Structural prevention: 6 named column constants
(TARGET_PREPROC_CLOSE / TARGET_PREPROC_NEXT / TARGET_RAW_CLOSE /
TARGET_RAW_NEXT / TARGET_RAW_OPEN / TARGET_MID_OPEN) added to
crates/ml/src/fxcache.rs co-located with the existing TARGET_DIM
(promoted from private to pub). Co-locating in the contract-owner
module means future renames or column adds force every consumer to
update at the same call site. Compile-time density test asserts
columns are dense and exhaustive. Host-side consumer training_loop.rs
imports TARGET_RAW_CLOSE; GPU kernels reference the constant module
in comments + use literal stride 6 with a header block documenting
the contract (kernels can't import Rust constants).
PPO consumer NOT in scope: ppo_experience_kernel.cu reads at stride 4
but PPO has its own set_raw_market_data path uploading 4 columns from
Vec<f64>. PPO write/read pair is internally consistent at stride 4,
unaffected by fxcache stride bump. Production train_baseline_rl.rs
flow doesn't invoke set_raw_market_data, so PPO GPU collector is
currently orphaned. Consolidating PPO onto fxcache target buffer is a
separate refactor.
Verification:
- SQLX_OFFLINE=true cargo check -p ml --offline clean
- cargo build -p ml --release --offline --features cuda clean
(cubin compiles via nvcc; release build under 2 min)
- cargo test -p ml --lib -- target_layout 1/1 pass
(target_columns_dense_and_exhaustive)
- audit grep "targets[var * 4 +]" returns ZERO hits in production
- re-run of train-multi-seed-p5qzw will show label_scale ~22-30
(not 808) — gating production validation
Refs: cancelled train-multi-seed-p5qzw at 96769d171, 50-epoch
validation blocker. Bug origins: 063fd2716 (target_dim 4→6 bump) and
5a5dd0fed (Bug 1 column-0 semantic fix). feedback_no_partial_refactor
(every consumer of shared buffer contract migrates in same commit),
feedback_trust_code_not_docs (kernel doc OHLCV/OHLC strings stale
post-bump), feedback_no_quickfixes (proper structural fix not
one-line patch), feedback_wire_everything_up (column constants land
co-located with fxcache::TARGET_DIM so writer + caller share single
contract module).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ml::validation is gated behind validation-mod, but dqn/config.rs:177
references crate::validation::RegimeMetrics unconditionally. Both
services pull in dqn::config via the trainer pipeline, so they need
the gate opened to compile. Cheaper than refactoring 700+ refs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Plan task #296. Pearl 7 was an INVESTIGATION task in the SP5 brainstorm:
the pre-SP5 50-epoch baseline (train-multi-seed-cv2mw, F0 epochs 4-9)
showed intent_dist freezing at exact Bin(2, 0.5) ratios (0.25/0.50/0.25),
suggesting a hidden binary action decomposition somewhere downstream.
The plan §C4 closure rule: if post-SP5 smokes show intent_dist drifting
normally (no freeze), Pearl 7 closes with no code changes.
Verdict from 3 retained SP5-era smokes: intent_dist drifts smoothly each
epoch. No Bin(2, 0.5) freeze observed at:
- smoke-test-ks2wf (post-spread-filter, 5845e4403)
- smoke-test-7pv9v (Layer D additive, f42b5fff8)
- smoke-test-w9nsw (D4 atomic, 2e9e276a0)
(smoke-test-cnlrw (sanitize-only, 8434737a6) cached log was pruned
before evidence-collection; 3 retained smokes sample post-spread-filter
and full Layer D production paths and decisively satisfy the closure
rule on their own.)
Likely cause: SP5 Layer A's per-branch parameter lifting (C51 atom span,
NoisyNet σ, IQN τ schedule, loss budgets, Adam β/ε, Kelly floors) added
enough independent variability at every shared site that no single
2-state decomposition can dominate intent_dist in steady state.
What lands:
- docs/dqn-wire-up-audit.md Pearl 7 closure entry
- memory/pearl_intent_dist_freeze_resolved.md (new)
- memory/MEMORY.md Topic Files Index entry
No code changes. No follow-up spec opened. The escalation trigger
(re-freeze for ≥3 consecutive epochs in a future run) is documented
in the audit entry.
Closes plan task #296.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wires the 3 Layer D producer kernels into the production hot path AND
removes the host-side EMA arithmetic for sharpe/max_dd/low_dd_ratio in
the same commit per feedback_no_partial_refactor + feedback_no_cpu_compute_strict.
Sites migrated (host → GPU kernel + ISV slot):
training_sharpe_ema → launch_training_metrics_ema → ISV[294]
max_dd_ema → launch_training_metrics_ema → ISV[295]
low_dd_ratio → launch_training_metrics_ema → ISV[296]
LearningHealth.compose → launch_health_composition → ISV[290..294)
trade-PnL aggregation → launch_sp5_pnl_aggregation → ISV[286..290)
Behavior preserved within float precision: every EMA β decay constant,
warmup/clamp wrapper, and downstream consumer formula migrates verbatim.
The kernels reproduce the host computations bit-for-bit (verified by the
GPU-gated unit tests landed in D1/D2/D3 at f42b5fff8 and the D1-rewrite
re-verification at 66f7e64d1; smoke-test-7pv9v PASSED with the additive
kernels loaded but unwired). The additional Pearl A+D smoothing layer
(ALPHA_META=1e-3) is the SP5 architectural change the entire Layer D
programme commits to (cf. pearl_first_observation_bootstrap +
pearl_wiener_optimal_adaptive_alpha).
Removed: host-side EMA arithmetic at training_loop.rs:5043-5099 (~57
lines covering max_dd α=0.1 + low_dd_ratio α=0.15 + training_sharpe
adaptive-α + sentinel branches). Struct fields self.training_sharpe_ema,
::_initialized, self.max_dd_ema, self.low_dd_ratio are KEPT because
external consumers exist (HEALTH_DIAG emit at training_loop.rs:3899,
adaptive-DSR aux-weight controller at 3772, smoke tests at
td_propagation.rs:126/135/155 + generalization.rs:102/103, registry log
emit at 6646); they now hold ISV-Pearl-smoothed values populated by
read_isv_signal_at(294/295/296) after the kernel chain completes.
Also added: GpuDqnTrainer::synchronize_isv_stream() — public cold-path
stream-sync helper at gpu_dqn_trainer.rs:~18475. Single cuStreamSynchronize
of the training stream, mirroring the existing per_branch_q_gap_ema()
embedded sync. Used once per epoch by D4's read-back path.
Out of scope (deferred to D5):
- HealthEmaTrackers::update host-side α=0.1 EMA (metrics.rs:23-25):
produces D2's inputs; D2's launcher takes already-EMA'd scalars.
Eliminating this needs a 4th kernel; out of D4 scope.
- LearningHealth::update warmup wrapper + [0.2, 0.95] clamp
(learning_health.rs:114-124): may have non-DQN consumers (PPO, eval);
deferred to a separate refactor.
- compute_epoch_financials per-bar equity walk: stays as host-side
input source for D1; D1 reproduces it on GPU as a PARALLEL producer
publishing to ISV.
Verification:
- cargo check + build clean
- sp5_isv_slots + state_reset_registry unit tests 8/8 pass
- cargo test -p ml --lib: 933 pass / 14 fail (identical to pre-D4
baseline; failing tests are pre-existing GPU-context-required tests,
not regressions)
- `git grep "= EMA_BETA *|= (1.0 - EMA_BETA)"` returns 0 (was already
0 — gate is a no-op for SP5 because production uses literal
coefficients, not a named EMA_BETA constant). Meaningful gate
`git grep "0.9 \* self.max_dd_ema|0.85 \* self.low_dd_ratio"` also
returns 0 — confirms host EMA arithmetic deletion.
- Order-of-ops: launches chained on training stream before consumer
reads; stream-event sync only (synchronize_isv_stream once per epoch
for cold-path host scalar refresh; producer→consumer paths on the
same stream observe FIFO order without explicit fence).
Mapped-pinned discipline (per feedback_no_htod_htoh_only_mapped_pinned):
D1's step_returns + done_flags consumed via fresh MappedF32Buffer
allocations populated by host_slice_mut — no HtoD copy. Once-per-epoch
cold-path; matches existing pattern at training_loop.rs:1318.
L40S smoke validation deferred to next dispatch — D4 is structurally
risky enough (HEALTH_DIAG visibility behaviour change + ISV-Pearl-
smoothing on host-scalar reads + first-epoch cold-start interaction
with Pearl A) to validate separately. Closes feedback_no_cpu_compute_strict
sweep for SP5 scope (modulo D5 follow-up).
Refs: SP5 plan §D Task D4. Builds on D1-rewrite (66f7e64d1), D2
(e49756ac9), D3 (f42b5fff8). First D4 attempt was blocked by D1
per-trade vs per-bar mismatch; resolved by D1-rewrite.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The original D1 kernel (commit 5ee795f14) was authored against a brief
that assumed per-trade event arrays exist in production. They don't.
Production has per-bar step_returns + done_flags with episode-boundary
resets, plus already-aggregated summary scalars (sum_returns,
sum_sq_returns) on TradeStats. The previous D4 agent found this
mismatch as a structural blocker and stopped before any code changes.
This rewrite fixes the kernel internals to match compute_epoch_financials
in financials.rs bit-for-bit. The slot allocation (ISV[286..290) for
PNL_TOTAL/MEAN/VAR/MAX_DD), Pearls A+D chain, and reset registry are
unchanged — only the kernel interface and max_dd internals shift.
What changed:
- pnl_aggregation_kernel.cu: full rewrite. New signature consumes
step_returns[N], done_flags[N], num_bars, n_trades, sum_returns,
sum_sq_returns, initial_capital. pnl_total uses log-space
compounded growth (financials.rs:80-97). pnl_mean/pnl_var are
per-trade (financials.rs:122-124). pnl_max_dd implements per-bar
equity walk over last 10K bars with reset at done_flags > 0.5
AFTER processing the bar's return (matches financials.rs:163-194
line-by-line, including 1.0 cap).
- launch_sp5_pnl_aggregation: signature updated to match. 2
MappedF32Buffers + 5 i32/f32 scalars instead of 3 mapped-pinned
buffers + 1 count. Same Pearls A+D chain on the same stream.
- sp5_isv_slots.rs: PNL slot docstring updated to reflect the actual
output semantics (slot constants unchanged).
- SCRATCH_PNL_AGG_BASE docstring + cubin static docstring + field
docstring updated.
- sp5_producer_unit_tests::pnl_aggregation_kernel_correctness:
rewritten with new oracle. Inputs cover an episode boundary at
bar 2 so the reset semantics are observable; expected values
derived analytically from the host formula (no CPU oracle in the
test process, per feedback_no_cpu_test_fallbacks).
- Audit doc: D1-rewrite entry supersedes original D1 entry, with a
line-by-line formula provenance table for D4 reviewers.
Unchanged: ISV slot allocation (sp5_isv_slots.rs constants),
SP5_SLOT_END/SP5_PRODUCER_COUNT/ISV_TOTAL_DIM, wiener offsets,
state_reset_registry arm + dispatch, build.rs cubin registration.
The kernel filename stays the same; only the file contents change.
Formula fidelity provenance (financials.rs as the oracle):
- pnl_total: lines 80-97 (log-space sum, 1e-10 floor, exp − 1)
- pnl_mean: line 122 (per-trade sum_returns / n_trades)
- pnl_var: lines 123-124 (per-trade E[X²] − E[X]², .max(0))
- pnl_max_dd: lines 163-194 (10K window, post-update reset, 1.0 cap)
Tests: cargo check + cargo build --features cuda clean; ISV slot +
state_reset_registry tests pass (8/8 unchanged); rewritten GPU
correctness test fires on next L40S smoke.
This unblocks D4 atomic wiring (D1+D2+D3 into the production hot
path with host-EMA removal). D4 dispatch resumes once this lands.
Refs: SP5 plan §D Task D1 + the structural blocker investigation by
the prior D4 agent (no commit; investigation reported up).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Third of 3 Layer D producer kernels. Replaces host-side training_sharpe_ema,
max_dd_ema, low_dd_ratio updates (host-side EMAs in training_loop.rs) with
a fused GPU kernel chained through apply_pearls_ad_kernel. Per
feedback_no_cpu_compute_strict.
Note: agent investigated training_loop.rs and found the third metric is
low_dd_ratio (not gamma_blend as the original plan brief named). The
authored kernel reproduces the actual host-side EMA triplet present in
the codebase.
Additive only — no consumer wiring, no behavior change. The kernel +
launcher are loaded into GpuDqnTrainer and the slots are reserved on the
ISV bus, but the host-side updates continue to run unchanged. D4 (atomic
Layer D commit) wires this and D1+D2 to call sites in the same atomic
refactor per feedback_no_partial_refactor. Also note: training_loop.rs
gains two state-reset-registry dispatch arms (sp5_health_composition for
D2 + sp5_training_metrics_ema for D3) — registry plumbing required by
the new entries, NOT consumer wiring of the kernels themselves; same
pattern as SP5 Layer A bug-fix #281.
What this lands:
- training_metrics_ema_kernel.cu (single-block 3-thread fused EMA)
- Rust launcher launch_training_metrics_ema()
- 3 new ISV slots (TRAINING_SHARPE_EMA_INDEX..LOW_DD_RATIO_INDEX, 294..297)
- ISV_TOTAL_DIM 294 → 297, SP5_PRODUCER_COUNT 120 → 123 (linear span)
- LAYOUT_FINGERPRINT_SEED bump (auto via slot string)
- StateResetRegistry entries for D2 (sp5_health_composition) + D3
(sp5_training_metrics_ema) with reset_named_state dispatch arms
- build.rs cubin registration
- GPU-gated unit test in sp5_producer_unit_tests.rs (analytical EMA)
- SP5 contiguity slot test
training_metrics_ema_slots_contiguous_and_above_health_block
- Audit doc append
Formula fidelity: kernel reproduces the host-side EMA update for
sharpe/max_dd/low_dd_ratio bit-for-bit within float precision. β decay
constants and any warmup/clamp logic migrate as Invariant 1 anchors with
no algorithmic change. Verified by unit test asserting kernel output
matches an analytical EMA sequence within 1e-6 rel-err.
Tests: cargo check + cargo build clean; ISV slot + state_reset_registry
unit tests pass (8/8 incl. new contiguity check); GPU correctness test
fires on next L40S smoke.
Refs: SP5 plan §D Task D3, builds on D1 (5ee795f14) + D2 (e49756ac9).
Layer D additive infrastructure complete after this commit; D4 (atomic
5-site host-EMA → GPU migration) is next.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Second of 3 Layer D producer kernels. Replaces multi-step host arithmetic
in LearningHealth composition (q_gap_ema + q_var_ema + grad_norm_ema →
composed health score) with a fused GPU kernel chained through
apply_pearls_ad_kernel. Per feedback_no_cpu_compute_strict.
Additive only — no consumer wiring, no behavior change. The kernel +
launcher are loaded into GpuDqnTrainer and the slots are reserved on the
ISV bus, but the host-side composition continues to run unchanged. D4
(atomic Layer D commit) wires this and D1+D3 to call sites in the same
atomic refactor per feedback_no_partial_refactor.
What this lands:
- health_composition_kernel.cu (single-block fused composition)
- Rust launcher launch_health_composition()
- 4 new ISV slots (HEALTH_SCORE_INDEX..GRAD_NORM_NORM_INDEX, slots 290..294)
- ISV_TOTAL_DIM 290 → 294, SP5_PRODUCER_COUNT 116 → 120 (linear span)
- LAYOUT_FINGERPRINT_SEED bump (auto via slot string)
- StateResetRegistry entries (sentinel 0.0; per-fold reset enabled)
- build.rs cubin registration
- GPU-gated unit test in sp5_producer_unit_tests.rs (formula fidelity)
- Audit doc append
Formula fidelity: kernel reproduces the host-side LearningHealth::compose
(or equivalent) computation bit-for-bit within float precision. Migration
is structural only — no algorithmic change. Verified by unit test
asserting kernel output matches a Rust copy of the host formula within
1e-6 rel-err.
Tests: cargo check + cargo build clean; ISV slot + state_reset_registry
unit tests pass; GPU correctness test fires on next L40S smoke.
Refs: SP5 plan §D Task D2, builds on D1 (5ee795f14).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First of 3 Layer D producer kernels. Replaces host-side trade-PnL
aggregation loop in training_loop.rs with a GPU kernel chained through
apply_pearls_ad_kernel for smoothing. Per feedback_no_cpu_compute_strict.
Additive only — no consumer wiring, no behavior change. The kernel +
launcher are loaded into GpuDqnTrainer and the slots are reserved on the
ISV bus, but the host-side aggregation continues to run unchanged. D4
(atomic Layer D commit) wires this and the other two D2/D3 kernels to
call sites in the same atomic refactor per feedback_no_partial_refactor.
What this lands:
- pnl_aggregation_kernel.cu (single-block tree-reduce, no atomicAdd)
- Rust launcher launch_sp5_pnl_aggregation()
- 4 new ISV slots (PNL_TOTAL_INDEX..PNL_MAX_DD_INDEX, slots 286..290)
- ISV_TOTAL_DIM 286 → 290; SP5_PRODUCER_COUNT semantics formalised as
wiener-buffer linear-span (110 → 116; the unique-slot count diverged
from the linear span at D1 — pre-D1 they coincided by accident at the
Pearl 6 carve-out's introduction)
- LAYOUT_FINGERPRINT_FRAGMENT extended with PNL_* entries
- StateResetRegistry sp5_pnl_aggregation entry + dispatch arm
(sentinel 0.0; fold-reset; PnL is NOT cross-fold persistent unlike
Pearl 6 Kelly stats, so it takes the standard sentinel-bootstrap
path per pearl_first_observation_bootstrap)
- build.rs cubin registration
- GPU-gated unit test pnl_aggregation_kernel_correctness with
analytical ground truth (no CPU reference per
feedback_no_cpu_test_fallbacks)
- Audit doc append documenting D1 + the SP5_PRODUCER_COUNT semantic
clarification
Tests: cargo check + cargo build --release --features cuda clean;
ISV slot + state_reset_registry unit tests 6/6 pass (incl new
pnl_aggregation_slots_contiguous_and_above_kelly_block); GPU
correctness test fires on next L40S smoke alongside existing 17 SP5
producer unit tests.
Refs: SP5 plan §D Task D1, validated SP5 Layer A/B at 5845e4403.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Documents the SP5 implementation (110 ISV slots, 8 layer-A producer kernels,
1 atomic Layer B consumer migration). Closes plan task #295 partial — the
8th pearl (pearl_sp5_close_out) and full close-out commit are DEFERRED
until the 50-epoch 3-seed × 3-fold L40S validation (#289) completes.
7 new memory pearls landed:
pearl_per_branch_c51_atom_span — Pearl 1
pearl_per_branch_loss_budget — Pearl 2
pearl_per_branch_noisy_sigma — Pearl 3
pearl_per_group_adam_hyperparams — Pearl 4
pearl_per_branch_iqn_tau_schedule — Pearl 5
pearl_kelly_cap_signal_driven_floors — Pearl 6 (cross-fold-persistent)
pearl_trail_stop_signal_driven — Pearl 8
Audit doc append covers Pearl 6's cross-fold carve-out, Pearl 4's ε-only
fall-back path, and the smoke-test-ks2wf @ 5845e4403 validation metrics
(sharpe 4.5→9.77, label_scale 19-28, trade_count 6468-9432).
Refs: SP5 spec at 6e6e0fa11
Direct inspection of ES.FUT_2024-Q1.dbn.zst (databento python client, offline
kubectl-cp from PVC) pinpoints the source of the 8,799 corrupt bars that
sanitize_bars (Fix 25) caught at the bar-level gate.
Q1 file content:
Outrights (correct): ESH4/ESM4/ESU4/ESZ4/ESH5 — 43,353 records (76.87%)
price range $5,063–$5,478
Spreads (poison): ESH4-ESM4/ESM4-ESU4/... — 13,048 records (23.13%)
price range $47.30–$219.70
Calendar spreads trade at the price DIFFERENCE between adjacent contracts
(~$60-150 cost-of-carry roll basis). Databento's stype_in=parent resolution
for ES.FUT returns BOTH outrights AND every spread combination in the same
DBN stream. The legacy decoder keyed only by ts_event + dedup-by-volume;
during low-volume overnight windows + active rollover periods, spread bars
beat the outright on volume and survived the dedup. 780 spread bars
survived in Q1 alone; ~8,800 across 2024-2026.
Fix: build dbn::TsSymbolMap from metadata once, resolve each record's
instrument_id → symbol, skip any symbol containing `-` (spread separator).
Same-ts dedup-by-volume continues to handle legitimate front/back-month
overlap among outrights.
Adds `time = "0.3"` to ml/Cargo.toml (dbn::TsSymbolMap uses time::Date).
Why complementary to the Fix 25 sanitize_bars gate:
- Spread filter (this commit) catches the cause precisely by symbol pattern,
but only for instruments where parent-symbol resolution is the source.
- sanitize_bars catches the symptom universally by close-ratio bound, but
can't distinguish a low-basis spread from a fast-moving outright in
extreme cases.
Both layers active: spread filter dispatches at parser level, sanitize as
defense-in-depth backstop for unknown future contamination shapes.
Validation: `cargo check -p ml --example train_baseline_rl` clean.
Refs: Bug 2 chain (#191, #194, label_scale=5443 leaks), today's
sanitize_bars find (Fix 25). Closes the contamination-source investigation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Stops chasing one-off corruption bugs. Three+ historical fixes patched
specific writers (#191 fxcache column-0, Bug 1 target schema,
label_scale=5443 leaks, today's state[0] heavy-tail). Each new
corruption shape found the next hole. This installs a structural
defense so corruption is REJECTED at the data-loading boundary
regardless of source.
Five independent layers, each mandatory:
1. Bar-level sanity (baseline_common::sanitize_bars):
drop bars with non-positive OHLC, high<low, non-finite values, or
close/prev_close outside [0.5, 2.0]. Catches DBN parse glitches,
broker tick errors, near-zero-open bars at source.
2. safe_log_return result clamp (extraction.rs:1246):
ratio.ln().clamp(±0.1). Real-market 1-bar log returns rarely
exceed ±0.05; ±0.1 traps every legitimate move while rejecting
the corruption shape (corrupt bar with bar.open ≈ 0 → ln = -30
→ previously normalized to -30000-magnitude state[0] outliers).
3. validate_features pre-norm bound (extraction.rs:538):
|val| ≤ 5.0 post-extraction. Pre-norm features come from
safe_normalize ([0,1]/[-1,1]), safe_clip (max ±3), or clamped
log-returns (±0.1); ±5 catches extractor invariant breaks.
4. NormStats::normalize post-norm clamp (walk_forward.rs:688):
((val - mean) / std).clamp(±20.0). Even if upstream produces
outliers, every value uploaded to GPU is bounded.
5. Shared validate_normalized_features gate (walk_forward.rs):
single source-of-truth invariant enforced at THREE sites:
- fxcache fast path (after discover_and_load)
- DBN fallback (after normalize_batch)
- precompute writer (before fxcache write — never persist
a poisoned cache)
Removed: DIAG_BUG2 + DIAG_BUG2_v2 one-shot diagnostics
(~125 lines of host-side download + outlier scan in
training_loop.rs). Replaced by structural defense — instrumentation
isn't needed when corruption can't reach state[0].
FEATURE_SCHEMA_HASH auto-bumps via build.rs FNV-1a hash over
SCHEMA_FILES (extraction.rs included). All pre-fix .fxcache files
on PVC are invalidated at load time; ensure-fxcache regen produces
clean cache with new clamps applied.
Why this finally closes the chapter: per-writer fixes are reactive
(land after corruption hits prod). Boundary validation is
proactive — every future regression to extraction or normalization
trips the gate at load, not at epoch 5 of a 50-epoch run. The 5
layers are independent: a bug in any one leaves the others as
backstop.
Validation: cargo check -p ml --all-targets --offline clean.
NormStats unit tests (walk_forward.rs:706+) still pass — clamp +
validate are additive; existing test inputs are well within bounds.
Refs: SP5 Bug 2 (state[0] std=570 outliers on smoke-test-xb78r),
historical #191#210#214#193#195 chains.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Augments the existing DIAG_BUG2 one-shot block with two narrower scans
that disambiguate where state[0] heavy-tail (mean=6e-3, std=570,
mean_abs=20 over 3200 samples on smoke-test-xb78r) actually originates.
H1 (data-source corruption): scan features_buf.host_ptr for bars where
|feat[bar*42 + 0]| > 100. host_ptr aliases the same physical memory
the kernel reads (mapped-pinned) — direct read of the data
state_gather sees. Reports up to 10 outlier bar indices + mean/std/
max_abs over up to 2M bars.
H2 (kernel/gather bug): scan gpu_batch.states/next_states (already
DtoH-downloaded) for sample indices where |state[i*sd + 0]| > 100.
Reports up to 10 outlier sample indices.
Verdict line logged: feat-outliers nonempty → H1; state-outliers
nonempty AND feat-outliers empty → H2.
Context: smoke-test-xb78r at HEAD cba9f25ed went through DBN fallback
(train_baseline_rl.rs:611, "Loaded N bars (ts to ts)" message) — no
fxcache file exists on training-data PVC. DBN-fallback applies z-score
normalization (line 633) which has small stddev for log-return columns;
a single bar with bar.open ≈ 0 produces ln(ratio) = -30 → normalized
= -30000. Few bars in 700K could carry this signature.
Triggers once via static AtomicBool. Pre-graph-capture, no perf impact.
Cleanup gate: remove DIAG_BUG2 block once Bug 2 root cause is fixed
(likely a tighter clamp inside safe_log_return for log-return columns).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two related blocks landing together (Bug 2 diagnostic forces the structural
conversion of touched files; pre-commit guard at 5275932f4 enforces zero
*_via_pinned helpers per `feedback_no_hiding`).
== Bug 2 instrumentation (training_loop.rs after collect_experiences_gpu) ==
One-shot AtomicBool-gated DIAG_BUG2 dump of first 5 samples × 5 cols of
both gpu_batch.states and gpu_batch.next_states + col-0 mean/std/mean_abs
across the full batch. Prints once at first rollout — pre-graph-capture so
no hot-path impact. Interpretation key embedded:
mean_abs ~0.001 = normalized log_return ✓
mean_abs ~5000 = raw price ✗ (Bug 2 confirmed)
Resolves the smoke-vs-production state[0] divergence question that pure
static code reading couldn't pin: production aux_label_scale=5300 traces
back through gather kernel to "feat[0] of next_states_buf" but the fxcache
shows feat[0] stddev=1.0 (z-normalized log_return). Either next_states_buf
is populated from a different source than fxcache feat[0], or some kernel
mutates state[0] post-gather. The diagnostic prints both states (post-gather)
and next_states (post-shift) to disambiguate.
== Structural conversion: targets_raw_cuda + features_raw_cuda + 10 sites ==
Field types (DQNTrainer in trainer/mod.rs:621/624):
Option<CudaSlice<f32>> → Option<MappedF32Buffer>
init_gpu_raw_buffers_from_slices (training_loop.rs): clone_to_device_f32_via_pinned
calls (lines 1308/1312) replaced with `MappedF32Buffer::new + write_from_slice`.
Consumer signatures (3 functions across 2 files):
- gpu_experience_collector::collect_experiences_gpu(market_features_buf,
targets_buf): &CudaSlice<f32> → &MappedF32Buffer (each)
- gpu_experience_collector::launch_timestep_loop: same
- gpu_experience_collector::compute_difficulty_scores(targets): same
- decision_transformer::build_dt_trajectories(features_gpu, targets_gpu): same
Launch sites: `.arg(buf)` → `.arg(&buf.dev_ptr)` at all 5 launch_builder
invocations in gpu_experience_collector.rs and the 2 in build_dt_trajectories.
== Cold-path init conversions (forced by guard touching shared files) ==
gpu_dqn_trainer.rs (9 sites → 4 buffer-migration groups):
- spec_u_s1/v_s1/u_s2/v_s2 (4 buffers, explicit)
- spec_u/v macro pairs (alloc_spec_pair! body, expands to ~22 buffers)
- graph_params (cross-branch graph message-passing, 60 floats)
- denoise_params (diffusion Q-refinement MLP, 1800 floats)
- qlstm_weights (xLSTM mLSTM-cell, 528 floats)
gpu_experience_collector.rs (1 site):
- upload_ofi_features → ofi_gpu field type Option<CudaSlice<f32>> → MappedF32Buffer
Inherited from prior worktree-agent attempts (compile clean, included here):
- sel_clip_buf in gpu_dqn_trainer.rs
- RmsNormWeightSet γ buffers in gpu_weights.rs
PPO trainer (trainers/ppo.rs) deferred — not on eval-collapse hot path.
== Validation ==
cargo check -p ml --offline: clean
pre-commit hook (check_no_dtod_via_pinned + Invariant 7 + GPU hot-path guard): pass
Bug 2 diagnostic will fire on next L40S run and print state[0] stats for
first batch. If mean_abs ~0.001 → state[0] is correctly normalized and Bug 2
is elsewhere (maybe label_scale_ema initialization). If mean_abs ~5000 →
state[0] really is raw price; the rollout state-builder is the bug.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bug 1 of the eval-Hold-collapse diagnosis. The fxcache target schema
documented in experience_kernels.cu:1556 + cuda_pipeline/mod.rs:508 specifies:
target[0] = preproc_close — log-return-normalized close (network input)
target[1] = preproc_next — log-return-normalized next close
target[2] = raw_close — raw price for portfolio simulation
target[3] = raw_next — raw price
target[4] = raw_open — raw price
target[5] = mid_price_open — MBP-10 midpoint (fallback raw_open)
Both writers — `precompute_features.rs:360` and `data_loading.rs:510` —
violated the contract by storing raw OHLCV close prices in slots [0:1].
Empirical fxcache inspection: target[0..4] mean=$5967, stddev=$582 (raw
prices throughout). The raw-price values at target[0:1] were never directly
consumed by training (production aux head reads next_states[i][0] = MARKET
feat[0] = log_return), but they corrupted any code reading targets per the
documented contract.
Both writers now compute (raw_curr / prev_close).ln() and (raw_next /
raw_curr).ln() for the preproc columns. FXCACHE_VERSION bumped 7→8 to
invalidate existing caches and trigger ensure-fxcache regen.
A second bug — eval label_scale=5300 (raw price magnitude) at production
binary despite source state[0] tracing back to z-normalized log_return —
remains unresolved. Bug 2 instrumentation lands in the next commit; that
runtime trace will pin which production-binary code path injects raw_close
into state[0] post-gather.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
default-features which included cuda; my prior commit dropped both
default-features=false AND added only `financial`, breaking compile.
ml's source references cudarc unconditionally (cudarc::driver::CudaSlice
etc. are not behind #[cfg(feature = "cuda")]). With `cuda` feature off,
cudarc is not pulled in, leading to 722 unresolved-module errors.
Add `cuda` explicitly to the features list. The 8 leaf sub-crates
(ml-backtesting, ml-paper-trading, etc.) stay dropped — the win on
service compile time is preserved.
Real fix is to gate cudarc references in crates/ml/src/ behind
`#[cfg(feature = "cuda")]`, but that's a separate refactor.
compile-and-deploy netpol allows port 2222 (gitlab-shell) and 8181
(webservice) but NOT port 5000 (gitlab-registry). The deps-cache build
needs port 5000 to push the resulting image, so kaniko was failing
with: dial tcp 10.32.4.209:5000: i/o timeout.
ci-pipeline netpol (used by build-ci-image-template, which also pushes
to gitlab-registry via kaniko) does allow port 5000 to app: registry.
Reuse that label.
Setting `app.kubernetes.io/component: deps-cache-build` left the pod
without any matching NetworkPolicy — the cluster's default-deny then
blocked egress to gitlab-shell:2222 and the git clone init container
timed out (verified in refresh-deps-cache-bgttw run).
Switch to `compile-and-deploy` so the pod picks up
argo-compile-and-deploy-workflow netpol's existing allow-list:
gitlab-shell:2222, gitlab-registry, gitlab-webservice, MinIO, etc.
Same egress targets the deps-cache build needs anyway.
The cluster's CronWorkflow CRD (argoproj.io/v1alpha1, current Argo
release) requires `spec.schedules` as an array. Older versions accepted
`spec.schedule` (singular string) as well; the newer CRD rejects it
with: `spec.schedules: Required value`.
Surfaced when applying refresh-deps-cache-template.yaml — the
WorkflowTemplate part applied fine, the CronWorkflow part failed.
