Two latent bugs converged in the cancelled 50-epoch run
(train-multi-seed-p5qzw at 96769d171, label_scale=808 vs smoke
baseline 22-28):
Bug A (latent since 063fd2716, 2026-04-19): TARGET_DIM was bumped 4→6
in fxcache (added raw_open at col 4 + mid_price_open at col 5), but
N consumer kernels in experience_kernels.cu + dt_kernels.cu hardcoded
stride 4 when indexing targets[bar*4+col]. Reading at the old stride
against the stride-6 buffer slid every lookup into the wrong bar's
data. Smoke harness paths (multi_fold_convergence) didn't exercise
expert_action_override / compute_difficulty_scores / hindsight_relabel
heavily; production 368k-bar run surfaced the drift via the
compounded label_scale = 30× expected magnitude.
Bug B (introduced today at 5a5dd0fed, 2026-05-02): the Bug 1 fix
changed targets[0] from raw_close to log-return-normalized
preproc_close. training_loop.rs::epoch_vol_normalizer's Welford pass
still read targets[0] expecting raw_close → output was
ln(log_return/log_return) ~ stddev 0.6 → triggered sanity-band warning
+ default fallback (5e-4); but downstream label_scale consumers fed
the corrupt value through.
Sites fixed:
HOST (1 site):
- training_loop.rs:570-573, :603 — w[i].1[0] → w[i].1[TARGET_RAW_CLOSE]
+ warning message updated
GPU (5 sites in experience_kernels.cu):
- line 3768 — kernel param doc OHLCV → fxcache TARGET_DIM=6
- lines 3806-3808 — bar*4+3 → bar*6+2 (raw_close column)
- lines 3974-3975 — i*4+2 → i*6+2 (stride-only)
- lines 4015, 4020 — bar*4+2 → bar*6+2 (stride-only)
- lines 3400-3404 — t_offset = global_idx*4 → *6 (stride-only)
- lines 1553-1561 — kernel header doc updated to 6-column layout
GPU (1 site in dt_kernels.cu):
- line 793 — kernel param doc OHLC → fxcache TARGET_DIM=6
- lines 803, 807 — i*4+3 → i*6+2 (raw_close col 2)
HOST docstring (decision_transformer.rs:1049) — [num_bars, 4]
→ TARGET_DIM=6 with reference to fxcache constant module.
Structural prevention: 6 named column constants
(TARGET_PREPROC_CLOSE / TARGET_PREPROC_NEXT / TARGET_RAW_CLOSE /
TARGET_RAW_NEXT / TARGET_RAW_OPEN / TARGET_MID_OPEN) added to
crates/ml/src/fxcache.rs co-located with the existing TARGET_DIM
(promoted from private to pub). Co-locating in the contract-owner
module means future renames or column adds force every consumer to
update at the same call site. Compile-time density test asserts
columns are dense and exhaustive. Host-side consumer training_loop.rs
imports TARGET_RAW_CLOSE; GPU kernels reference the constant module
in comments + use literal stride 6 with a header block documenting
the contract (kernels can't import Rust constants).
PPO consumer NOT in scope: ppo_experience_kernel.cu reads at stride 4
but PPO has its own set_raw_market_data path uploading 4 columns from
Vec<f64>. PPO write/read pair is internally consistent at stride 4,
unaffected by fxcache stride bump. Production train_baseline_rl.rs
flow doesn't invoke set_raw_market_data, so PPO GPU collector is
currently orphaned. Consolidating PPO onto fxcache target buffer is a
separate refactor.
Verification:
- SQLX_OFFLINE=true cargo check -p ml --offline clean
- cargo build -p ml --release --offline --features cuda clean
(cubin compiles via nvcc; release build under 2 min)
- cargo test -p ml --lib -- target_layout 1/1 pass
(target_columns_dense_and_exhaustive)
- audit grep "targets[var * 4 +]" returns ZERO hits in production
- re-run of train-multi-seed-p5qzw will show label_scale ~22-30
(not 808) — gating production validation
Refs: cancelled train-multi-seed-p5qzw at 96769d171, 50-epoch
validation blocker. Bug origins: 063fd2716 (target_dim 4→6 bump) and
5a5dd0fed (Bug 1 column-0 semantic fix). feedback_no_partial_refactor
(every consumer of shared buffer contract migrates in same commit),
feedback_trust_code_not_docs (kernel doc OHLCV/OHLC strings stale
post-bump), feedback_no_quickfixes (proper structural fix not
one-line patch), feedback_wire_everything_up (column constants land
co-located with fxcache::TARGET_DIM so writer + caller share single
contract module).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
29b1d34c6d