Closes the L40S deploy-bug class where stale fxcache passed
FXCACHE_VERSION validation despite incompatible feature semantics.
Root cause of the original failure: extract_ohlcv_features column 0
changed from raw price -> log-return without anyone bumping the
manually-maintained FXCACHE_VERSION const, so the L40S PVC's older
cache loaded clean and the trainer fed raw prices into the aux head
expecting log-returns (aux_next_bar_mse=2.587e7).
Fix:
* crates/ml/build.rs::emit_feature_schema_hash() runs unconditionally
(before the existing CUDA-feature gate so non-CUDA builds also pick
up the env var) and FNV-1a-hashes the raw bytes of the three
schema-defining sources -- crates/ml/src/features/extraction.rs,
crates/ml/src/fxcache.rs, crates/ml-core/src/state_layout.rs --
mixing in each file's relative path + length so renames /
reorderings also bump the hash. Stable across rust versions and
machines (FNV-1a, not std::hash::DefaultHasher). Emits
cargo:rustc-env=FEATURE_SCHEMA_HASH=<decimal_u64> + three
cargo:rerun-if-changed= lines.
* crates/ml/src/fxcache.rs::FEATURE_SCHEMA_HASH consumes the env var
via env! + const u64::from_str_radix(_, 10) (const-stable since
rust 1.83; workspace MSRV 1.85). FxCacheHeader grows a
feature_schema_hash: u64 field; header size 64->72 bytes;
FXCACHE_VERSION bumped 5->6 to flag the wire-format change.
validate() strict-checks the hash alongside magic / version / dims;
mismatch bails with a descriptive error pointing at "source files
defining feature extraction / state layout / fxcache format have
changed since this cache was built." The existing
precompute_features.rs:218 delete-and-regen-on-Err path handles
recovery automatically; the Argo ensure-fxcache step is unchanged.
* FXCACHE_VERSION docstring now declares it tracks WIRE-FORMAT
changes only -- schema-level changes (feature column semantics,
dimensionality) are tracked automatically by FEATURE_SCHEMA_HASH.
Removes the manual ritual that broke the L40S deploy.
* docs/dqn-wire-up-audit.md entry under Plan 5 Task 5 Phase F.
Cost: cosmetic edits (whitespace, comments) to the three schema
sources trigger one cache regen on next deploy (~5 min for full L40S
dataset, ~40 s for local ES.FUT). Acceptable trade -- false negatives
(missed schema drift) are not.
Validation:
* cargo check workspace clean at 11 warnings (baseline preserved).
* Local ES.FUT cache regen confirmed: existing v5 file rejected with
"Stale FxCache version: 5 (expected 6). Delete and regenerate.",
regenerated v6 cache loads clean on retry (40 s, 175874 bars).
* Auto-detection verified: comment-only edit to extraction.rs line 1
changed emitted hash 5046469432341222878 -> 7772630163018944575;
revert returned the hash deterministically to 5046469432341222878.
* multi_fold_convergence smoke PASSED (1 passed, 0 failed; 689.24 s,
~11.5 min). All 3 folds produced best-checkpoints. Per-fold best
Sharpe: F0=-9.7831 (epoch 1), F1=37.9597 (epoch 2),
F2=40.4789 (epoch 5). aux next_bar_mse range across all 15 epochs:
6.097e-2 -- 4.722e-1 (O(0.1), not 1e7 as in the L40S regression).
No new pip/cargo deps (FNV-1a is ~10 LOC stdlib).
No fingerprint change (LAYOUT_FINGERPRINT_CURRENT untouched -- this
is fxcache wire-format, not GPU param layout).
Files touched:
* crates/ml/build.rs
* crates/ml/src/fxcache.rs
* docs/dqn-wire-up-audit.md
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
93126504ca