plan5(task5-F): compile-time fxcache schema fingerprint via build.rs

Closes the L40S deploy-bug class where stale fxcache passed
FXCACHE_VERSION validation despite incompatible feature semantics.
Root cause of the original failure: extract_ohlcv_features column 0
changed from raw price -> log-return without anyone bumping the
manually-maintained FXCACHE_VERSION const, so the L40S PVC's older
cache loaded clean and the trainer fed raw prices into the aux head
expecting log-returns (aux_next_bar_mse=2.587e7).

Fix:

* crates/ml/build.rs::emit_feature_schema_hash() runs unconditionally
  (before the existing CUDA-feature gate so non-CUDA builds also pick
  up the env var) and FNV-1a-hashes the raw bytes of the three
  schema-defining sources -- crates/ml/src/features/extraction.rs,
  crates/ml/src/fxcache.rs, crates/ml-core/src/state_layout.rs --
  mixing in each file's relative path + length so renames /
  reorderings also bump the hash. Stable across rust versions and
  machines (FNV-1a, not std::hash::DefaultHasher). Emits
  cargo:rustc-env=FEATURE_SCHEMA_HASH=<decimal_u64> + three
  cargo:rerun-if-changed= lines.

* crates/ml/src/fxcache.rs::FEATURE_SCHEMA_HASH consumes the env var
  via env! + const u64::from_str_radix(_, 10) (const-stable since
  rust 1.83; workspace MSRV 1.85). FxCacheHeader grows a
  feature_schema_hash: u64 field; header size 64->72 bytes;
  FXCACHE_VERSION bumped 5->6 to flag the wire-format change.
  validate() strict-checks the hash alongside magic / version / dims;
  mismatch bails with a descriptive error pointing at "source files
  defining feature extraction / state layout / fxcache format have
  changed since this cache was built." The existing
  precompute_features.rs:218 delete-and-regen-on-Err path handles
  recovery automatically; the Argo ensure-fxcache step is unchanged.

* FXCACHE_VERSION docstring now declares it tracks WIRE-FORMAT
  changes only -- schema-level changes (feature column semantics,
  dimensionality) are tracked automatically by FEATURE_SCHEMA_HASH.
  Removes the manual ritual that broke the L40S deploy.

* docs/dqn-wire-up-audit.md entry under Plan 5 Task 5 Phase F.

Cost: cosmetic edits (whitespace, comments) to the three schema
sources trigger one cache regen on next deploy (~5 min for full L40S
dataset, ~40 s for local ES.FUT). Acceptable trade -- false negatives
(missed schema drift) are not.

Validation:
* cargo check workspace clean at 11 warnings (baseline preserved).
* Local ES.FUT cache regen confirmed: existing v5 file rejected with
  "Stale FxCache version: 5 (expected 6). Delete and regenerate.",
  regenerated v6 cache loads clean on retry (40 s, 175874 bars).
* Auto-detection verified: comment-only edit to extraction.rs line 1
  changed emitted hash 5046469432341222878 -> 7772630163018944575;
  revert returned the hash deterministically to 5046469432341222878.
* multi_fold_convergence smoke PASSED (1 passed, 0 failed; 689.24 s,
  ~11.5 min). All 3 folds produced best-checkpoints. Per-fold best
  Sharpe: F0=-9.7831 (epoch 1), F1=37.9597 (epoch 2),
  F2=40.4789 (epoch 5). aux next_bar_mse range across all 15 epochs:
  6.097e-2 -- 4.722e-1 (O(0.1), not 1e7 as in the L40S regression).

No new pip/cargo deps (FNV-1a is ~10 LOC stdlib).
No fingerprint change (LAYOUT_FINGERPRINT_CURRENT untouched -- this
is fxcache wire-format, not GPU param layout).

Files touched:
* crates/ml/build.rs
* crates/ml/src/fxcache.rs
* docs/dqn-wire-up-audit.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-04-26 16:18:22 +02:00
parent 43d173a4eb
commit 93126504ca
3 changed files with 148 additions and 29 deletions

View File

@@ -4,6 +4,20 @@ use std::process::Command;
fn main() {
println!("cargo:rerun-if-changed=build.rs");
// ── Feature schema fingerprint ────────────────────────────────────────────
// Stamped into every fxcache file so caches built against a different
// feature-extractor / state-layout / fxcache-format version fail
// header-validation at load time and trigger automatic regeneration via
// the Argo `ensure-fxcache` step. Removes the manual "remember to bump
// FXCACHE_VERSION on schema change" ritual that broke the L40S deploy.
//
// Hashing strategy: FNV-1a 64-bit over the raw bytes of these source
// files. Stable across rust versions and machines (unlike
// `std::hash::DefaultHasher`). Cost: harmless cosmetic edits to these
// files (whitespace / comments) trigger one cache regen on next deploy
// (~5min). That cost is acceptable; missed schema drift is not.
emit_feature_schema_hash();
// Only compile CUDA kernels when the cuda feature is enabled
if std::env::var("CARGO_FEATURE_CUDA").is_err() {
return;
@@ -249,6 +263,55 @@ fn try_compile_kernel(
}
}
/// Compute and emit the feature-schema fingerprint as `FEATURE_SCHEMA_HASH`
/// (decimal u64 string) so `fxcache.rs` can pick it up via `env!`.
///
/// Hashes the raw bytes of every source file that defines the on-disk fxcache
/// schema. Any edit (including whitespace / comments) bumps the hash. The
/// fxcache header records this value at write time; `validate()` rejects
/// caches whose recorded hash mismatches the compiled-in const, and
/// `precompute_features` regenerates them.
fn emit_feature_schema_hash() {
// Source files whose bytes determine the on-disk schema.
// Touching ANY of these files implies "fxcache regen on next deploy".
let schema_sources: &[&str] = &[
"src/features/extraction.rs",
"src/fxcache.rs",
"../ml-core/src/state_layout.rs",
];
// FNV-1a 64-bit init constants (RFC-style).
const FNV_OFFSET: u64 = 0xcbf2_9ce4_8422_2325;
const FNV_PRIME: u64 = 0x0000_0100_0000_01b3;
let mut hash: u64 = FNV_OFFSET;
for rel in schema_sources {
let path = PathBuf::from(rel);
let bytes = std::fs::read(&path)
.unwrap_or_else(|e| panic!("Failed to read schema source {}: {e}", path.display()));
// Mix file path bytes too so reordering / renaming bumps the hash even
// if the bytes happen to match.
for b in rel.as_bytes() {
hash ^= *b as u64;
hash = hash.wrapping_mul(FNV_PRIME);
}
// Length separator so concatenation boundaries can't collide.
for b in (bytes.len() as u64).to_le_bytes() {
hash ^= b as u64;
hash = hash.wrapping_mul(FNV_PRIME);
}
for b in &bytes {
hash ^= *b as u64;
hash = hash.wrapping_mul(FNV_PRIME);
}
println!("cargo:rerun-if-changed={}", path.display());
}
// Decimal so the consumer can use the const-stable `u64::from_str_radix`
// with radix 10 without any hex-prefix stripping.
println!("cargo:rustc-env=FEATURE_SCHEMA_HASH={hash}");
}
/// Find nvcc: prefer $CUDA_HOME/bin/nvcc, then check PATH
fn find_nvcc() -> Option<PathBuf> {
// Try $CUDA_HOME/bin/nvcc first

View File

@@ -7,21 +7,22 @@
//! ## Format
//!
//! ```text
//! ┌───────────────────────────────────────────────────┐
//! │ FxCacheHeader (64 bytes) │
//! │ magic [u8; 8] = b"FXCACHE\0" │
//! │ version u16 = 5 (f32)
//! │ feat_dim u16 = 42 │
//! │ target_dim u16 = 6 │
//! │ ofi_dim u16 = 32 │
//! │ bar_count u64 │
//! │ cache_key [u8; 32] (SHA256 raw bytes) │
//! │ reserved [u8; 8]
//! └───────────────────────────────────────────────────┘
//! │ Body (bar_count records) │
//! │ Each record starts with an i64 timestamp (ns).
//! │ Version 5: [i64 ts][80 × f32] = 328 bytes/bar
//! └───────────────────────────────────────────────────┘
//! ┌─────────────────────────────────────────────────────────────
//! │ FxCacheHeader (72 bytes)
//! │ magic [u8; 8] = b"FXCACHE\0"
//! │ version u16 = 6 (f32 + schema-hash)
//! │ feat_dim u16 = 42
//! │ target_dim u16 = 6
//! │ ofi_dim u16 = 32
//! │ bar_count u64
//! │ cache_key [u8; 32] (SHA256 raw bytes)
//! │ feature_schema_hash u64 (FNV-1a from build.rs)
//! │ reserved [u8; 8] │
//! └─────────────────────────────────────────────────────────────┘
//! │ Body (bar_count records)
//! │ Each record starts with an i64 timestamp (ns).
//! │ Version 6: [i64 ts][80 × f32] = 328 bytes/bar │
//! └─────────────────────────────────────────────────────────────┘
//! ```
use anyhow::{bail, Context, Result};
@@ -35,17 +36,47 @@ use tracing::{debug, info};
const FXCACHE_MAGIC: [u8; 8] = *b"FXCACHE\0";
/// Header size in bytes (fixed).
const HEADER_SIZE: usize = 64;
const HEADER_SIZE: usize = 72;
/// Cache format version. Bump on ANY format change (OFI_DIM, FEAT_DIM, etc.).
/// Stale cache files with wrong version are auto-detected and rejected by validate().
/// The ensure-fxcache Argo step catches the error and regenerates.
/// Cache format version. Bump on ANY *wire-format* change (header layout,
/// body record shape). Schema-level changes (feature column semantics,
/// dimensionality) are tracked automatically by `FEATURE_SCHEMA_HASH` —
/// callers do NOT need to bump this constant when they edit
/// `features/extraction.rs` or `ml-core::state_layout`.
/// Stale cache files with wrong version OR wrong schema hash are
/// auto-detected and rejected by `validate()`. The ensure-fxcache Argo step
/// catches the error and regenerates.
/// v5: OFI_DIM 20→32 — adds 10 MicrostructureState slots (ofi_trajectory,
/// realized_variance, hawkes_intensity, book_pressure, spread_dynamics,
/// aggression_ratio, queue_depletion_asymmetry, order_count_flux,
/// intra_bar_momentum, regime_score) + 2 TLOB-novel slots
/// (order_count_imbalance, microprice_residual).
pub const FXCACHE_VERSION: u16 = 5;
/// v6: header grows 64→72 bytes, adds `feature_schema_hash: u64` between
/// `cache_key` and `reserved`. Stamped at write time from the
/// compile-time `FEATURE_SCHEMA_HASH` const (built from the bytes of
/// extraction.rs / fxcache.rs / state_layout.rs by `build.rs`).
/// Recover-from-stale path is identical: validate() bails, Argo regens.
pub const FXCACHE_VERSION: u16 = 6;
/// Compile-time fingerprint over the feature-schema source files. Bumps
/// automatically whenever `features/extraction.rs`, `fxcache.rs`, or
/// `ml-core::state_layout` change (any byte — whitespace counts).
///
/// Encoded into the fxcache header at write time and strict-checked at load
/// time, so caches built against a different schema fail validation and
/// trigger automatic regeneration via `precompute_features`. This removes
/// the manual "remember to bump `FXCACHE_VERSION` on schema change" ritual.
///
/// The actual value is set by `build.rs` (FNV-1a 64-bit). Stable across
/// rust versions and machines (unlike `std::hash::DefaultHasher`).
pub const FEATURE_SCHEMA_HASH: u64 = {
// build.rs emits the hash as a decimal u64 string; `from_str_radix` is
// const since rust 1.83 and the workspace MSRV is 1.85.
match u64::from_str_radix(env!("FEATURE_SCHEMA_HASH"), 10) {
Ok(v) => v,
Err(_) => panic!("build.rs emitted invalid u64 for FEATURE_SCHEMA_HASH"),
}
};
/// Feature vector dimensionality.
const FEAT_DIM: usize = 42;
@@ -65,7 +96,7 @@ const RECORD_F32_COUNT: usize = RECORD_F64_COUNT;
// ── Header ───────────────────────────────────────────────────────────────────
/// 64-byte fixed header for `.fxcache` files.
/// 72-byte fixed header for `.fxcache` files (v6+).
#[derive(Debug, Clone)]
pub struct FxCacheHeader {
/// Magic bytes: `b"FXCACHE\0"`.
@@ -82,6 +113,11 @@ pub struct FxCacheHeader {
pub bar_count: u64,
/// SHA256 cache key (raw 32 bytes).
pub cache_key: [u8; 32],
/// Compile-time feature-schema fingerprint — see `FEATURE_SCHEMA_HASH`.
/// Mismatch on load means the cache was built against different feature
/// extraction / state-layout / fxcache-format source than the current
/// binary. Strict-checked in `validate()`; regen via `precompute_features`.
pub feature_schema_hash: u64,
/// Reserved for future use.
pub reserved: [u8; 8],
}
@@ -99,15 +135,18 @@ impl FxCacheHeader {
ofi_dim: OFI_DIM as u16,
bar_count,
cache_key,
feature_schema_hash: FEATURE_SCHEMA_HASH,
reserved,
}
}
/// Validate header integrity.
///
/// Strictly enforces the current `FXCACHE_VERSION`. Any older version
/// (including v4 with OFI_DIM=20) is rejected — regenerate via
/// `precompute_features`.
/// Strictly enforces the current `FXCACHE_VERSION` AND the current
/// `FEATURE_SCHEMA_HASH`. Any older version (including v4 with
/// OFI_DIM=20, v5 without schema-hash) or any cache built against a
/// different feature-extractor / state-layout source is rejected —
/// regenerate via `precompute_features`.
pub fn validate(&self) -> Result<()> {
if self.magic != FXCACHE_MAGIC {
bail!(
@@ -144,13 +183,22 @@ impl FxCacheHeader {
self.ofi_dim
);
}
if self.feature_schema_hash != FEATURE_SCHEMA_HASH {
bail!(
"Stale FxCache feature schema: hash {:#018x} (expected {:#018x}). \
Source files defining feature extraction / state layout / \
fxcache format have changed since this cache was built. \
Delete and regenerate via precompute_features.",
self.feature_schema_hash, FEATURE_SCHEMA_HASH
);
}
if self.bar_count == 0 {
bail!("FxCache bar_count is zero — empty cache files are not valid");
}
Ok(())
}
/// Serialize header to 64 bytes (little-endian).
/// Serialize header to 72 bytes (little-endian).
fn to_bytes(&self) -> [u8; HEADER_SIZE] {
let mut buf = [0u8; HEADER_SIZE];
buf[0..8].copy_from_slice(&self.magic);
@@ -160,11 +208,12 @@ impl FxCacheHeader {
buf[14..16].copy_from_slice(&self.ofi_dim.to_le_bytes());
buf[16..24].copy_from_slice(&self.bar_count.to_le_bytes());
buf[24..56].copy_from_slice(&self.cache_key);
buf[56..64].copy_from_slice(&self.reserved);
buf[56..64].copy_from_slice(&self.feature_schema_hash.to_le_bytes());
buf[64..72].copy_from_slice(&self.reserved);
buf
}
/// Deserialize header from 64 bytes (little-endian).
/// Deserialize header from 72 bytes (little-endian).
fn from_bytes(buf: &[u8; HEADER_SIZE]) -> Self {
let mut magic = [0u8; 8];
magic.copy_from_slice(&buf[0..8]);
@@ -180,8 +229,12 @@ impl FxCacheHeader {
let mut cache_key = [0u8; 32];
cache_key.copy_from_slice(&buf[24..56]);
let feature_schema_hash = u64::from_le_bytes([
buf[56], buf[57], buf[58], buf[59], buf[60], buf[61], buf[62], buf[63],
]);
let mut reserved = [0u8; 8];
reserved.copy_from_slice(&buf[56..64]);
reserved.copy_from_slice(&buf[64..72]);
Self {
magic,
@@ -191,6 +244,7 @@ impl FxCacheHeader {
ofi_dim,
bar_count,
cache_key,
feature_schema_hash,
reserved,
}
}
@@ -326,7 +380,7 @@ fn write_body_f32(
/// Load an `.fxcache` file into memory.
///
/// Reads the 64-byte header, validates it, then reads the f32 body,
/// Reads the 72-byte header, validates it, then reads the f32 body,
/// converting each f32 back to f64 on load.
///
/// # Arguments

View File

@@ -340,6 +340,8 @@ Plan 4 Task 2c.3c.4 (2026-04-25): GRN backward chain wired through all 3 panic-g
Plan 4 Task 2c.3a (2026-04-24): GRN trunk param-tensor reshuffle (build-clean, runtime-broken intentionally). `compute_param_sizes()` flat layout reshuffled from 86 → 95 tensors: the 4 legacy trunk Linear tensors (`w_s1`, `b_s1`, `w_s2`, `b_s2` at indices [0..4)) are replaced with **13 GRN tensors** at indices [0..13) — 7 for h_s1 GRN block (`w_a`, `b_a`, `w_b`, `b_b`, `w_residual`, `gamma`, `beta`; SD→SH1 with `Linear_residual` GEMM since dimensions differ) plus 6 for h_s2 GRN block (`w_a`, `b_a`, `w_b`, `b_b`, `gamma`, `beta`; SH1→SH2 with identity residual since SH1==SH2). Every tensor index ≥ 4 in the OLD layout shifts +9 in the NEW layout. `NUM_WEIGHT_TENSORS` 86→95, `FIRST_ISV_TENSOR` 68→77. `layout_fingerprint_seed()` updated to mirror the new tensor names + positions; new `LAYOUT_FINGERPRINT_CURRENT = 0xcf3a24b0a1f70057` (was `0xa504d3c2f275b8af`). All 93 `padded_byte_offset` call sites + ancillary index references migrated in lockstep per `feedback_no_partial_refactor.md`. `xavier_init_params_buf` extended with init paths for the 13 new tensors: Linear matrices (`w_a`, `w_b`, `w_residual`) get Xavier; LayerNorm γ initialised to 1.0; β + biases stay zero. Trunk-forward / trunk-backward / IQN-trunk / ensemble-diversity-backward callers gated by explicit `panic!("Plan 4 Task 2c.3a: trunk param layout migrated to GRN, …")` at function entry — `BatchedForward::encoder_forward_only`, `BatchedForward::forward_target_raw`, `BatchedForward::forward_online_f32`, `BatchedBackward::backward_full`, `GpuDqnTrainer::apply_iqn_trunk_gradient`, `GpuDqnTrainer::apply_ensemble_diversity_backward`. This makes accidental runtime execution fail loudly rather than producing silent garbage from running legacy `Linear→ReLU→Linear` GEMMs against GRN-shaped param tensors. Spectral-norm descriptor (13 matrices) keeps slots [0]/[1] mapped to `w_a_h_s1`/`w_a_h_s2` (shapes match legacy W_s1/W_s2 exactly — the GRN's first Linear has the same shape as the original Linear), so the existing spectral-norm constraint transfers cleanly to the GRN's first Linear; Linear_b / Linear_residual are NOT yet spectral-normed (Task 2c.3c follow-up). **Smoke intentionally NOT run** — build-clean is the validation; runtime would hit the panic at the first encoder forward (the desired behaviour, no need to verify explicitly). Task 2c.3b swaps in the GRN forward (gpu_grn::GrnBlock::forward) at all panic-gated forward callers in place; Task 2c.3c does the same for backward callers and runs the smoke test. cargo check clean at 11 warnings (baseline preserved); cargo build compiles all 58 cubins. No new module / kernel / ISV slot in this commit — pure structural reshuffle + assertion gates.
Plan 5 Task 5 Phase F (2026-04-26): **compile-time fxcache schema fingerprint** — closes the L40S deploy-bug class where stale fxcache passed `FXCACHE_VERSION` validation despite incompatible feature semantics. Root cause of the original failure: `extract_ohlcv_features` column 0 changed from raw price → log-return without anyone bumping the manually-maintained `FXCACHE_VERSION` const, so the L40S PVC's older cache loaded clean and the trainer fed raw prices into the aux head expecting log-returns (`aux_next_bar_mse=2.587e7`). Fix has three pieces: (1) `crates/ml/build.rs::emit_feature_schema_hash()` runs unconditionally (before the existing CUDA-feature gate so non-CUDA builds also get the env var) and FNV-1a-hashes the raw bytes of the three schema-defining sources — `crates/ml/src/features/extraction.rs`, `crates/ml/src/fxcache.rs`, `crates/ml-core/src/state_layout.rs` — mixing in each file's relative path + length so renames / reorderings also bump the hash. Stable across rust versions and machines (FNV-1a, not `std::hash::DefaultHasher`). Emits `cargo:rustc-env=FEATURE_SCHEMA_HASH=<decimal_u64>` plus three `cargo:rerun-if-changed=` lines. (2) `crates/ml/src/fxcache.rs::FEATURE_SCHEMA_HASH` consumes the env var via `env!` + const `u64::from_str_radix(_, 10)` (const-stable since rust 1.83; workspace MSRV 1.85). `FxCacheHeader` grows a `feature_schema_hash: u64` field; header size 64→72 bytes; wire-format `FXCACHE_VERSION` bumped 5→6 to flag the layout change. `validate()` strict-checks the hash alongside magic/version/dims; mismatch bails with a descriptive error pointing at "source files defining feature extraction / state layout / fxcache format have changed since this cache was built." `precompute_features.rs:218` already deletes-and-regenerates on any `load_fxcache` Err, so the existing Argo `ensure-fxcache` step (and local users) recover automatically. (3) `FXCACHE_VERSION` doc comment now declares it tracks **wire-format** changes only — schema-level changes are tracked automatically by `FEATURE_SCHEMA_HASH`. Removes the manual ritual that broke the L40S deploy. **Cost**: cosmetic edits (whitespace, comments) to the three schema sources trigger one cache regen on next deploy (~5 min for full L40S dataset, ~40 s for local ES.FUT). Acceptable trade — false negatives (missed schema drift) are not. **Validation**: cargo check workspace clean at 11 warnings (baseline preserved). Local ES.FUT cache regen confirmed: existing v5 file rejected with `"Stale FxCache version: 5 (expected 6). Delete and regenerate."`, regenerated v6 cache loads clean on retry. Auto-detection verified: comment-only edit to `extraction.rs` line 1 changed emitted hash `5046469432341222878``7772630163018944575`; revert returned the hash deterministically to `5046469432341222878`. No new pip/cargo deps (FNV-1a is ~10 LOC stdlib). Files touched: `crates/ml/build.rs`, `crates/ml/src/fxcache.rs`, `docs/dqn-wire-up-audit.md` (this entry). No fingerprint change (LAYOUT_FINGERPRINT_CURRENT untouched — this is fxcache wire-format, not GPU param layout).
| Classification | Count |
|---|---|
| Wired | 88 |