Stops chasing one-off corruption bugs. Three+ historical fixes patched
specific writers (#191 fxcache column-0, Bug 1 target schema,
label_scale=5443 leaks, today's state[0] heavy-tail). Each new
corruption shape found the next hole. This installs a structural
defense so corruption is REJECTED at the data-loading boundary
regardless of source.
Five independent layers, each mandatory:
1. Bar-level sanity (baseline_common::sanitize_bars):
drop bars with non-positive OHLC, high<low, non-finite values, or
close/prev_close outside [0.5, 2.0]. Catches DBN parse glitches,
broker tick errors, near-zero-open bars at source.
2. safe_log_return result clamp (extraction.rs:1246):
ratio.ln().clamp(±0.1). Real-market 1-bar log returns rarely
exceed ±0.05; ±0.1 traps every legitimate move while rejecting
the corruption shape (corrupt bar with bar.open ≈ 0 → ln = -30
→ previously normalized to -30000-magnitude state[0] outliers).
3. validate_features pre-norm bound (extraction.rs:538):
|val| ≤ 5.0 post-extraction. Pre-norm features come from
safe_normalize ([0,1]/[-1,1]), safe_clip (max ±3), or clamped
log-returns (±0.1); ±5 catches extractor invariant breaks.
4. NormStats::normalize post-norm clamp (walk_forward.rs:688):
((val - mean) / std).clamp(±20.0). Even if upstream produces
outliers, every value uploaded to GPU is bounded.
5. Shared validate_normalized_features gate (walk_forward.rs):
single source-of-truth invariant enforced at THREE sites:
- fxcache fast path (after discover_and_load)
- DBN fallback (after normalize_batch)
- precompute writer (before fxcache write — never persist
a poisoned cache)
Removed: DIAG_BUG2 + DIAG_BUG2_v2 one-shot diagnostics
(~125 lines of host-side download + outlier scan in
training_loop.rs). Replaced by structural defense — instrumentation
isn't needed when corruption can't reach state[0].
FEATURE_SCHEMA_HASH auto-bumps via build.rs FNV-1a hash over
SCHEMA_FILES (extraction.rs included). All pre-fix .fxcache files
on PVC are invalidated at load time; ensure-fxcache regen produces
clean cache with new clamps applied.
Why this finally closes the chapter: per-writer fixes are reactive
(land after corruption hits prod). Boundary validation is
proactive — every future regression to extraction or normalization
trips the gate at load, not at epoch 5 of a 50-epoch run. The 5
layers are independent: a bug in any one leaves the others as
backstop.
Validation: cargo check -p ml --all-targets --offline clean.
NormStats unit tests (walk_forward.rs:706+) still pass — clamp +
validate are additive; existing test inputs are well within bounds.
Refs: SP5 Bug 2 (state[0] std=570 outliers on smoke-test-xb78r),
historical #191#210#214#193#195 chains.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
8434737a69