jgrusewski 8434737a69 fix(data): structural defense at data-loading boundary — 5 layers ...

Stops chasing one-off corruption bugs. Three+ historical fixes patched
specific writers (#191 fxcache column-0, Bug 1 target schema,
label_scale=5443 leaks, today's state[0] heavy-tail). Each new
corruption shape found the next hole. This installs a structural
defense so corruption is REJECTED at the data-loading boundary
regardless of source.

Five independent layers, each mandatory:

1. Bar-level sanity (baseline_common::sanitize_bars):
   drop bars with non-positive OHLC, high<low, non-finite values, or
   close/prev_close outside [0.5, 2.0]. Catches DBN parse glitches,
   broker tick errors, near-zero-open bars at source.

2. safe_log_return result clamp (extraction.rs:1246):
   ratio.ln().clamp(±0.1). Real-market 1-bar log returns rarely
   exceed ±0.05; ±0.1 traps every legitimate move while rejecting
   the corruption shape (corrupt bar with bar.open ≈ 0 → ln = -30
   → previously normalized to -30000-magnitude state[0] outliers).

3. validate_features pre-norm bound (extraction.rs:538):
   |val| ≤ 5.0 post-extraction. Pre-norm features come from
   safe_normalize ([0,1]/[-1,1]), safe_clip (max ±3), or clamped
   log-returns (±0.1); ±5 catches extractor invariant breaks.

4. NormStats::normalize post-norm clamp (walk_forward.rs:688):
   ((val - mean) / std).clamp(±20.0). Even if upstream produces
   outliers, every value uploaded to GPU is bounded.

5. Shared validate_normalized_features gate (walk_forward.rs):
   single source-of-truth invariant enforced at THREE sites:
     - fxcache fast path (after discover_and_load)
     - DBN fallback (after normalize_batch)
     - precompute writer (before fxcache write — never persist
       a poisoned cache)

Removed: DIAG_BUG2 + DIAG_BUG2_v2 one-shot diagnostics
(~125 lines of host-side download + outlier scan in
training_loop.rs). Replaced by structural defense — instrumentation
isn't needed when corruption can't reach state[0].

FEATURE_SCHEMA_HASH auto-bumps via build.rs FNV-1a hash over
SCHEMA_FILES (extraction.rs included). All pre-fix .fxcache files
on PVC are invalidated at load time; ensure-fxcache regen produces
clean cache with new clamps applied.

Why this finally closes the chapter: per-writer fixes are reactive
(land after corruption hits prod). Boundary validation is
proactive — every future regression to extraction or normalization
trips the gate at load, not at epoch 5 of a 50-epoch run. The 5
layers are independent: a bug in any one leaves the others as
backstop.

Validation: cargo check -p ml --all-targets --offline clean.
NormStats unit tests (walk_forward.rs:706+) still pass — clamp +
validate are additive; existing test inputs are well within bounds.

Refs: SP5 Bug 2 (state[0] std=570 outliers on smoke-test-xb78r),
historical #191 #210 #214 #193 #195 chains.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-02 14:02:53 +02:00

35 KiB

Raw Blame History

View Raw