cnjfl wall-time analysis: 80% of training time was disk IO. The
MultiHorizonLoader was constructed fresh per epoch in the CLI loop,
forcing 9 file deserializations from bincode (~50s/file × 9 = 7-8
min) per epoch. For a 6-epoch run that was ~45 min of pure IO out
of ~50 min total.
Now loaders preload ALL files into RAM at construction (one-time
~7 min startup) and expose a `reset(seed: u64)` method that
re-seeds anchor sampling per epoch — no disk IO between epochs.
Memory cost: ~13-15 GB for the 9-quarter ES.FUT dataset (45M
snapshots × ~280 bytes). Well under the training pod's 64 GB
limit; current 16 GB request remains sufficient since the resident
set fits.
Expected wall-time impact at K=96, B=8, 16K seqs:
6 epochs: ~50 min → ~13 min (~3.7×)
15 epochs: ~120 min → ~23 min (~5×)
The internal LoadedFile-cache-with-cycling logic is gone — the
loader now holds Vec<LoadedFile> with all files resident. Per-call
`next_sequence` picks a uniformly random file + uniformly random
anchor inside it (instead of cycling files with a per-file budget).
Distribution is equivalent: each file contributes ~n_max_sequences /
n_files samples per epoch in expectation.
77 ml-alpha tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
3f089210f3