Lays the consumer-side infrastructure for true E7 hindsight synthetic
injection. Phase 6.5b will follow with the producer wireup.
What lands:
1. EvalTrade.window_index + HindsightExperience.window_index fields
(host-side only; set in read_per_trade_tape from the w loop var
and propagated by compute_hindsight_labels).
2. GpuBacktestEvaluator.retained_states_buf — mapped-pinned
MappedF32Buffer sized [max_len × n_windows × state_dim_padded].
Populated by a DtoD copy after every launch_gather_chunk inside
submit_dqn_step_loop_cublas; layout matches chunked_states_buf
so the copy is a single contiguous block per chunk (no
transpose).
3. pub fn read_retained_state(window_idx, bar_idx) — zero-copy
host read via std::ptr::read_volatile on host_ptr (no
memcpy_dtoh per feedback_no_htod_htoh_only_mapped_pinned).
Mapped-pinned decision (jgrusewski review):
- Initial draft used CudaSlice<f32> + memcpy_dtoh for host read,
caught at review: violates feedback_no_htod_htoh_only_mapped_pinned.
- Refactored to MappedF32Buffer (cuMemHostAlloc DEVICEMAP). The
DtoD copy remains (rule forbids HtoD/DtoH, not DtoD; kernel
writes via dev_ptr aliasing pinned host memory). Caller must
sync eval stream before read_retained_state — production path's
consume_metrics_after_event already does this.
Memory cost at production cfg (max_len=200_000, n_windows=5,
state_dim_padded≈128): ~512 MB pinned host RAM. Substantial but
feasible on L40S host (192 GB+).
Files changed:
- crates/ml/src/cuda_pipeline/gpu_backtest_evaluator.rs: +retained
buffer + accessor; EvalTrade.window_index field
- crates/ml/src/trainers/dqn/trainer/enrichment.rs: HindsightExperience
.window_index field
- docs/dqn-wire-up-audit.md: 2026-05-11 audit entry
Verification (passing):
- cargo check -p ml --tests --features cuda: 0 errors
- cargo test -p ml --lib sp21_isv_slots: 3/3
- sp20_aggregate_inputs_test: 12/12
- sp20_phase1_4_wireup_test: 2/2
- sp20_emas_compute_test: 4/4
- sp20_controllers_compute_test: 7/7
- sp21_per_trade_predicted_q_test: 3/3
Total: 34 tests, 0 failures. Infrastructure works without exercising
it (accessor returns None until eval populates the retained buffer
— graceful degradation for test scaffolds bypassing the full eval).
After this commit (Phase 6.5b):
- Mapped-pinned synthetic-tuple scratch on GpuReplayBuffer
- New insert_synthetic_via_pinned API (raw dev_ptrs, no HtoD)
- training_loop hindsight injection wireup (~300 LOC)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wires EnrichmentResult::branch_lr_scale (E4's [f32; 4] clamped [0.5,
2.0] per-branch LR multiplier) into the DQN Adam optimizer. Splits
the existing single DqnBranches Adam sub-launch into 4 per-branch
sub-launches, each consuming its own LR scale from ISV[521..525).
Plan amendment from on-paper design:
- Plan said "via existing per-group Adam infrastructure" but per-group
operates at PARAM_GROUP granularity (8 groups, all 4 action branches
lumped into DqnBranches). E4 needs per-action-BRANCH granularity.
- Resolution: split DqnBranches sub-launch into 4 per-branch sub-launches
using the already-canonical branch byte ranges (4 param tensors per
branch: Dir [17..21), Mag [21..25), Order [25..29), Urgency [29..33)).
Coverage invariant updated to 7-way (was 4-way).
- Pearl C engagement tracking on branches DEFERRED to Phase 4.5: the
shared DqnBranches engagement counter offset would collide on writes
if 4 sub-launches use the same offset. Pearl C is a diagnostic system
(not load-bearing) so its temporary unavailability for branches is
acceptable; Trunk + Value + Trunk-extras Pearl C still active.
Phase 4.5 follow-up will re-instate via ParamGroup expansion or
sub-block offsetting scheme.
ISV slot allocation:
- BRANCH_LR_SCALE_{DIR,MAG,ORDER,URGENCY}_INDEX = 521..525
- ISV_TOTAL_DIM 521 → 525 (bus extension)
- Layout fingerprint adds 4 SLOT entries
- New branch_lr_scale_index(branch_idx) accessor for clean mapping
- Cold-start floor: launcher reads ISV, floors at 1.0 if at sentinel
0.0 (per pearl_first_observation_bootstrap — first emit replaces
directly with no intermediate state)
ABI surgery:
- dqn_adam_update_kernel: ONE new arg float lr_scale at end of
signature; lr = *lr_ptr * lr_scale inside kernel
- 5 callers migrated atomically per feedback_no_partial_refactor:
- launch_adam_update (main DQN): 7 sub-launches with per-branch
lr_scale (Trunk + Value + 4 branch + Trunk-extras)
- 4 post-aux launchers (ofi_embed/aux_trunk/denoise/sel) pass 1.0
- decision_transformer Adam launch passes 1.0
Producer wireup (training_loop.rs post-enrichment block):
```rust
for (branch_idx, &scale) in result.branch_lr_scale.iter().enumerate() {
fused.trainer().write_isv_signal_at(
branch_lr_scale_index(branch_idx),
scale,
);
}
```
Files changed:
- crates/ml/src/cuda_pipeline/sp21_isv_slots.rs: +4 slots + accessor + tests
- crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs: ISV_TOTAL_DIM bump +
fingerprint + 7-way Adam split with per-branch lr_scale
- crates/ml/src/cuda_pipeline/dqn_utility_kernels.cu: lr_scale arg + apply
- crates/ml/src/cuda_pipeline/decision_transformer.rs: lr_scale=1.0
- crates/ml/src/trainers/dqn/trainer/training_loop.rs: write 4 ISV slots
- docs/dqn-wire-up-audit.md: 2026-05-11 audit entry
Verification (passing):
- cargo check -p ml --tests --features cuda: 0 errors
- cargo test -p ml --lib sp21_isv_slots --features cuda: 3/3 (bus bounds
+ slot uniqueness + branch index mapping)
- sp20_aggregate_inputs_test: 12/12
- sp20_phase1_4_wireup_test: 2/2
- sp20_emas_compute_test: 4/4
- sp20_controllers_compute_test: 7/7
- sp21_per_trade_predicted_q_test: 3/3
Total: 31 tests, 0 failures. Behavioral gate (per-branch LR divergence)
is the upcoming smoke training run.
After this commit (T2.2 Phases 5-7 + 8 + 4.5):
- Phase 5: E6 winner indices → PER priority bumps
- Phase 6: E7 hindsight → replay buffer injection
- Phase 7: E8 curriculum weights → segment sampling
- Phase 8: signal-drive remaining controller GAINS
- Phase 4.5: re-instate Pearl C engagement tracking for branches
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 1 of the foxhunt specialized agents/skills rollout is complete: 14
commits, 5 auditor agents, 7 workflow/maintenance skills, 2 helper scripts,
warn-only PostToolUse hook router. All 8 acceptance criteria from the spec
verified. Hook latency 11-13 ms per call (target <200 ms). Memory-write
invariant held: only pearl-distiller and memory-curator are authorized
writers, no agent-driven memory edits during rollout.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replicates the user's second/third critical-review iteration cycle. Cites
every claim against a memory file. Enforces no-deferrals-for-complementary-
fixes, ISV-slot enumeration, smoke kill conditions, anti-pattern callouts.
Composes the four foxhunt auditor agents for domain-specific verification.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Classifies worktrees (NO-COMMITS / GONE / MERGED / UNCOMMITTED / ACTIVE) and
proposes cleanup. User confirms each removal individually; never --force,
never -D. Composes commit-commands:clean_gone where applicable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Audits memory/ corpus: stale references (gone files/symbols/flags/commits),
duplicates, superseded pearls, orphans (file vs MEMORY.md index drift).
Proposes archive/merge/update; never auto-deletes. Archive ≠ delete.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Monitors argo smoke training; kills on first useful anomaly signal per
stop-on-anomaly + kill-quickly discipline. Distinguishes anomaly from
metric-pipeline inflation. Streams via argo logs -f, no run_in_background
Monitor layering.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
One of two authorized writers into memory/ (the other is memory-curator).
Templatizes new-pearl creation: structured body (pattern / detection signal /
fix / canonical commit:file:line / related pearls) + MEMORY.md index entry
in the right section, ≤150 chars.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Templatizes the ISV-slot scaffolding step. Reproduces the c146c4fff commit
shape: spN_isv_slots.rs with named constants, ISV_TOTAL_DIM bump at canonical
site, fold-boundary reset registration. Enforces wire-everything-up discipline.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Templatizes the SP design spec format used in docs/superpowers/specs/.
Enforces pearl/feedback grounding, ISV-slot enumeration, smoke kill conditions,
sister-fix check (no-deferrals-for-complementary-fixes pearl).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pattern-cluster auditor for general code-quality rules: no stubs, no TODO,
no hiding, no legacy aliases, no feature flags, no partial refactors,
v7-gem methodology, invariant tests not observed-value tests, no deferrals
of complementary fixes. Bundles 12 feedback rules and 3 pearls.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pattern-cluster auditor for ISV-driven adaptive bounds, controller anchors,
EMA bootstrap, Wiener-α floors, blend formulas, per-branch budgets, per-group
Adam, Kelly floors, trail-stop. Bundles 2 feedback rules and 16 pearls.
Read-only; cites memory file by name on every finding.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pattern-cluster auditor for CUDA/GPU host code. Bundles 6 feedback rules and
7 pearls (no atomicAdd, mapped-pinned, no nvrtc, f64/f32 ABI, no host branches
in graph capture, fused per-group stats, build.rs rerun, canary launch order).
Read-only; cites memory file by name on every finding.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two Important review findings:
1. docs/superpowers/plans/*.md missing from sp-critical-reviewer routing
table — sp-critical-reviewer would never self-suggest on plan files
(its primary use case).
2. STATE_FILE and REPO_ROOT diverged when CLAUDE_PROJECT_DIR was unset
(one used "." relative, the other "$(pwd)" absolute). Compute REPO_ROOT
first and derive STATE_FILE from it; remove the duplicate later
assignment.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds .claude/helpers/foxhunt-audit-router.sh that suggests foxhunt-* auditor
agents based on edited file path. Warn-only, <200ms, never blocks. Dedup
state file .claude/.foxhunt-audit-state cleared at SessionStart by sibling
script. Settings.json additive merge — existing hooks preserved.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
14-task plan covering: foundation hook router (Task 1), 5 auditor agents
(Tasks 2-5, 13), 5 workflow skills (Tasks 6-10), 2 maintenance skills
(Tasks 11-12), end-to-end acceptance check (Task 14). Tasks 2-5, 6-8, 9-10,
11-12 fan out in parallel. Task 13 (sp-critical-reviewer) composes the four
domain auditors and is built last.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Promotes the Phase 2.5 follow-up from the plan to a load-bearing test.
Three #[ignore = "requires GPU"] tests exercise backtest_env_step_batch
directly with controlled inputs:
1. predicted_q_populated_on_close_with_real_q_values — open Long Full
at step 0 with q_values_per_window[w=0, a=Long_Full]=2.5, hold for
two bars, close Flat at step 3. Asserts the per-trade tape's
predicted_q[0] ≈ 2.5 (entry_q captured at open, persisted in
entry_q_state across holds, snapshotted before close emit).
2. predicted_q_stays_zero_when_q_values_is_null — same scenario with
q_values_per_window=NULL (single-step evaluate() semantics). The
kernel's open-time write is gated on the NULL guard, so
entry_q_state[w] stays at buffer-init 0.0 and the tape's
predicted_q is 0.0. Proves the NULL-tolerance contract is
kernel-enforced, not just launcher convention.
3. no_trades_no_predicted_q_emission — all-Flat action sequence
produces zero trades. Path-coverage check that entry_q's open
guard fires only on actual opens / reverses.
Test design:
- Kernel-direct (loads ENV_CUBIN via include_bytes!), no eval-pipeline
scaffolding (no QValueProvider, no cuBLAS forward, no chunked state
gather). Avoids the heavy mock infrastructure the plan flagged.
- Flat-market 4-bar single-window window: every OHLC=100, zero costs,
initial_capital=100k. Isolates the entry_q signal from P&L noise
so the assert-predicted_q is exact under IEEE-754 (kernel writes
the Q-value verbatim with no math).
- NULL fallback for isv_signals/conviction/exploration_scale matches
the existing single-step launcher pattern; FEATURE_DIM=4 (<41) ⇒
compute_regime_trail_scales takes the fixed-width fallback (no
feature deref needed).
Verification:
- SQLX_OFFLINE=true cargo test -p ml --test sp21_per_trade_predicted_q_test
--features cuda -- --ignored --nocapture
- 3/3 pass on RTX 3050 Ti (sm_86).
- All 4 prior SP20 GPU oracle suites still pass (25 tests).
- Total: 28 GPU oracle tests, 0 failures.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the multi-step T2.2 work. Phase 1.5 captures entry_q (predicted
Q at trade open) on the per-trade tape; Phase 2 wires
enrichment::run_enrichments to the real GpuBacktestEvaluator
per-trade tape and deletes the fake-trade synthesizer
(extract_eval_trades_from_metrics) per feedback_no_stubs.
Plan amendment from on-paper design:
- Audit caught plan's "portfolio_state[ps+6] is unused" claim was
wrong — slot 6 is in active use as cum_return. Replaced with a
dedicated entry_q_state_buf [n_windows] separate from
portfolio_state. Doesn't touch shmem layout, gather kernel, or
model state-dim.
- E5 design question resolved as option (2): quartile-spread Sharpe
with ISV-driven significance anchor read from
ISV[VAL_SHARPE_VAR_EMA_INDEX=351] (per
pearl_controller_anchors_isv_driven). Hardcoded 1.5σ/0.5σ
thresholds eliminated; the noise floor is the val_sharpe variance
EMA already produced per epoch by the early-stopping pipeline.
- Single-step evaluate() launcher passes NULL q_values_per_window
(forward_fn closure exposes only action indices, not Q-values);
same NULL-tolerant pattern as exploration_scale_ptr et al. The
production val pipeline (chunked path) DOES wire real Q-values.
Files (atomic per feedback_no_partial_refactor):
- backtest_env_kernel.cu: 4 new args at end of both kernels
(entry_q_state, q_values_per_window, num_actions,
per_trade_predicted_q_out); pre_entry_q snapshot + close emit +
open/reverse capture.
- gpu_backtest_evaluator.rs: entry_q_state_buf + per_trade_predicted_q_buf
allocated; both reset in reset_evaluation_state; both launchers
migrated; EvalTrade.predicted_q field added; read_per_trade_tape
populates it.
- trainer/enrichment.rs: EvalTrade is now pub(crate) use re-export
from gpu_backtest_evaluator (drops ensemble_var); E5 refactored
to quartile-spread Sharpe with ISV-driven anchor;
extract_eval_trades_from_metrics deleted; HindsightExperience
retained (Phase 6 wiring).
- trainer/training_loop.rs: enrichment block replaced with
evaluator.read_per_trade_tape() + ISV slot 351 read; val_bars /
real_trade_count / real_total_pnl / real_win_rate plumbing
dropped.
- trainer/{mod,constructor,metrics}.rs: last_val_metrics field
removed (last consumer gone — feedback_no_hiding).
- docs/dqn-wire-up-audit.md: 2026-05-11 audit entry.
Verification (passing):
- SQLX_OFFLINE=true cargo check -p ml --tests --features cuda
- sp20_aggregate_inputs_test (12/12)
- sp20_phase1_4_wireup_test (2/2)
- sp20_emas_compute_test (4/4)
- sp20_controllers_compute_test (7/7)
After-this scope (Phase 3-7 + Phase 8 in T2.2 multi-phase):
- E1 q_correction → ISV slot consumer
- E4 per-branch LR scaling via per-group Adam
- E6 winner indices → PER priority bumps
- E7 hindsight → replay buffer injection
- E8 curriculum weights → segment sampling
- Signal-drive remaining controller GAINS (0.9/1.1 in E5; 2.0/0.5/-0.5 in E2)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the continuation plan for SP21 T2.2 Phase 1.5 (entry_q tracking
in portfolio_state slot 6) + Phase 2 (wire enrichment to real
per-trade tape from gpu_backtest_evaluator). Self-contained plan
that a fresh session can pick up cold:
- State-at-session-start summary with all 8 prior commits
- Phase 1.5 design (storage slot, capture site, plumbing,
EvalTrade extension)
- Phase 2 design (training_loop wire-up, enrichment refactor,
E5 alternate-signal options with recommendation)
- Hard rules carried from prior session (no NULLs, no stubs,
atomic per feedback_no_partial_refactor)
- Verification plan (5 GPU oracle test suites)
- Open design question (E5 alternate signal) with recommendation
- Multi-phase continuation map (Phases 3-7 + Phase 8)
- Final cascade verification (smoke run criteria)
Also amends the parent SP21 plan
(2026-05-10-sp21-train-eval-coherence-isv-defrost.md) with the
T2.2 multi-phase scope section that documents the full per-trade-tape
expansion the user committed to (option 3 — full wiring across
multiple sessions instead of the original recommended option (b)
aggregate-stats refactor).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces Step A's NULL launcher passes with real device buffers.
Both kernel launch sites in gpu_backtest_evaluator.rs (backtest_env_step
and backtest_env_step_batch) now pass real per-trade tape pointers.
The kernel's per-trade emission block fires unconditionally on close
events — single-threaded per-window writes preserve event ordering and
enable race-free counter increment without atomicAdd.
New constants + types:
- MAX_TRADES_PER_WINDOW = 200_000 (typical eval window bar count;
per-window memory: 4 SoA buffers × 4 bytes × 200k = 3.2MB)
- pub struct EvalTrade with 5 fields: bar_index, pnl, holding_bars,
direction, magnitude. Does NOT include predicted_q / ensemble_var
— those need entry-time captures (entry_q, entry_var in portfolio
state) deferred to Phase 1.5.
New struct fields on GpuBacktestEvaluator:
- per_trade_pnl_buf: CudaSlice<f32> [n_windows × MAX_TRADES]
- per_trade_holding_bars_buf: CudaSlice<u32>
- per_trade_bar_index_buf: CudaSlice<u32>
- per_trade_dir_mag_buf: CudaSlice<u32> (packed dir/mag)
- per_trade_count_buf: CudaSlice<u32> [n_windows]
New methods:
- reset_per_trade_tape (folded into reset_evaluation_state): zeros
the count buffer at the start of each eval window. SoA value
buffers don't need zeroing — read up to count[w] only.
- pub fn read_per_trade_tape(&self) -> Result<Vec<EvalTrade>, MLError>:
reads count buffer first (cheap), early-returns empty if no trades,
else reads 4 SoA buffers (~16MB DtoH at PCIe ≈ 1ms) and flattens
window-major into chronological Vec<EvalTrade>.
Phase 2 follow-up (next commit) — wire read_per_trade_tape to the
enrichment caller in training_loop.rs:1510-1568, replacing
extract_eval_trades_from_metrics (the fake-trade synthesizer).
Phase 1.5 follow-up (if Phase 2 keeps E1+E5) — add entry_q + entry_var
to portfolio_state at trade open, extend per-trade tape with 6th/7th
SoA buffers.
Affected files:
- crates/ml/src/cuda_pipeline/gpu_backtest_evaluator.rs
(constants, struct fields, alloc, construction, 2 launcher sites,
reset_evaluation_state addition, read_per_trade_tape method)
Verification:
- cargo check -p ml --tests: passes (warnings only)
- GPU oracle tests: behavior preserved by construction (existing
WindowMetrics aggregator unaffected — separate kernel)
Plan reference: docs/plans/2026-05-10-sp21-train-eval-coherence-isv-defrost.md
T2.2 multi-phase scope; Phase 1 Step B closure.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extends backtest_env_step + backtest_env_step_batch kernel signatures
with 6 new NULL-tolerant args for per-trade tape emission:
- float* per_trade_pnl_out
- unsigned int* per_trade_holding_bars_out
- unsigned int* per_trade_bar_index_out
- unsigned int* per_trade_dir_mag_out (packed hi-16 dir, lo-16 mag)
- unsigned int* per_trade_count
- int max_trades_per_window
Each kernel snapshots entry_price + hold_time BEFORE the
unified_env_step_core call, then post-call mirrors
record_kelly_trade_outcome's close-detection predicate
(is_exit || is_reversal with positive entry_price + valid pre-trade
equity guards) to recompute the realized return for per-trade tape
emission. Single thread per window writes its own slot — race-free
counter increment without atomicAdd per feedback_no_atomicadd.
Step A only — Step B (host alloc + readback + enrichment integration)
follows in a separate commit. This commit is the kernel ABI extension
with NULL launchers, bit-identical to pre-Phase-1 behavior.
NULL-tolerant ABI extension is the codebase's canonical phased-rollout
pattern. Same shape as alpha_per_env / is_win_per_env additive contracts
in sp20_aggregate_inputs_kernel. Per feedback_no_partial_refactor's
"consumer migrates with contract change" rule, the consumer (launcher)
MIGRATES here by passing NULL — output behavior preserved bit-identically.
Step B (next commit):
- Allocate per-trade buffers in GpuBacktestEvaluator constructor
- Pass real device pointers from launcher
- Reset per-window counter at fold start
- Host-side readback into Vec<EvalTrade>
- Wire to enrichment caller, replace extract_eval_trades_from_metrics
Affected files:
- crates/ml/src/cuda_pipeline/backtest_env_kernel.cu (both kernels)
- crates/ml/src/cuda_pipeline/gpu_backtest_evaluator.rs (both launchers)
Verification:
- cargo check -p ml --tests: passes (warnings only)
- GPU oracle tests: NULL-tolerance contract preserved by construction
(per_trade_count == NULL gates the entire emission block)
Plan reference: docs/plans/2026-05-10-sp21-train-eval-coherence-isv-defrost.md
T2.2 multi-phase scope section.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes two architectural-debt items from SP21 Tier 2.
T2.1 — check_early_stopping(avg_q_value) deleted entirely:
- Combined two failed mechanisms: (a) Q-value floor — not a learning
signal (high Q can mean edge OR value explosion, indistinguishable);
(b) Sharpe plateau with hardcoded `improvement < 0.01` threshold,
structurally meaningless against typical val-sharpe deltas O(1-10).
- Both subsumed by the SP21 T1.1a+T1.1b val-loss patience early-stop
with signal-driven min_delta from VAL_SHARPE_VAR_EMA.
- Legacy `old_should_stop` branch + function body deleted.
- Per feedback_no_legacy_aliases.
T2.4 — MIN_HOLD_TARGET / MIN_HOLD_PENALTY_MAX #defines deleted:
- Investigation: macros referenced ONLY in comments and the defining
line itself — no actual code use. The SP12 v3 production callers
were removed in SP20 Phase 2 Task 2.2.
- HEALTH_DIAG line at training_loop.rs:5159 updated to drop the dead
30.0/3.0 literals.
- Scope boundary: MIN_HOLD_TEMPERATURE_* chain is NOT a zombie —
actively wired (kernel producer + SP16 controller consumer).
- Per feedback_no_legacy_aliases.
T2.5 — PER hyperparams disposition (no code change):
- per_alpha=0.6, per_beta_start=0.6 are paper-canonical (Schaul et al.).
Per the SP21 plan recommendation, kept fixed for SP21. Filed for a
separate SP if later identified as a leverage point.
Affected files:
- crates/ml/src/trainers/dqn/trainer/metrics.rs:435-488
(check_early_stopping body deleted)
- crates/ml/src/trainers/dqn/trainer/training_loop.rs (7253 caller +
7291-7322 old_should_stop branch + 5159-5167 HEALTH_DIAG line)
- crates/ml/src/cuda_pipeline/state_layout.cuh:317-318
(#defines deleted)
Verification:
- cargo check -p ml --tests: passes (warnings only)
Cumulative SP21 Tier 2 status: T2.1 ✓, T2.4 ✓, T2.5 ✓ (deferred-doc).
T2.2+T1.3 (enrichment.rs constants soup, ~400 LOC) remaining.
Plan reference: docs/plans/2026-05-10-sp21-train-eval-coherence-isv-defrost.md
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the patience-based early-stopping bug that ran xmd6b 30 epochs
past peak val performance (epoch 2: val_Sharpe=90, total_pnl=0.44 →
epoch 30: val_Sharpe=26, total_pnl=0.16) — a 70% loss of alpha to
training-induced overfitting.
Two intertwined bugs, fixed atomically:
T1.1a (wrong-source) at training_loop.rs:7234:
- was: self.early_stopping.should_stop(-log_output.epoch_sharpe, epoch)
reading the TRAINING ROLLOUT Sharpe (Thompson-noisy, in-sample,
oscillates even when the model is frozen)
- now: self.early_stopping.should_stop(log_output.val_loss, min_delta, epoch)
reading the deterministic-backtest val_loss
- The comment 5733 lines earlier (line 1499) explicitly says "Use
val_Sharpe (deterministic backtest), NOT epoch_sharpe" — patience
path was the inconsistency, backtracking already honored it.
T1.1b (hardcoded threshold) in early_stopping.rs:
- was: EarlyStopping::new(patience, min_delta) with min_delta=0.001
constructor-set, struct field, structurally meaningless
against the val_loss noise floor (typical val_sharpe deltas
are O(1-10), so 0.001 essentially never gates)
- now: EarlyStopping::new(patience), should_stop(val_loss, min_delta,
epoch) with min_delta computed per-call from
ISV[VAL_SHARPE_VAR_EMA_INDEX=351] as
sqrt(var_ema).max(0.5)
- Floor 0.5 covers cold-start before var_ema bootstraps from sentinel
per pearl_blend_formulas_must_have_permanent_floor.
- Per feedback_isv_for_adaptive_bounds + feedback_adaptive_not_tuned.
T2.3 (test signature update) absorbed:
- 6 existing unit tests migrated to new should_stop signature.
- 1 NEW test (test_min_delta_can_change_per_call) verifying per-call
threshold change works correctly.
- EarlyStopping::min_delta struct field deleted.
- Atomic per feedback_no_partial_refactor.
Affected files:
- crates/ml/src/trainers/dqn/early_stopping.rs (struct + tests)
- crates/ml/src/trainers/dqn/trainer/constructor.rs (new() arg)
- crates/ml/src/trainers/dqn/trainer/training_loop.rs (call site)
Verification:
- cargo check -p ml --tests: passes
- cargo test -p ml --lib early_stopping: 8/8 pass
Behavioral expectation post-fix: xmd6b-shape runs (val_Sharpe rising
31→90 epochs 0-2, declining 90→26 epochs 3-30) will trigger early-stop
near the peak. With patience=5 and var_ema bootstrapping by epoch 2-3,
the controller detects "no improvement of ≥ 1σ for 5 consecutive
epochs" by ~epoch 7-8 and stops, saving ~22 epochs of overfitting.
Plan reference: docs/plans/2026-05-10-sp21-train-eval-coherence-isv-defrost.md
Tier 1 status: T1.1a ✓, T1.1b ✓, T2.3 ✓ (this commit). T1.2 + T1.4 next.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the Phase 3.2 forward-reference loop in the SP20 aggregator.
Previously `out_inputs->per_bar_hold_reward = 0.0f` was hardcoded; the
per-bar Hold opportunity-cost producer existed
(experience_kernels.cu:3823 — `per_bar_opp_cost = -aux_conf × cost_scale`)
and wrote to `hold_baseline_buffer` and `r_micro` directly, but never
reached HOLD_REWARD_EMA. Result: HOLD_REWARD_EMA frozen at sentinel 0.0
across all observed epochs; the SP20 reward centering loop
(`r_micro += per_bar_opp_cost - HOLD_REWARD_EMA`) stayed uncentered,
biasing the policy's reward signal away from zero-mean.
T3.3 wireup mirrors the T2.2 alpha refactor: a per-env scratch buffer
that the producer writes at every step (alongside the existing
`hold_baseline_buffer` write), and the aggregator reads at the same
step. Sums opp_cost over Hold-direction envs only; emits the mean as
`per_bar_hold_reward`. The HOLD_REWARD_EMA's gate (`hold_fraction > 0.5f`
from T3.2) preserves the strict-majority semantic from before.
Atomic across producer site, kernel signature, aggregator, launcher,
collector, and tests (per feedback_no_partial_refactor):
- experience_kernels.cu — new `float* per_bar_opp_cost_per_env`
kernel arg, NULL-tolerant write at line ~3823.
- sp20_aggregate_inputs_kernel.cu — new arg, 6th shmem stripe
(`sh_opp_cost_sum`), per-thread Hold-gated accumulation, tree
reduction extended to 6 stripes, output `per_bar_hold_reward =
opp_cost_sum / hold_count` when hold_count > 0 else 0.
- sp20_aggregate_inputs.rs — launcher signature, dynamic_shmem_bytes
5→6 stripes, doc table, internal test.
- gpu_experience_collector.rs — new `per_bar_opp_cost_per_env:
CudaSlice<f32>` field, alloc, struct construction, kernel-arg
pass at both experience_env_step and sp20_aggregate_inputs
launches (raw_ptr).
- sp20_aggregate_inputs_test.rs — helper renamed
`run_kernel_with_is_win_and_opp_cost`, 4 existing sites pass
NULL, 2 NEW oracle tests verifying real producer + NULL fallback.
- sp20_phase1_4_wireup_test.rs — NULL fallback at the wireup site;
HOLD_REWARD_EMA-stays-at-zero assertion remains valid.
Verification:
- cargo check -p ml --tests: passes (warnings only)
- cargo test -p ml --test sp20_aggregate_inputs_test --features cuda
-- --ignored: 12/12 GPU oracle tests pass on RTX 3050 Ti, including
both new T3.3 tests (per_bar_hold_reward_means_over_hold_envs_only,
null_per_bar_opp_cost_emits_zero).
- cargo test -p ml --test sp20_phase1_4_wireup_test --features cuda
-- --ignored: 2/2 pass under the new NULL-tolerant contract.
Plan reference: docs/plans/2026-05-10-sp21-train-eval-coherence-isv-defrost.md
Tier 3 status: T3.1 ✓, T3.2 ✓, T3.3 ✓ (this commit), T3.4 withdrawn,
T3.5 cascade-pending, T3.6 withdrawn.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SP21 Tier-3 foundation: defrost two production-frozen ISV slots that
pinned at 0 across every observed training epoch (d7bj7, xmd6b). Both
bugs in the same aggregator kernel, same structural shape: per-step
binary majority-vote indicators feeding fractional EMAs.
T3.1 — WR_EMA defrost (sp20_aggregate_inputs_kernel.cu):
- was: is_win_out = (2 * wins_count >= closed_count) ? 1 : 0
binary "majority won this step" indicator
- now: win_fraction_out = (float)wins_count / (float)closed_count
fractional in [0, 1]
- With actual val WR≈0.46, the binary signal was 0 most of the time,
pinning WR_EMA at 0 across 30+ epochs in xmd6b. EMA now converges
to population win rate.
T3.2 — HOLD_PCT_EMA defrost (same kernel, same pattern):
- was: action_is_hold_out = (hold_count * 2 > n_envs) ? 1 : 0
strict-majority indicator
- now: hold_fraction_out = (float)hold_count / (float)n_envs
- Cascade: HOLD_COST_SCALE controller compared hold_pct_ema=0 to
tgt±0.05, always saw < lower, ramped × 0.95 → clamped at floor
0.01 (the hold_cost_scale=0.0100 observation in d7bj7 logs).
T3.5 expected to cascade-fix in next training run.
- HOLD_REWARD_EMA gate preserves strict-majority semantic via
hold_fraction > 0.5f test in the consumer kernel.
Atomic across struct fields, kernel logic, Rust mirror, byte
serialization, doc tables, and 4 test files (per
feedback_no_partial_refactor):
- sp20_aggregate_inputs_kernel.cu (struct + 2 computations)
- sp20_emas_compute_kernel.cu (struct + reader + gate)
- sp20_aggregate_inputs.rs (doc table)
- sp20_emas_compute.rs (Rust struct + serialize + 3 tests)
- sp20_aggregate_inputs_test.rs (reader sig + 4 test assertions)
- sp20_emas_compute_test.rs (5 struct literal updates)
- sp20_phase1_4_wireup_test.rs (HOLD_PCT_EMA expected 0.625)
Verification:
- cargo check -p ml --tests: passes (warnings only)
- cargo test -p ml --lib sp20_emas: 6/6 unit tests pass
Pearl candidate: binary-majority aggregator over a fractional
underlying signal cannot serve as input to a fractional-target EMA.
The previous fix (commit 64bbbe418) addressed the per-bar vs segment
predicate at the producer site, but didn't notice the aggregator's
binarization step still collapsed the fraction to {0, 1}. Two bugs
in series, both now resolved.
Plan reference: docs/plans/2026-05-10-sp21-train-eval-coherence-isv-defrost.md
Tiers 3.3-3.6 remaining; T3.5 expected to cascade-fix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reverts the forced H=30 diagnostic in aux_horizon_update_kernel.cu
(commit c78c4766c) — the experiment confirmed the data ceiling
hypothesis (aux_dir_acc dropped 0.46→0.50 with pred_tanh collapsing
to ~0, opposite of the bootstrap-failure prediction). Restores the
adaptive Pearl-A + Wiener-α floor logic.
Adds SP21 plan: train/eval coherence + ISV defrost. Catalogs 26
findings across 5 tiers from a pair-audit of MinIO-archived training
logs (xmd6b 30 epochs, d7bj7 2 epochs).
Cross-cutting principle: every threshold or bound in training
control flow must be signal-driven from an ISV slot, not hardcoded
(per feedback_isv_for_adaptive_bounds + feedback_adaptive_not_tuned).
Canonical pattern documented: ISV[CURIOSITY_PRESSURE_INDEX=346]
(SP11 Fix 39) is the reference implementation.
Meta-finding: across SP3-SP20 we've been monitoring training-rollout
metrics (Thompson-noisy) instead of val metrics (deterministic
backtest). val_PF=1.18-1.33 with WR=46-48% across 30 epochs of
xmd6b shows the policy DOES extract asymmetric-payoff alpha — we
just couldn't see it through the meter inflation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The multi-seed path already does `kubectl apply` before `argo submit`,
so cluster template stays in sync with source. The single-job path used
`argo submit --from=wftmpl/train` directly, expecting the cluster's
template to already match — which silently drifts when defaults change.
Caused workflow train-jpxvn (2026-05-10) to dispatch with stale
imbalance-bar-threshold=0.5 default (the cluster's old value) when the
source had been bumped to 20.0. Triggered near-OOM in feature extraction.
One-line fix: apply the template before submission. Mirrors what
multi-seed already does. No behavior change for users who pass explicit
flags; just makes implicit defaults track source.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Throwaway diagnostic edit on aux_horizon_update_kernel.cu. Tests whether
the adaptive horizon's self-defeating loop (WR=50% → H~2 → noise horizon →
53.5% acc → WR=50%) is the actual bottleneck. If aux_dir_acc rises to
58%+ at fixed H=30, bootstrap failure confirmed and the proper fix is
constraint-based (min hold time during exploration). Audit-doc updated.
DO NOT merge to mainline — branch is throwaway after the experiment.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Threads `self.aux_conf_at_state_buf` into the `c51_loss_batched` launch
in `GpuDqnTrainer::launch_c51_loss`. Position matches the kernel's
appended trailing arg from the previous commit.
Tests added in `crates/ml-dqn/src/gpu_replay_buffer.rs::tests`:
- `aux_gate_high_confidence_passes_full_target` (CPU pure-math):
gate(aux_conf=0.5, threshold=0.10, temp=0.05) > 0.99 proves
high-confidence reward pass-through.
- `aux_gate_low_confidence_attenuates_reward` (CPU pure-math):
gate(aux_conf=0.02, threshold=0.10, temp=0.05) < 0.20 proves
the uncertain-state neutralizer semantic.
- `aux_gate_temp_floor_keeps_gate_finite` (CPU pure-math):
sweeps {temp, aux_conf, threshold} and asserts finite gate ∈ [0,1]
across the ISV-controllable parameter range — proves the
fmaxf(temp, 1e-3) floor keeps the kernel numerically safe.
- `aux_conf_direct_to_trainer_gather_populates_destination` (GPU
behavioral): wires a fresh CudaSlice<f32> as the trainer
destination, inserts 8 transitions with strictly-positive distinct
aux_conf values, samples 1, asserts the trainer destination
buffer post-sample holds a value from the inserted set (NOT the
alloc_zeros sentinel) — proves the direct-gather wiring actually
populates the trainer buffer with non-trivial data.
All 3 CPU math tests + 1 GPU integration test pass on RTX 3050.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the Phase 5 consumer kernel-side gate. New kernel arg
`const float* __restrict__ aux_conf_at_state` appended to
`c51_loss_batched`'s signature. Gate computation runs once per sample
at the kernel-entry reward-setup site (after the #27 ensemble-
disagreement adjustment), then the gated `reward` propagates through
every branch's `block_bellman_project_f` call without per-branch changes.
Formula:
gate = sigmoid((aux_conf - threshold) / temp)
reward = gate * reward
where:
threshold = ISV[AUX_CONF_THRESHOLD_INDEX=518]
temp = max(ISV[AUX_GATE_TEMP_INDEX=519], 1e-3)
Mathematical interpretation: at low aux confidence (gate→0),
`r_used → 0`, so the Bellman target becomes `gamma * Q(s', a')`. The
Q value at the current state collapses toward `gamma * Q(s', a')` —
model gets no reward feedback on uncertain transitions. Effectively
"don't update Q on uncertain transitions" — the "uncertain-state
neutralizer" semantic from the Phase 3 Task 3.4 audit doc spec §4.4.
NULL-tolerant: `aux_conf_at_state == NULL` OR `isv_signals == NULL`
⇒ gate skipped (identity, no-op = pre-Phase-5 behaviour). Test
scaffolds without a wired aux head still work.
Out of scope: `iqn_dual_head_kernel.cu` — IQN is the auxiliary loss,
C51 is production. Gating IQN is more complexity for marginal gain.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 5 plumbing — consumer-side wire-up. Allocates `aux_conf_at_state_buf:
CudaSlice<f32>` ([batch_size]) on `GpuDqnTrainer`, exposes the raw_ptr via
`aux_conf_at_state_buf_ptr()` and the `FusedTrainerCtx` delegating accessor
`trainer_aux_conf_at_state_buf_ptr()`, and invokes
`GpuReplayBuffer::set_trainer_aux_conf_ptr` at both fused_ctx init sites in
training_loop.rs (init + re-init, atomic per `feedback_no_partial_refactor`).
The PER `gather_f32_scalar` now writes the SAMPLED bar's per-batch aux_conf
directly into the trainer's f32 buffer on every step — same direct-to-trainer
pattern as the SP13 B1.1b `aux_nb_label_buf` (i32) wire-up immediately above
the new call. The c51_loss_batched reward gate (lands in the next commit)
reads this buffer to compute `gate = sigmoid((aux_conf - ISV[AUX_CONF_THRESHOLD])
/ ISV[AUX_GATE_TEMP])` and applies `r_used = gate * reward` at the Bellman
projection.
`alloc_zeros` cold-start: 0.0 sentinel → at threshold ≈ 0.10 and temp ≈ 0.05
the gate is `sigmoid(-2) ≈ 0.12` → reward is mostly suppressed pre-population.
This is the "graceful degradation" semantic from the Phase 3 Task 3.4 audit
doc spec §4.4. Once PER's direct-gather populates from the producer ring on
the first sample step, the per-bar aux_conf values drive the gate as designed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
L40S (48GB) and H100 (80GB) have plenty of headroom; the 2GB cap was a
conservative leftover that tripped on workflow zgjgc with 17.8M imbalance
bars (3.2GB). 8GB budget leaves ~28GB free on L40S after model + activations
+ workspace.
Updates both call sites:
- DqnGpuData::upload_slices (training data, the failing one on zgjgc)
- PpoGpuData::upload (market data, same constant)
Error message format strings updated 2.0 GB → 8.0 GB.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bottleneck B of two: replaces the sequential `ImbalanceBarSampler` walk
in `mbp10_to_imbalance_bars` with a two-pass parallel decomposition.
Bit-identical to sequential — verified by 6/6 passing tests in
`crates/ml-features/tests/imbalance_bars_parallel_bit_equiv_test.rs` at
both dense (T=20, ~23k bars) and sparse (T=500, ~300 bars) emission
regimes, exact 0.0 diff on every f64 field.
Why two-pass (NOT time-bucket sharding)
=======================================
Time-bucket sharding (the SP20 OFI pattern in
`compute_ofi_per_bar_parallel`) bit-equates to sequential because OFI
features derive from BOUNDED rolling windows (VPIN ≤50, Kyle ≤100,
trade-imb ≤100, OFI-stats ≤300). After ≥window-size warmup updates,
two rolling buffers started from different initial states converge
bit-identically.
`ImbalanceBarSampler` is fundamentally different: its `cumulative_imbalance`
is a path integral that resets only when crossing ±threshold. There is
NO bounded-lookback window. Two replays of the same trade tape starting
from different cum_imb offsets emit at DIFFERENT trade indices, and the
phase offset is NOT guaranteed to vanish at any future trade — even
after many emissions, a bounded phase difference can persist
indefinitely. An earlier time-bucket-sharded prototype with WARMUP=10000
trades passed K=4/K=8 at threshold=20 (dense emissions, ~30 emissions
per shard's warmup window) but FAILED at threshold=500 with a 1-bar
count drift (golden=303, parallel=304) — exactly the kind of phase-
offset divergence predicted by the path-integral analysis.
Two-pass decomposition
======================
Pass 1 (sequential, lightweight): walk the trade tape ONCE tracking only
cum_imb, prev_price, last_direction. Record the trade index of every
emission-triggering trade as the end of a segment. No OHLCV state, no
Vec<OHLCVBar> allocation, no per-trade conditional bar construction.
O(N) simple arithmetic — for 50k-2M trades it runs in 1-15ms.
Pass 2 (parallel rayon): each emission segment [boundary[i-1]+1,
boundary[i]] produces exactly one bar via independent OHLCV reduction
over its trade slice. Segments are fully independent; `par_iter()` over
segment ranges is trivially correct.
Bit-equivalence guarantee
=========================
Pass 1 mirrors `ImbalanceBarSampler::update` arithmetic verbatim:
- zero-volume early-return (alternative_bars.rs:484-486)
- direction tie-break (alternative_bars.rs:495-506)
- cum_imb update (alternative_bars.rs:508-510)
- emission threshold check (alternative_bars.rs:522-523)
- prev_price/last_direction kept across emissions (reset() at :558-566)
- NO trailing-partial-bar flush (matches sequential exactly)
Pass 2's per-segment OHLCV reduction matches the sampler's per-bar OHLCV
update verbatim: open = first non-zero-volume trade in segment, close =
last non-zero-volume trade, high/low = max/min over non-zero-volume
trades, volume = sum, timestamp = open's timestamp.
Performance (release-mode bench, 16-thread rayon pool)
======================================================
n=100k: seq=1.39ms, par=1.88ms, speedup=0.74x (rayon overhead dominates)
n=500k: seq=8.04ms, par=3.35ms, speedup=2.40x
n=2M: seq=27.59ms, par=12.39ms, speedup=2.23x
Speedup is bounded by Pass 1 (sequential, ~5-7ms at 2M trades) since
Pass 2 (parallel, ~2-3ms at 2M trades on 16 threads) is ~3x faster
than Pass 1's sequential floor. At production scale (50k-500k trades
per `mbp10_to_imbalance_bars` call) we get 2-2.4x speedup over the
all-sequential baseline.
The `min_bars_per_task` cutoff falls back to a sequential Pass 2 when
segment count < 256 (the rayon spawn overhead exceeds the parallel
benefit). Tunable via the second function arg.
Files changed
=============
- crates/ml-features/src/alternative_bars.rs (+1 -1):
derive `Clone` on `ImbalanceBarSampler` for parallel-shard use
(carried through this commit even though Pass 1's hand-rolled walk
in `mbp10_loader.rs` doesn't need it — keeps the type clonable for
future test/bench infrastructure).
- crates/ml-features/src/lib.rs (+3 -2):
re-export `imbalance_bars_parallel`, `imbalance_bars_sequential`,
`DEFAULT_IMBALANCE_BAR_MIN_BARS_PER_TASK`.
- crates/ml-features/src/mbp10_loader.rs (+253 -15):
new `imbalance_bars_parallel`, `imbalance_bars_sequential`,
`DEFAULT_IMBALANCE_BAR_MIN_BARS_PER_TASK` const. Replace the inline
sampler-walk loop in `mbp10_to_imbalance_bars` with a call into
`imbalance_bars_parallel`. Module note explains why time-bucket
sharding doesn't apply.
- crates/ml-features/tests/imbalance_bars_parallel_bit_equiv_test.rs
(+272 NEW): hermetic synthetic-trade bit-equivalence tests at:
* default cutoff (par_iter Pass 2)
* forced par_iter (min_bars_per_task=1)
* forced sequential Pass 2 (min_bars_per_task=usize::MAX)
* empty input
* small input (Pass 2 below par cutoff)
* high-threshold sparse emissions (the case that broke the
time-bucket sharding prototype). All pass exact 0.0 diff.
Also note: 293/293 ml-features lib tests pass (no regression).
Pairs with the file-level par_iter trade extraction in
8f5c64e10 (Bottleneck A). Together they parallelise both halves of
`mbp10_to_imbalance_bars`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bottleneck A of two: the loop in `mbp10_to_imbalance_bars` that calls
`extract_trades_from_dbn_file` + `filter_front_month_mbp10` per file
ran serially across the 9 quarterly DBN files. Each file is independent
(different contract universe per quarter), the front-month filter is
purely intra-file, and the final `all_trades.sort_by(|a, b| timestamp)`
re-sequences across files — so reduce order across files is irrelevant
to correctness.
Switched the loop to `dbn_files.par_iter().filter_map(...).collect()`,
mirroring the trades_loader-side pattern in
`precompute_features.rs:551-565`. Per-file logging (raw count → filtered
front-month count) preserved verbatim. On the `ci-compile-cpu` 28-core
node decoding 9 .dbn.zst files, the file-decoder/zstd-decompress phase
should drop from ~9× single-file latency to ~1× — bounded by the slowest
single file.
This is the trivial half of the parallelisation. Bottleneck B (the
sequential `ImbalanceBarSampler` pass over the concatenated trade list)
follows in the next commit with time-bucket sharding + warmup overlap +
bit-equivalence test, mirroring the SP20 OFI pattern at
`crates/ml-features/src/ofi_calculator.rs::compute_ofi_per_bar_parallel`.
Build: `cargo check -p ml-features` clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the sequential per-bar OFI loop in
`crates/ml/examples/precompute_features.rs:578-734` with a call into a
new `ml-features` library function that shards bars across CPU cores
with warmup overlap. On `ci-compile-cpu` (28 cores), large fxcache
runs (~50-200k bars) get a meaningful speedup; small datasets (<10k
bars) may run slightly slower due to amortisation cost — see test
output below.
Strategy: time-bucket sharding with warmup overlap
- Partition core bars [0, n) into K contiguous shards (K = rayon
current threads, or `OFI_SHARD_COUNT` env override).
- For each shard [start_k, end_k):
1. Construct a fresh `OFICalculator`.
2. Replay the `WARMUP_BARS = 5000` preceding bars (or fewer if
start_k < 5000) by feeding trades + calling
`calculate(last_snap)` and DISCARDING outputs. This populates
VPIN (50 buckets × 50k contracts), Kyle (100 trades),
trade-imbalance (100 trades), OFI-stats (300 snapshots), and
prev_snapshot to bit-identical sequential-walk state.
3. Process core bars and emit ofi_per_bar rows.
- Concatenate shard outputs in order, then run post-loop passes
(lag-1 deltas, log_bar_duration) over the full Vec — both are pure
functions of the concatenated output / bar timestamps and have no
shard interaction.
Bit-equivalence guarantee
=========================
Each shard's warmup phase replays the same prefix of trades+snapshots
into a freshly initialised local `OFICalculator`. Once the warmup has
covered enough bars to fully populate every rolling window AND any
post-warmup-window state has been carried forward, the shard's
calculator state at the warmup→core boundary is bit-identical to the
sequential walk. `MicrostructureState` is constructed fresh per-bar
(no cross-bar state) so its semantics are unchanged.
Verified by `crates/ml-features/tests/ofi_parallel_bit_equiv_test.rs`:
| test | result | max abs diff |
|-------------------------------------------------|--------|--------------|
| parallel_matches_sequential_within_1e9_k4 | pass | ≤ 1e-9 |
| parallel_matches_sequential_within_1e9_k8 | pass | ≤ 1e-9 |
| parallel_k1_is_sequential | pass | exactly 0 |
| parallel_handles_no_trades | pass | ≤ 1e-9 |
| parallel_speedup_smoke (ignored, bench-shaped) | n/a | n/a |
Speedup smoke (release, K=8, synthetic data):
n=8000 bars : seq=16.36ms par=18.98ms speedup=0.86x
n=30000 bars: seq=50.89ms par=29.56ms speedup=1.72x
n=80000 bars: seq=142.12ms par=55.13ms speedup=2.58x
Speedup grows with N because the 5000-bar warmup overhead
amortises. At production scale (50-200k bars × ~3 snap/bar × ~5
trades/bar) real workloads will see closer to K-1.x speedup. Small
datasets (<10k) regress slightly — by design; warmup must be ≥
rolling-window depth for bit-equivalence and we will not relax that
for marginal wall-clock gains on tiny inputs.
Diff
====
- `crates/ml-features/src/ofi_calculator.rs` (+356 LOC):
`compute_ofi_per_bar_sequential` (reference impl),
`compute_ofi_per_bar_parallel` (rayon par_iter over shard ranges),
`process_one_bar` (shared per-bar body — single source of truth
for the loop semantics, no duplication between paths),
`apply_post_loop_passes`, `PerBarOfiInputs` struct,
`PER_BAR_OFI_DIM=32` const, `DEFAULT_OFI_PARALLEL_WARMUP_BARS=5000`
const.
- `crates/ml-features/src/lib.rs` (+4 LOC): re-export new public API.
- `crates/ml/examples/precompute_features.rs` (-132 +35 LOC): replace
the inline 132-line OFI loop with a call to
`compute_ofi_per_bar_parallel`. Reads `OFI_SHARD_COUNT` env or
falls back to `rayon::current_num_threads()`.
- `crates/ml-features/tests/ofi_parallel_bit_equiv_test.rs` (+~280 LOC,
new): synthetic-data bit-equivalence tests at K=4, K=8, K=1, and
no-trades.
Memory budget per shard: one cloned `OFICalculator` (≤10 MB carrying
the rolling-window state). K=8 ≈ 80 MB extra. Manageable on the 28-core
CPU node.
No `mbp10_to_imbalance_bars` / `filter_front_month_mbp10` changes
(out of scope; those were just fixed and a separate smoke is running).
No audit-doc update required: changes are confined to ml-features +
ml/examples and do not touch crates/ml/src/(cuda_pipeline|trainers/dqn)/
which is the trigger scope for the Invariant-7 audit-doc check.
Pre-req for sp20 parallel OFI extraction (next commit). Time-bucket
sharding with warmup overlap requires each shard to own a deep-cloned
calculator so the warmup walk replays trades+snapshots into shard-local
rolling-window state without contention.
State that needs Clone:
- OFICalculator (top-level; Option<Mbp10Snapshot> + 5 sub-state fields)
- VPINCalculator (VecDeque<f64> signed-volume buckets, max 50)
- KyleLambdaCalculator (2x VecDeque<f64>, max 100 each)
- TradeImbalanceTracker (4 primitives)
- OFIStats (VecDeque<f64>, max 300)
All field types already implement Clone (Mbp10Snapshot derives Clone in
data/providers/databento/mbp10.rs:72; VecDeque<f64> + primitives are
trivially Clone). Pure-derive change, zero behavioral diff. Verified via
SQLX_OFFLINE=true cargo check -p ml-features (clean build, 42s).
No audit-doc update required: changes are confined to ml-features and
do not touch crates/ml/src/(cuda_pipeline|trainers/dqn)/ which is the
trigger scope for the Invariant-7 audit-doc check.
At threshold=0.5 the sampler produced 209M bars from 209M trade ticks (1:1
ratio, essentially per-tick) on workflow f5wnd today. Feature extraction hit
54Gi/56Gi memory limit — near OOM. The default was wrong: with EWMA bypassed
(per 1aaf94306), threshold=0.5 contracts of imbalance is below the natural
ES.FUT trade size, so every trade fires a bar.
threshold=20 produces ~5-6M bars matching the volume-bar density baseline
(5.74M bars at 100 contracts). Math: 209M / 5.74M = 36×, so threshold needs
0.5 × 36 ≈ 18 → round to 20.
Three files updated atomically per feedback_no_partial_refactor:
- config/training/dqn-production.toml: 2.5 → 20.0 (wgdc8 left it at 2.5)
- infra/k8s/argo/train-template.yaml: workflow default 0.5 → 20.0
- infra/k8s/argo/train-multi-seed-template.yaml: workflow default 0.5 → 20.0
The 1:1 bar:tick observation alongside the price-continuity proof (22 jumps
out of 209M = ~1e-7 — front-month filter works) confirms the front-month
fix on this branch (6c1ab8850 + 3b5f17913) is correct. Threshold was the
remaining mistuning.
Cache-key implication: changing the threshold invalidates all existing
fxcache files (per the cache-key architectural fix). Next dispatch on this
branch will regenerate fxcache from scratch.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Removes the `instrument_id: 0` backward-compat default introduced in 6c1ab8850.
That default was nonsensical: a sentinel value for an identifier creates the
exact half-state that future filtering code can't distinguish from real data.
Now extract_trades_from_snapshots takes `instrument_id: u32` as an explicit
required parameter. Caller must provide the contract id the snapshot stream
belongs to (typically captured at the data-acquisition site alongside symbol).
Tests:
- test_extract_trades_from_snapshots updated to pass id=12345 + asserts every
emitted trade carries that id (verifies the explicit-param contract).
- New test_filter_front_month_mbp10_picks_highest_volume covers the helper
added in 6c1ab8850 (no test before).
- New test_filter_front_month_mbp10_empty for the empty-input edge case.
15/15 mbp10 tests pass. Workspace + examples compile clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pre-fix: mbp10_to_imbalance_bars loaded ALL trades from MBP-10 .dbn files
without filtering by instrument_id, then fed them through ImbalanceBarSampler.
When the dominant ES contract rolls over (ESZ24 → ESH25 → ESM25), price
jumps appear at every rollover, failing the precompute_features.rs:371-376
price-continuity gate (3.5% jump rate vs 1% threshold). First observed on
workflow train-multi-seed-crb2k yesterday.
Fix: mirror the per-file front-month pattern from precompute_features.rs:347-365
(which uses filter_front_month on DbnTrade slices). Same logic applied to
Mbp10Trade — but Mbp10Trade didn't carry instrument_id, so:
- Added `instrument_id: u32` field to Mbp10Trade.
- Populated from `mbp10.hd.instrument_id` in extract_trades_from_dbn_file
(the production extraction path).
- Defaulted to 0 in extract_trades_from_snapshots (snapshot-diff path used by
tests; no per-record id available; backward-compatible — filter is a no-op
when all ids are 0).
- New filter_front_month_mbp10() helper that mirrors trades_loader's
filter_front_month exactly, just operating on Mbp10Trade.
- mbp10_to_imbalance_bars now applies filter_front_month_mbp10 per-file
in the extraction loop, identical to precompute_features.rs:347-365.
Compiles clean. Replaces tracing::debug! with tracing::info! at the per-file
log site to make the raw-vs-filtered count visible at production INFO level.
The wgdc8 commit 1aaf94306 (EWMA bypass via ImbalanceBarSampler::new) is
preserved — fixed-threshold semantics remain correct per
pearl_imbalance_bar_ewma_washes_out_configured_threshold. This fix is
orthogonal: front-month filter at the trade-tape level, EWMA bypass at the
sampler level. Both are needed for imbalance bars to produce sensible output.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
