# Loader Instrument Filter Fix — Restore Clean Single-Instrument Baseline

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development.

**Goal:** Add an `instrument_id` filter to the MBP-10 decoder so the loader processes only front-month ES (instrument_id=17077, ~74% of records) instead of all 14 instruments in the file. This is upstream of EVERY training failure we've diagnosed this session — fixing it must come before any further architectural iteration.

**Architecture:** Decoder-level filter (3-4 files), no model changes. Re-train baseline. Compare new auc_h1000 to prior contaminated 0.7137 to quantify the contamination impact.

## Discovery summary

C1 NumPy analysis on `test_data/futures-baseline-mbp10/ES.FUT/ES.FUT_2024-Q1.dbn.zst` revealed:
- 14 distinct `instrument_id` values in the file
- 74.18% records have id=17077 (front-month ES, likely ESH4 for Q1 2024)
- 23.42% records have id=5602 (back-month ES, likely ESM4)
- ~3% records from 12 other instruments (calendar spreads, micro-ES, etc.)
- Loader reads ALL records sequentially without filtering
- K-windows spanning across instruments produce ΔP = $5000+ → spurious y_prof=1 labels
- This explains: pos_fraction inflation (20% → 35%), aux supervision failures, possibly the BCE plateau

## File scope (3 files modified, 0 new)

| File | Action |
|---|---|
| `crates/data/src/providers/databento/mbp10.rs` | Add `pub instrument_id: u32` to `Mbp10Snapshot`; populate in `from_*` constructors |
| `crates/ml-alpha/src/data/loader.rs` | Add `instrument_id_filter: Option<u32>` field to `MultiHorizonLoaderConfig`; skip records where snapshot.instrument_id != filter (when Some) |
| `crates/ml-alpha/examples/alpha_train.rs` | Add CLI flag `--instrument-id`; default to None (legacy behavior); document recommendation to use 17077 for ES |
| `crates/ml-backtesting/src/harness.rs` | Pass `instrument_id_filter` through MultiHorizonLoaderConfig (if applicable to inference path) |

## Sub-decisions

| # | Decision | Default | Rationale |
|---|---|---|---|
| 1 | Default filter behavior | `None` (no filter, legacy) | Conservative; users opt-in to filtering. Avoids breaking existing checkpoints' implicit assumption. |
| 2 | Filter value for ES | 17077 | Empirically the front-month ES in 2024-Q1. Documented in CLI flag help text. |
| 3 | Where to filter | At decoder level (skip the snapshot during deserialization) | Avoids polluting downstream code; loader sees only filtered records. |
| 4 | Reporting | Log the number of records kept vs skipped per file at load time | Diagnostic; confirms filter is firing. |

## Task breakdown

### Task L1 — Add `instrument_id` to Mbp10Snapshot

**Files:**
- Modify: `crates/data/src/providers/databento/mbp10.rs`

- [ ] **Step 1: Read the file**, understand `Mbp10Snapshot` struct and its construction sites.
- [ ] **Step 2: Add field** `pub instrument_id: u32` to the struct.
- [ ] **Step 3: Populate** in all decoder constructors. Databento MBP-10 records have `instrument_id: u32` field directly.
- [ ] **Step 4: Verify** existing tests still pass with the new field.

### Task L2 — Filter at decoder/loader level

**Files:**
- Modify: `crates/ml-alpha/src/data/loader.rs`

- [ ] **Step 1: Add** `instrument_id_filter: Option<u32>` to `MultiHorizonLoaderConfig`.
- [ ] **Step 2: Filter** records in `LoadedFile::open` (or equivalent decode step):
  ```rust
  let snapshots: Vec<Mbp10Snapshot> = raw
      .into_iter()
      .filter(|s| cfg.instrument_id_filter.is_none()
                  || cfg.instrument_id_filter == Some(s.instrument_id))
      .collect();
  ```
- [ ] **Step 3: Log** filter statistics:
  ```rust
  let kept = snapshots.len();
  let skipped = raw_count - kept;
  tracing::info!(kept, skipped, "instrument filter applied");
  ```
- [ ] **Step 4: Verify** existing in-file tests pass when filter=None (legacy behavior unchanged).

### Task L3 — CLI flag exposure

**Files:**
- Modify: `crates/ml-alpha/examples/alpha_train.rs`

- [ ] **Step 1: Add** `--instrument-id <u32>` CLI flag (default: None).
- [ ] **Step 2: Pass through** to MultiHorizonLoaderConfig.
- [ ] **Step 3: Document** in flag help text: "Filter MBP-10 records to a single instrument. For ES.FUT, use 17077 (front-month). Without this flag, ALL instruments in the file are processed sequentially — multi-instrument deltas inflate labels."

### Task L4 — Backtesting harness

**Files:**
- Modify: `crates/ml-backtesting/src/harness.rs`

- [ ] **Step 1: Check** if the harness uses MultiHorizonLoaderConfig (it does per CB2).
- [ ] **Step 2: Add** the field to the inference-only construction site (default None for backward compat).

### Task L5 — Smoke validation

- [ ] **Step 1: Local synthetic** — perception_overfit still passes (filter is None for synthetic).
- [ ] **Step 2: Commit + push**.
- [ ] **Step 3: Dispatch Argo Smoke 1** with `--instrument-id 17077`.

```bash
./scripts/argo-alpha-perception.sh --instrument-id 17077  # if the script forwards the flag
# OR submit via argo with workflow parameter override
```

**Validation gates**:
- auc_h1000 ≥ 0.65 (baseline gate)
- auc_h1000 IDEALLY > 0.7137 (improvement over contaminated baseline)
- Aux metrics: pos_fraction should DROP from 0.35-0.46 to closer to local's 0.20

If auc_h1000 INCREASES, contamination was real and significant.
If auc_h1000 stays the same, the cross-instrument contamination didn't affect the trunk's per-horizon ranking (just the aux labels).
If auc_h1000 DROPS, something else is going on.

### Task L6 — Document findings + pearls

- [ ] **Step 1: Write pearl** `pearl_multi_instrument_contamination_in_databento_files` — DBN files for "ES.FUT" symbol can contain calendar spreads + related futures; loaders MUST filter by instrument_id.
- [ ] **Step 2: Update MEMORY.md** index.

## Validation

- `cargo check --workspace --all-targets` clean
- 43 lib tests pass
- perception_overfit 4-5 pass (synthetic uses None filter)
- Argo Smoke 1 with `--instrument-id 17077` returns
- auc_h1000 documented vs prior 0.7137 baseline

## After this fix

The session's prior diagnostics need re-validation against the cleaner baseline:
- **F1 dir_acc fix** — should still help (metric design issue, orthogonal to data quality)
- **D1 signed-conviction EMA** — should still help (policy-side fix, no data dependency)
- **F2 mid_price NaN** — already reverted (wrong hypothesis)
- **Aux supervision (CB1-CB5)** — re-evaluate after clean baseline; the labels were contaminated so all aux iterations were doomed
- **Layer C regime gate** — re-evaluate; spread regime might mean something different once instruments don't mix

## Strategic upside (deferred to follow-up plan)

After clean single-instrument baseline:
- Multi-instrument training as FEATURE: include front+back-month spread as input
- Per-instrument trunks with shared encoder (the architecture B3+B4 already built supports this — aux trunk could become "back-month trunk")
- Calendar spread prediction as a separate output head
- Cross-asset signals from the 3% miscellaneous instruments

This is a significant architectural opportunity but requires the clean baseline first.

## Estimated cost

- Tasks L1-L5: ~5 hours total
- L1 (decoder field): 1h
- L2 (loader filter): 1h
- L3 (CLI): 0.5h
- L4 (harness): 0.5h
- L5 (smoke): 1.5h + 12 min wallclock
- L6 (pearl): 0.5h
