# Deeper Diagnosis: Aux Supervision Mysteries Before Next Pivot

> **For agentic workers:** Diagnostic plan, not implementation. Read each step's pass/fail criteria; branch decisions are explicit.

**Goal:** Resolve four unexplained mysteries before committing to another aux-supervision iteration OR pivoting to Layer C (regime gate). Avoid the "fix-and-pray" loop that's already burned 5 Argo smokes without convergence.

**Architecture of the diagnosis:** Cheap falsifiers first. Each step narrows the hypothesis space; later steps run only if earlier ones don't yield a verdict.

## The four mysteries

| # | Mystery | What we know | What we don't |
|---|---|---|---|
| M1 | `pos_fraction` bit-identical pre/post F2 fix (0.351→0.351) | Different seeds, different code, same value to 6 d.p. | Whether F2 actually fired at runtime |
| M2 | Why is `pos_fraction ≈ 40%` at K=10 (~700µs)? | At ES tick=0.25, "ΔP > 4 ticks in 700µs" should be rare | Whether one-sided books / data quality cause spurious positives, OR if the data really has this property |
| M3 | Epoch-4 NaN explosion (BCE 0.84→18.74, size→NaN) | First 4 epochs bit-identical to all prior runs; epoch 4 diverges only after F2 fix | Where the NaN first appears + which buffer chain propagates it |
| M4 | What's the structural ceiling? | Synthetic test passes; real-data fails consistently; ES heavily arbitraged | Whether ANY aux signal exists at K∈[10,100,1000] worth pursuing |

## Pre-conditions

- Branch: `ml-alpha-phase-a` HEAD `85d3ca5c7` (F1+F2 applied; epoch-4 NaN observed in `alpha-perception-7shgw`)
- Working tree clean
- Local validation: 43 lib tests + 15 multi_horizon_labels pass
- 6 prior aux-supervision smokes available for cross-comparison

---

## Step 1 — Resolve M1 (did F2 actually fire?) [10 min, no compute]

**Question:** Is `mid_price_f32 → NaN` actually being executed on real data, or is the code path bypassed?

### 1.1 Check binary actually rebuilt

```bash
git log -1 --format=%H crates/ml-alpha/src/data/loader.rs
# Expected: 85d3ca5c7 (the F2 commit)

cd /tmp && mc ls fh/argo-logs/alpha-perception-7shgw/ 2>&1 | head -3
# Confirm the workflow at commit 85d3ca5c7 actually ran (build timestamp matches)
```

### 1.2 Inspect the actual computed code path

Read `crates/ml-alpha/src/data/loader.rs::mid_price_f32` — confirm the NaN branch landed in the current commit. Read `Mbp10Snapshot.levels[0]` source — verify it's the right level. Read where `mid_price_f32` is called — verify the result feeds `generate_outcome_labels_ab`.

**PASS:** Code path verified, F2 fix is in the binary.
**FAIL:** F2 fix isn't in the binary (revert / re-push and skip to Step 4).

### 1.3 The pos_fraction bit-identity test

Run a Rust unit test that constructs a 1000-sample fixture with N% one-sided books and computes pos_fraction with both old (blind avg) and new (NaN-guarded) mid_price_f32. If the resulting pos_fractions are within 0.01 of each other, **one-sided books are NOT the contamination source** — the loader's input data has near-zero one-sided rate, and F2's hypothesis was wrong.

```rust
#[test]
fn f2_pos_fraction_sensitivity_to_one_sided_books() {
    // Construct synthetic Mbp10Snapshot stream with X% one-sided books.
    // Compute pos_fraction with old vs new mid_price_f32.
    // Assert: |new - old| < 0.01 when X < 0.5%, gap visible when X > 5%.
}
```

**Decision branch:**
- If gap visible at X=5% but not at X=0.5%: F2 fix WORKS, real data must have <0.5% one-sided. Skip to Step 2.
- If no gap at X=10%: F2 fix doesn't work as designed. Inspect why (cascade issue).

---

## Step 2 — Resolve M2 (is 40% pos_fraction real?) [30 min, NumPy on local data]

**Question:** Does ES real data actually have ~40% of K=10 windows with ΔP > 4 ticks, or is this an artifact of decoder/contamination?

### 2.1 Compute pos_fraction LOCALLY using the Rust loader logic

Write a self-contained Python script reusing the existing decoded local fixture (`test_data/futures-baseline-mbp10/ES.FUT/ES.FUT_2024-Q1.dbn.zst` via databento):

```python
# Decode K snapshots from the local Q1 file
# Compute mid = (bid_px[0] + ask_px[0]) / 2 (raw, no guards)
# For each K in [10, 100, 1000]:
#   Compute delta_K = mid[t+K] - mid[t]
#   y_prof_long = (delta_K > 1.0).mean()  # 2 ticks cost = 1.0 price unit
#   y_prof_short = (delta_K < -1.0).mean()
# Print pos_fraction per (direction, K)
```

This mirrors what the loader does. Expected: if matches cluster's 0.351/0.389/0.456 → data really has this property. If different → cluster data path differs from local.

### 2.2 Histogram of |ΔP| at K=10 on local data

Plot or print quantile statistics of `|mid[t+10] - mid[t]|`. If p50 > 1.0, most K=10 windows naturally exceed the cost threshold — the 40% pos_fraction is correct. If p50 << 1.0, something's wrong.

**Decision branch:**
- If matches cluster + p50 > 1.0: 40% pos_fraction is REAL ES microstructure. F2's contamination hypothesis was wrong. Don't fix what isn't broken; focus on F1.
- If matches cluster + p50 << 1.0: there's a SHARED bug in mid-price computation that's NOT one-sided books. Need to find it.
- If doesn't match cluster: cluster data is contaminated differently (decoder issue, normalized prices, etc.) — investigate cluster decode path.

---

## Step 3 — Resolve M3 (epoch-4 NaN explosion) [45 min, log analysis + code trace]

**Question:** What buffer first showed NaN at epoch 4?

### 3.1 Cross-compare epoch-by-epoch metrics across runs

| Run | HEAD | Epoch 0 BCE | Epoch 4 BCE | Notes |
|---|---|---|---|---|
| dz6r4 | 0f5d5c7b4 (F1+F2 pre) | 0.845 | 0.806 | Pre F1+F2 |
| 779fp | 24289f40a (pw=10) | 0.845 | 0.806 | Same as above |
| 7shgw | 85d3ca5c7 (F1+F2 fix) | 0.845 | **18.74** | NaN at epoch 4 |

**Observation:** Epochs 0-3 BCE values ARE bit-identical across all three runs. The NaN appears ONLY at epoch 4 of the F1+F2 run.

This is highly suspicious: identical labels and gradients for 3 epochs, then suddenly diverge. Possible causes:
- **Hidden statefulness**: σ_K Welford EMA accumulates NaN over epochs without resetting between runs
- **Random seed exhaustion**: validation data shifts to a new region at epoch 4 with more zero-priced books
- **Numerical accumulation**: head weights drift to a regime where one bad batch tips them to NaN

### 3.2 Add NaN-detection tracing (write code, but only diagnostic, not in critical path)

```rust
// At end of each train_step in step_batched, after BCE/Huber loss readback:
if !mean_prof_bce.is_finite() || !mean_size_huber.is_finite() {
    tracing::warn!(
        step = self.train_step_count,
        epoch_step = epoch_step,
        prof_long_loss = total_loss_pl,
        prof_short_loss = total_loss_ps,
        size_long_loss = total_loss_sl,
        size_short_loss = total_loss_ss,
        n_pl = n_pl, n_ps = n_ps, n_sl = n_sl, n_ss = n_ss,
        "AUX NAN DETECTED — first occurrence"
    );
}
```

Re-run the smoke. The first NaN warning will tell us EXACTLY at which step + which loss buffer it appeared. Then we trace BACKWARD: what input made that buffer NaN?

### 3.3 If NaN appears in size_huber first (likely): trace σ_K cascade

The σ_K computation in `generate_outcome_labels_ab` uses a rolling Welford. If a NaN enters the window, it propagates. The current code:

```rust
// Look at the log_ret computation: log(p[t+K] / p[t])
// If p[t] == 0 OR p[t+K] == 0 → log_ret = -inf or NaN
// Welford running sum gets NaN → all subsequent σ_K[h][t'] = NaN
```

Check `crates/ml-alpha/src/multi_horizon_labels.rs::generate_outcome_labels_ab` Pass 1 (σ_K computation). Find where log_ret is computed and the Welford update. Confirm there's no NaN-skip.

**Decision branch:**
- If σ_K Welford propagates NaN: fix σ_K to NaN-skip (the proper F2 strengthening).
- If σ_K is fine but size head still NaNs: investigate `aux_huber_masked` kernel for divide-by-zero or overflow.

---

## Step 4 — Resolve M4 (structural ceiling test) [conditional on Steps 1-3]

**Run only if Steps 1-3 conclude "F1+F2 should work properly" but we want to bound the upside.**

### 4.1 Cheapest structural ceiling test: label-swap experiment

Per F4's recommendation: swap `y_prof_long ↔ y_prof_short` at the loader emission and run a 3-epoch smoke. If dir_acc inverts to >0.5, the model has learnable signal. If still at chance, the aux signal at this trunk size is structurally absent.

```rust
// One-line gated swap in alpha_train.rs
if std::env::var("AUX_LABEL_SWAP_DIAGNOSTIC").is_ok() {
    std::mem::swap(&mut prof_long_row, &mut prof_short_row);
    std::mem::swap(&mut size_long_row, &mut size_short_row);
}
```

3-epoch Argo smoke takes ~7 minutes. The verdict is decisive:
- dir_acc > 0.55 → real signal exists, find the right metric / loss
- dir_acc near 0.5 → no signal at this trunk size

### 4.2 Pivot decision

Based on M1-M4 verdicts:

| M1 | M2 | M3 | M4 | Next |
|---|---|---|---|---|
| Pass | 40% real | σ_K NaN | dir_acc > 0.55 | Fix σ_K, keep aux supervision |
| Pass | 40% real | σ_K NaN | dir_acc ~0.5 | Aux has no signal at this size; pivot to Layer C |
| Pass | 40% NOT real | * | * | Cluster data bug; fix decoder first |
| Fail | * | * | * | F2 fix wasn't applied; either re-apply or revert |

---

## Tasks (executable order)

- [ ] **Task 1** — Step 1.1 + 1.2: verify F2 fix actually landed in binary. Read code. (5 min)
- [ ] **Task 2** — Step 1.3: write Rust unit test for pos_fraction sensitivity to one-sided book rate. (15 min)
- [ ] **Task 3** — Step 2.1: write Python script to recompute pos_fraction on LOCAL test_data Q1 file, compare to cluster's bit-identical value. (20 min)
- [ ] **Task 4** — Step 2.2: histogram of |ΔP| at K=10 on local data. (10 min)
- [ ] **Task 5** — Step 3.2: add NaN-detection tracing to step_batched, re-dispatch smoke (only if Tasks 1-4 don't already explain everything). (20 min code + 12 min Argo)
- [ ] **Task 6** — Step 3.3: inspect σ_K Welford for NaN-propagation IF Task 5 confirms σ_K is the cascade point. (15 min)
- [ ] **Task 7** — Decision: based on Tasks 1-6 outcomes, choose:
  - (a) Strengthen F2 with σ_K NaN-skip → re-run smoke
  - (b) Revert F2, keep F1 only → re-run smoke
  - (c) Run label-swap diagnostic (Step 4.1) → branch on dir_acc
  - (d) Pivot to Layer C
- [ ] **Task 8** — Pearls: capture findings regardless of outcome (don't lose the diagnostic work).

## Estimated cost

- Steps 1-2: 30 min (no compute beyond local Python/cargo)
- Step 3 (NaN tracing): 30-45 min if needed, 0 if M1/M2 closes the question
- Step 4 (label-swap): 45 min if needed

**Total floor: 30 min if Steps 1-2 give us a clear answer.**
**Total ceiling: 2 hours if we go through all steps.**

## Memory rules applicable

- `feedback_no_quickfixes` — diagnose root cause before fixing
- `pearl_no_deferrals_for_complementary_fixes` — combine related fixes when scope overlaps
- `pearl_canary_input_freshness_launch_order` — staging vs kernel ordering bugs
- `pearl_trade_level_vol_for_stop_distance` — σ_K should be floored at cost-tick scale (we did this; verify it's not being bypassed)
- `pearl_first_observation_bootstrap` — sentinel handling for rolling stats

## Risks

| Risk | Mitigation |
|---|---|
| Step 1 reveals F2 fix didn't apply at runtime — wasted earlier compute | Cheap to detect (Task 1), and the learning is real |
| Step 2 shows local data ≠ cluster data — broader decoder investigation needed | Document the divergence; cluster-side investigation gets its own plan |
| Step 5 NaN tracing modifies hot path | Tracing is conditional on `!is_finite()` — negligible perf cost |
| Label-swap diagnostic doesn't decisively answer (dir_acc oscillates near 0.5) | Run for 5 epochs instead of 3; if still ambiguous, the answer IS "ambiguous" — pivot to Layer C anyway |

## Falsification gates (when to STOP)

- After Step 2: if 40% pos_fraction confirmed REAL on local data, stop digging F2 — it was a wrong hypothesis. Focus on F1 + σ_K cascade.
- After Step 3: if NaN cascade source identified, the fix is direct.
- After Step 4: if label-swap gives dir_acc still ~0.5, aux signal at this trunk size is bounded by architectural ceiling. Pivot to Layer C.

## Out-of-scope (DO NOT do here)

- Implement CB6 (per-horizon Huber kernel split) — defer until aux supervision is validated as worth pursuing
- Build Layer C regime gate — that's the post-diagnosis pivot, not the diagnosis itself
- Modify aux trunk dimensions — that's an upsize architectural change, not a diagnosis step

## After this plan

Either:
- **Aux supervision validated**: continue to B7 (decision policy aux gate) with confidence
- **Aux supervision falsified**: pivot to Layer C (regime gate) per the original anti-calibration plan; aux infrastructure stays in code as future-work scaffolding

Plan saved at `docs/superpowers/plans/2026-05-22-aux-diagnosis-deeper.md`.
