---
name: smoke-pilot
description: Monitors a running argo workflow smoke; auto-kills on first useful anomaly signal (NaN loss, magnitude collapse, EVAL_DIST anomalies, return-explosion, controller saturation) per stop-on-anomaly + kill-quickly discipline. Pairs with argo-deploy-helper. Streams logs via argo logs -f, never via Bash run_in_background Monitor.
---

# Foxhunt Smoke Pilot

Watches an in-flight smoke training and applies the kill-fast-on-anomaly rule.

## When to invoke

- "Monitor smoke for workflow <wf-name>".
- After invoking `argo-deploy-helper`.

## Memory you MUST read first

In `~/.claude/projects/-home-jgrusewski-Work-foxhunt/memory/`:

- `feedback_stop_on_anomaly.md`
- `feedback_kill_runs_on_anomaly_quickly.md`
- `feedback_no_redundant_monitor.md`
- `project_metric_pipeline_inflation_audit.md` (honest meters; what *isn't* an anomaly)
- All `project_*.md` files documenting prior anomalies (these are signal libraries).

## What this skill does

1. Streams workflow logs:
   ```bash
   argo logs -f <workflow-name>
   ```
   Do NOT layer Bash `run_in_background` Monitor on this stream (`feedback_no_redundant_monitor.md`).

2. Watches each log line for known anomaly patterns:

   | Anomaly | Signal | Kill window |
   |---|---|---|
   | NaN loss | `loss = NaN` or `loss = inf` | immediate |
   | Magnitude collapse | EVAL_DIST Quarter pinned at 1.0 with no Kelly cap context | within 200 steps |
   | Return explosion | `Return = 1e29%` or similar (per project_metric_pipeline_inflation_audit) | immediate |
   | Controller saturation | `α = 1.0` or `α = 0.0` for >100 steps | within 500 steps |
   | Aux CE plateau at ln(K) | `aux_ce ≈ <ln(K)>` for >100 steps | warn but don't kill (per pearl_separate_aux_trunk_when_shared_starves — fix is structural) |
   | Reward distribution corruption | one-sided clamp regression | within 100 steps |

3. On detected anomaly:
   ```bash
   argo terminate <workflow-name>
   ```
   Then summarize the anomaly with the originating memory file cited.

## Discipline

- **Always cite the originating memory file** when killing — "Killed at step 230: NaN loss, per `feedback_stop_on_anomaly.md`".
- **Never kill on a noisy single-step blip**. Use a 3-of-5 confirmation window for non-immediate anomalies.
- **Never silently survive** an anomaly that warrants a kill. The whole point of this skill is `feedback_kill_runs_on_anomaly_quickly.md`: kill at first useful signal, diagnose, fix, re-run.
- **Inflation is not anomaly**. Per `project_metric_pipeline_inflation_audit.md`, several metrics are display caps or annualization conventions, not bugs. Read that file first; do not flag those as anomalies.

## After kill

Suggest:
1. Save the failing log slice for analysis.
2. Open a fresh investigation (likely brainstorming → spec).
3. Re-run only after the fix is committed and pushed.
