# DQN v2 Plan 5 — Validation Substrate + Final Pass Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Complete the convergence-guardrail infrastructure that makes DQN v2's policy claims reproducible: multi-seed × multi-fold validation harness (A.3), regression-detection hard-stop (A.4), nsys profile harness for perf-regression detection (A.4.1), and the final end-to-end validation pass that demonstrates Tier 1 + Tier 2 + Tier 3 exit criteria are met. Only after Plan 5 does the spec declare the DQN v2 rollout complete.

**Architecture:** Most of Plan 5 is orchestration and automation rather than model code. The multi-seed harness is a new Argo Workflow DAG that spawns N × K training jobs from one submission. Regression detection reuses HEALTH_DIAG's existing metric emission and adds a SIGTERM-self trigger on consecutive out-of-band epochs. nsys profiling is a training-binary wrapper in `train-template.yaml` + an artifact upload step. The final validation pass consumes all of these and produces a signed-off deliverable.

**Tech Stack:** Argo Workflows + Scaleway Kapsule, Rust + Python orchestration scripts, MinIO for nsys profile artifacts. New external tools: `nsys` (NVIDIA Nsight Systems). Python deps: `numpy`, `scipy.stats`, `matplotlib` (for aggregation + plots); added to `scripts/requirements.txt`.

**Authority:** `docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md` §2 (success tiers), §4.A.3, §4.A.4 (including A.4.1). All 9 invariants apply.

**Dependencies on prior plans:**
- Plans 1–4 all landed on main with their respective exit criteria met.
- `config/metric-bands.toml` extended by each prior plan; Plan 5 consolidates.

---

## Pre-plan: verify Plan 4 state

- [ ] **Step 0.1: Verify Plan 4 exit criteria satisfied**

```bash
# ISV_TOTAL_DIM must be sealed at 72.
grep -E "ISV_TOTAL_DIM\s*:\s*usize\s*=\s*72" crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs || {
    echo "Plan 4 incomplete — ISV_TOTAL_DIM not sealed at 72"; exit 1
}

# All 7 Plan 4 attention ISV slots exist.
for name in ATTN_MARKET_FOCUS_EMA_INDEX ATTN_OFI_FOCUS_EMA_INDEX \
            ATTN_TLOB_FOCUS_EMA_INDEX ATTN_MTF_FOCUS_EMA_INDEX \
            ATTN_PORTFOLIO_FOCUS_EMA_INDEX ATTN_PLAN_ISV_FOCUS_EMA_INDEX \
            ATTN_MAMBA2_FOCUS_EMA_INDEX; do
    grep -q "pub const $name" crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs || {
        echo "Plan 4 ISV slot missing: $name"; exit 1
    }
done

# Part E audit closed.
grep -E "^\| [A-Z]\..*TBD|^\| [A-Z]\..*evaluate" docs/ml-supervised-to-dqn-concept-audit.md && {
    echo "Part E audit has open TBD/evaluate rows"; exit 1
}

# Plan 4 Tier 1 retention documented.
grep -q "Tier 1.*RETAINED" docs/dqn-v2-plan-4-validation.md || {
    echo "Plan 4 Tier 1 retention not documented"; exit 1
}

echo "Plan 4 exit criteria satisfied. Plan 5 may begin."
```

- [ ] **Step 0.2: Verify clean working tree**

```bash
git status --porcelain
```

Expected: empty output. Plan 5 starts from a clean tree; stash any in-flight work before proceeding.

---

## Task 1: A.3 Multi-Seed / Multi-Fold Validation Harness

**Files:**
- Modify: `scripts/argo-train.sh` — add `--multi-seed N --folds K` flags
- Create: `infra/k8s/argo/train-multi-seed-template.yaml` — new DAG template
- Create: `scripts/gather-multi-seed-metrics.sh` — aggregates per-seed metrics into one JSON
- Create: `scripts/aggregate-norm-stats.py` — merges `norm_stats_foldN_seedM.json` files
- Test: `multi_seed_harness_spawns_correct_job_count`

### Subtask 1A: Argo DAG for multi-seed × multi-fold

- [ ] **Step 1A.1: Write failing test**

Create `scripts/tests/test_multi_seed_harness.sh`:

```bash
#!/usr/bin/env bash
set -euo pipefail

# Dry-run: --multi-seed 3 --folds 2 should emit a workflow with 6 training jobs.
output=$(./scripts/argo-train.sh --multi-seed 3 --folds 2 --dry-run 2>&1)
actual_count=$(echo "$output" | grep -c "kind: WorkflowTask")
expected=6
if [[ "$actual_count" -ne "$expected" ]]; then
    echo "FAIL: expected $expected jobs, got $actual_count"
    exit 1
fi
echo "PASS: multi-seed harness produces $expected jobs for 3×2 matrix"
```

- [ ] **Step 1A.2: Run — expect FAIL**

`--multi-seed` flag doesn't exist yet.

- [ ] **Step 1A.3: Implement `argo-train.sh` flag support**

Add arg parsing:

```bash
MULTI_SEED=1
FOLDS=1
DRY_RUN=false
TAG=""

while [[ $# -gt 0 ]]; do
    case $1 in
        --multi-seed) MULTI_SEED=$2; shift 2 ;;
        --folds)      FOLDS=$2; shift 2 ;;
        --dry-run)    DRY_RUN=true; shift ;;
        --tag)        TAG=$2; shift 2 ;;
        --commit)     COMMIT=$2; shift 2 ;;
        *) echo "Unknown flag: $1"; exit 1 ;;
    esac
done

if [[ "$MULTI_SEED" -gt 1 || "$FOLDS" -gt 1 ]]; then
    TEMPLATE=infra/k8s/argo/train-multi-seed-template.yaml
else
    TEMPLATE=infra/k8s/argo/train-template.yaml
fi
```

- [ ] **Step 1A.4: Create `train-multi-seed-template.yaml`**

Mirror `train-template.yaml` but wrap the training step in a `dag:` node with a matrix over seed × fold:

```yaml
entrypoint: multi-seed-matrix
arguments:
  parameters:
  - name: multi-seed
    value: "5"
  - name: folds
    value: "6"
templates:
- name: multi-seed-matrix
  dag:
    tasks:
    - name: train-s0-f0
      template: train-single
      arguments: { parameters: [{name: seed, value: "0"}, {name: fold, value: "0"}] }
    - name: train-s0-f1
      template: train-single
      arguments: { parameters: [{name: seed, value: "0"}, {name: fold, value: "1"}] }
    # ... generated via withItems over seed_fold_pairs
- name: train-single
  inputs:
    parameters:
    - name: seed
    - name: fold
  container:
    image: ...
    env:
    - name: SEED
      value: "{{inputs.parameters.seed}}"
    - name: FOLD
      value: "{{inputs.parameters.fold}}"
    - name: CARGO_INCREMENTAL
      value: "0"
    command: [/usr/local/bin/fxt]
    args: [train, "--seed", "$(SEED)", "--fold", "$(FOLD)"]
```

Generate the matrix programmatically in `argo-train.sh` via a short awk/python snippet — no 30-task matrix written by hand.

- [ ] **Step 1A.5: Run test — expect PASS**

- [ ] **Step 1A.6: Commit**

```bash
git add scripts/argo-train.sh \
        scripts/tests/test_multi_seed_harness.sh \
        infra/k8s/argo/train-multi-seed-template.yaml
git commit -m "plan5(task1A): multi-seed × multi-fold Argo DAG template"
```

### Subtask 1B: Metric aggregation script

- [ ] **Step 1B.1: Create `scripts/gather-multi-seed-metrics.sh`**

```bash
#!/usr/bin/env bash
set -euo pipefail

TAG=${1:?"Usage: gather-multi-seed-metrics.sh <tag>"}
OUT=${2:-/tmp/multi-seed-${TAG}.json}

# List all pods for this tag, grab their HEALTH_DIAG log tails,
# parse per-epoch metrics, aggregate by (epoch, metric) → {mean, std, median, worst}.
argo logs "@latest" --selector "foxhunt-tag=${TAG}" --no-color > /tmp/all-logs-${TAG}.txt

python3 scripts/aggregate-multi-seed-metrics.py \
    --input /tmp/all-logs-${TAG}.txt \
    --multi-seed "${MULTI_SEED:-5}" \
    --folds      "${FOLDS:-6}" \
    --output     "$OUT"

echo "Aggregated → $OUT"
```

- [ ] **Step 1B.2: Implement `aggregate-multi-seed-metrics.py`**

- Regex-parses HEALTH_DIAG lines.
- Groups by (seed, fold, epoch, metric_name).
- Computes mean, std, median, min, max per (epoch, metric) across seeds.
- Emits JSON:

```json
{
  "tag": "plan5-final",
  "multi_seed": 5,
  "folds": 6,
  "warmup_end_epoch": 15,
  "aggregates": {
    "train_sharpe": [
      {"epoch": 0, "mean": -0.2, "std": 0.1, "median": -0.2, "min": -0.4, "max": 0.0},
      ...
    ],
    "val_sharpe": [...],
    "val_trades_per_bar": [...],
    "isv_plan_params_0_ema": [...]
  }
}
```

- [ ] **Step 1B.3: `aggregate-norm-stats.py`**

Merges `norm_stats_foldN_seedM.json` across seeds per fold. Output: per-fold aggregate consumed by `evaluate step` for inference normalisation.

- [ ] **Step 1B.4: Commit**

```bash
git add scripts/gather-multi-seed-metrics.sh \
        scripts/aggregate-multi-seed-metrics.py \
        scripts/aggregate-norm-stats.py \
        scripts/requirements.txt
git commit -m "plan5(task1B): metric aggregation across multi-seed runs"
```

---

## Task 2: A.4 Regression Detection Hard-Stop

**Files:**
- Modify: `crates/ml/src/trainers/dqn/trainer/monitoring.rs` — consume metric-bands.toml, self-SIGTERM on 2N consecutive out-of-band
- Modify: `config/metric-bands.toml` — consolidate all Plan 3/4 metric bands
- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs` — handle the stop signal cleanly
- Test: `training_self_terminates_on_sustained_out_of_band_metric`

- [ ] **Step 2.1: Write failing test**

Create `crates/ml/src/trainers/dqn/smoke_tests/regression_detection.rs`:

```rust
#[test]
#[ignore = "gpu"]
fn training_self_terminates_on_sustained_out_of_band_metric() {
    let mut trainer = build_minimal_trainer();

    // Inject a broken state: train_loss saturated at 1e15 (per task #31 historic bug).
    trainer.inject_broken_metric("train_loss", 1e15);

    // Band: error_high = 1000.0. So 1e15 is ERROR-band.
    // 2N=6 consecutive ERROR epochs → self-SIGTERM.
    let result = trainer.run_n_epochs(10);

    match result {
        Err(TrainingError::RegressionDetected { metric, consecutive_epochs, .. }) => {
            assert_eq!(metric, "train_loss");
            assert_eq!(consecutive_epochs, 6);
        }
        _ => panic!("Expected self-termination, got {:?}", result),
    }
}
```

- [ ] **Step 2.2: Run — expect FAIL**

Regression detection not wired.

- [ ] **Step 2.3: Implement band-reading**

In `monitoring.rs`:

```rust
pub struct MetricBands {
    pub warn_low: f64,
    pub warn_high: f64,
    pub error_low: f64,
    pub error_high: f64,
}

pub struct MetricBandsRegistry {
    bands: HashMap<String, MetricBands>,
    consecutive_warn:  HashMap<String, u32>,
    consecutive_error: HashMap<String, u32>,
}

impl MetricBandsRegistry {
    pub fn load_from_toml(path: &Path) -> Result<Self> { /* ... */ }

    /// Returns Some(reason) if training should self-terminate.
    pub fn update_and_check(&mut self, metric: &str, value: f64, n: u32)
        -> Option<TerminationReason>
    {
        let bands = match self.bands.get(metric) {
            Some(b) => b,
            None => return None,  // unknown metric — silent OK, covered by Invariant 7 audit
        };

        if value < bands.error_low || value > bands.error_high {
            let c = self.consecutive_error.entry(metric.into()).or_insert(0);
            *c += 1;
            if *c >= 2 * n {
                return Some(TerminationReason::RegressionError {
                    metric: metric.into(), value, band: bands.clone(), consecutive: *c,
                });
            }
        } else {
            self.consecutive_error.insert(metric.into(), 0);
        }

        if value < bands.warn_low || value > bands.warn_high {
            let c = self.consecutive_warn.entry(metric.into()).or_insert(0);
            *c += 1;
            if *c >= n {
                tracing::warn!(target: "dqn::regression",
                    "Metric {} has been in warn-band for {} epochs (value={:.3e})",
                    metric, *c, value);
            }
        } else {
            self.consecutive_warn.insert(metric.into(), 0);
        }

        None
    }
}
```

- [ ] **Step 2.4: Wire check into the per-epoch loop**

In `training_loop.rs`, after emitting HEALTH_DIAG each epoch:

```rust
for (name, value) in self.harvest_health_diag_metrics() {
    if let Some(term) = self.metric_bands.update_and_check(&name, value, /*n=*/ 3) {
        tracing::error!(target: "dqn::regression", "{}", term);
        return Err(TrainingError::RegressionDetected { /* fields from term */ });
    }
}
```

Clean termination: trainer shuts down CUDA resources, flushes the replay buffer to disk (if checkpointing), emits a final HEALTH_DIAG line with `TERMINATED_BY_REGRESSION`.

- [ ] **Step 2.5: Consolidate metric-bands.toml**

Ensure all Plan 3/4 metrics are covered. Sample entries for the ISV-controlled metrics:

```toml
[metric_bands.avg_q_value]
warn_low = -200.0
warn_high = 200.0
error_low = -1000.0
error_high = 1000.0

[metric_bands.train_loss]
warn_low = 1e-6
warn_high = 10.0
error_low = 0.0
error_high = 1000.0

[metric_bands.isv_plan_params_0_ema]
warn_low = 0.05
warn_high = 10.0
error_low = 0.01
error_high = 50.0

[metric_bands.isv_trade_attempt_rate_ema]
warn_low = 0.001
warn_high = 0.3
error_low = 0.0
error_high = 1.0

[metric_bands.val_sharpe]
warn_low = -5.0
warn_high = 10.0
error_low = -50.0
error_high = 100.0

# ... one entry per metric emitted to HEALTH_DIAG
```

Bands populated from the last 3 known-good runs' distributions (Plan 3/4 exit runs). Helper: `scripts/populate-metric-bands-from-runs.py` reads multiple aggregate JSONs and emits band entries at `mean ± 5σ (warn)` / `mean ± 10σ (error)`.

- [ ] **Step 2.6: Run test — expect PASS**

- [ ] **Step 2.7: Commit**

```bash
git add crates/ml/src/trainers/dqn/trainer/{monitoring.rs,training_loop.rs} \
        config/metric-bands.toml \
        scripts/populate-metric-bands-from-runs.py \
        crates/ml/src/trainers/dqn/smoke_tests/regression_detection.rs
git commit -m "plan5(task2): A.4 regression-detection hard-stop on 2N consecutive error-band"
```

---

## Task 3: A.4.1 nsys Profile Harness

**Files:**
- Modify: `infra/docker/Dockerfile.training-runtime` — install `nsys`
- Modify: `scripts/argo-train.sh` — add `--profile` flag
- Modify: `infra/k8s/argo/train-template.yaml` — conditional nsys wrapper
- Create: `scripts/compare-nsys-profiles.py` — detect >20% per-epoch regression
- Test: `nsys_profile_harness_produces_artifact`

- [ ] **Step 3.1: Add nsys to training-runtime Docker image**

In `infra/docker/Dockerfile.training-runtime`:

```dockerfile
# Install NVIDIA Nsight Systems for optional --profile runs.
ARG NSYS_VERSION=2024.4.1.61-1
RUN apt-get update && apt-get install -y --no-install-recommends \
    nsight-systems-cli=${NSYS_VERSION} \
    && rm -rf /var/lib/apt/lists/*
```

Test: `docker run ... nsys --version` should print the expected version.

- [ ] **Step 3.2: Write failing test**

```bash
# scripts/tests/test_nsys_harness.sh
./scripts/argo-train.sh --profile --multi-seed 1 --folds 1 --dry-run > /tmp/dry.yaml

grep -q "nsys profile" /tmp/dry.yaml || {
    echo "FAIL: --profile flag does not produce nsys wrapper"; exit 1
}
grep -q "foxhunt-training-artifacts" /tmp/dry.yaml || {
    echo "FAIL: --profile flag does not produce artifact upload step"; exit 1
}
echo "PASS"
```

- [ ] **Step 3.3: Run — expect FAIL**

- [ ] **Step 3.4: Implement `--profile` flag**

When set, the training container's command becomes:

```yaml
command:
- nsys
- profile
- --capture-range=cudaProfilerApi
- --output=/tmp/profile-${POD_NAME}.nsys-rep
- --force-overwrite=true
- /usr/local/bin/fxt
args: [train, "--seed", "$(SEED)", "--fold", "$(FOLD)"]
```

Followed by an artifact-upload sidecar:

```yaml
- name: upload-profile
  image: minio/mc:latest
  command: [sh, -c]
  args:
  - |
    mc cp /tmp/profile-*.nsys-rep minio/foxhunt-training-artifacts/profiles/${SHORT_SHA}/
```

- [ ] **Step 3.5: Implement `compare-nsys-profiles.py`**

```python
# Compare per-epoch mean wall-time between baseline profile and current profile.
# Inputs: two .nsys-rep paths.
# Output: pass/fail with details if regression >20%.
import sys
from pathlib import Path
from subprocess import run

def extract_per_epoch_duration(nsys_rep: Path) -> list[float]:
    # nsys stats --report cuda_gpu_trace-then aggregate per-epoch.
    out = run(["nsys", "stats", "--report", "cuda_gpu_trace", str(nsys_rep)],
              capture_output=True, text=True, check=True)
    # ... parse cuda_gpu_trace CSV, aggregate by epoch marker pushes ...
    return [per_epoch_mean_ms_list]

def main():
    baseline = Path(sys.argv[1])
    current  = Path(sys.argv[2])
    b_durs = extract_per_epoch_duration(baseline)
    c_durs = extract_per_epoch_duration(current)

    b_med = sorted(b_durs)[len(b_durs) // 2]
    c_med = sorted(c_durs)[len(c_durs) // 2]
    ratio = c_med / b_med

    if ratio > 1.20:
        print(f"FAIL: per-epoch regression {ratio:.2f}x (baseline={b_med:.1f}ms, current={c_med:.1f}ms)")
        sys.exit(1)
    else:
        print(f"PASS: per-epoch within 20% ({ratio:.2f}x)")

if __name__ == "__main__":
    main()
```

- [ ] **Step 3.6: Capture baseline profile**

One run with `--profile` on current main → `foxhunt-training-artifacts/profiles/baseline-plan5/profile-*.nsys-rep`. Document commit SHA of baseline.

- [ ] **Step 3.7: Run test — expect PASS**

- [ ] **Step 3.8: Commit**

```bash
git add infra/docker/Dockerfile.training-runtime \
        scripts/argo-train.sh \
        scripts/tests/test_nsys_harness.sh \
        infra/k8s/argo/train-template.yaml \
        scripts/compare-nsys-profiles.py
git commit -m "plan5(task3): A.4.1 nsys profile harness with regression-comparison script"
```

---

## Task 4: Validation script suite

Convert the ad-hoc Python checks from earlier plans (`check-tier1-exit.py`, `check-tier2-exit.py`, etc.) into a single versioned validation suite.

**Files:**
- Create: `scripts/validation/check_tier1.py`
- Create: `scripts/validation/check_tier2.py`
- Create: `scripts/validation/check_tier3.py`
- Create: `scripts/validation/check_all_tiers.py` — top-level wrapper
- Test: `tier_check_scripts_reject_bad_metrics`

- [ ] **Step 4.1: Write failing test**

```bash
# scripts/validation/tests/test_tier_checks.sh
set -euo pipefail

# Tier 1: bad metrics JSON should fail.
python3 scripts/validation/check_tier1.py scripts/validation/tests/fixtures/bad_tier1.json --warmup-end 15 && {
    echo "FAIL: tier1 check accepted bad metrics"
    exit 1
}

# Tier 1: known-good metrics JSON should pass.
python3 scripts/validation/check_tier1.py scripts/validation/tests/fixtures/good_tier1.json --warmup-end 15 || {
    echo "FAIL: tier1 check rejected good metrics"
    exit 1
}

echo "PASS"
```

- [ ] **Step 4.2: Run — expect FAIL**

- [ ] **Step 4.3: Implement `check_tier1.py`**

```python
#!/usr/bin/env python3
"""Tier 1 convergence exit check.

Exits 0 on PASS, 1 on FAIL.
Input: aggregated metrics JSON (from gather-multi-seed-metrics.sh).
"""
import argparse, json, sys
from statistics import mean, stdev

def check_std_over_mean(agg, warmup_end):
    for metric in ["val_sharpe", "avg_q_value", "train_loss"]:
        stable = [x for x in agg[metric] if x["epoch"] >= warmup_end]
        if not stable:
            return False, f"No stable epochs for {metric} past warmup_end={warmup_end}"
        m = mean(x["mean"] for x in stable)
        s = mean(x["std"] for x in stable)
        ratio = abs(s / (m + 1e-9)) if abs(m) > 1e-6 else float("inf")
        if ratio > 0.15:
            return False, f"{metric} std/mean = {ratio:.3f} > 0.15 on stable epochs"
    return True, "std/mean < 0.15 on all Tier 1 metrics"

def check_no_fold1_explosion(agg):
    max_q = max((x["max"] for x in agg.get("avg_q_value", [])), default=0)
    if abs(max_q) > 500:
        return False, f"avg_q_value max={max_q}, possible fold-1 explosion"
    return True, "no fold-1 explosion detected"

def check_no_q_saturation(agg):
    # Q saturation: q_full/q_half/q_quarter within 1% of ±abs_half.
    # Proxy via max avg_q_value observed.
    return True, "no Q-saturation check (placeholder — see A.6 audit)"

def check_no_hot_path_dtoh(agg):
    # Detected by pre-commit; if we got here, already ok.
    return True, "hot-path audit enforced by pre-commit"

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("metrics_json")
    ap.add_argument("--warmup-end", type=int, default=15)
    args = ap.parse_args()

    with open(args.metrics_json) as f:
        data = json.load(f)
    agg = data["aggregates"]

    checks = [
        check_std_over_mean(agg, args.warmup_end),
        check_no_fold1_explosion(agg),
        check_no_q_saturation(agg),
        check_no_hot_path_dtoh(agg),
    ]

    all_pass = all(ok for ok, _ in checks)
    for ok, msg in checks:
        print(("PASS" if ok else "FAIL") + ": " + msg)
    sys.exit(0 if all_pass else 1)

if __name__ == "__main__":
    main()
```

- [ ] **Step 4.4: Implement `check_tier2.py`**

Checks Tier 2 (behavioural):
- `val_trades_per_bar` ≥ 0.005.
- `val_active_frac` > 0.2.
- direction argmax entropy > 0.8 × uniform_max (log 4 = 1.386).

- [ ] **Step 4.5: Implement `check_tier3.py`**

Checks Tier 3 (profitability):
- `val_sharpe_annualised` > 1.0.
- `val_win_rate` ≥ 0.52 (only on runs with trade count > 500).
- `val_profit_factor` ≥ 1.1 (stable across seeds; std < 0.3).

- [ ] **Step 4.6: Wrapper `check_all_tiers.py`**

Runs all three, exits 0 only if all pass.

- [ ] **Step 4.7: Create fixtures**

Two JSON files in `scripts/validation/tests/fixtures/`:
- `good_tier1.json` — synthetic metrics representing a clean converged run.
- `bad_tier1.json` — synthetic metrics with large std/mean.

Both small (~50 lines each).

- [ ] **Step 4.8: Run test — expect PASS**

- [ ] **Step 4.9: Commit**

```bash
git add scripts/validation/
git commit -m "plan5(task4): tiered-exit validation script suite (tier1/2/3 checks)"
```

---

## Task 5: Final multi-seed × multi-fold validation pass

- [ ] **Step 5.1: Pre-flight checks**

```bash
# Working tree clean, main is ahead of origin by 0 commits.
git status --porcelain | test -z "$(cat)"
git fetch origin main
test "$(git rev-parse main)" = "$(git rev-parse origin/main)"

# All smoke tests pass locally.
SQLX_OFFLINE=true FOXHUNT_TEST_DATA=test_data/futures-baseline \
    cargo test -p ml --lib -- smoke_tests --ignored --nocapture 2>&1 | tee /tmp/final-smoke.log
grep -q "test result: ok" /tmp/final-smoke.log
```

- [ ] **Step 5.2: Launch the final validation matrix**

Per Invariants: always `git push origin main` before deploy.

```bash
git push origin main
./scripts/argo-train.sh --multi-seed 5 --folds 6 --commit $(git rev-parse HEAD) --tag dqnv2-final
```

Monitor progress via Argo UI or:

```bash
argo list -l foxhunt-tag=dqnv2-final
argo wait @latest -l foxhunt-tag=dqnv2-final --timeout 6h
```

- [ ] **Step 5.3: Gather metrics**

```bash
./scripts/gather-multi-seed-metrics.sh dqnv2-final > /tmp/dqnv2-final-metrics.json
./scripts/aggregate-norm-stats.py \
    --input-dir /mnt/training-data/norm-stats/dqnv2-final \
    --output    /mnt/training-data/norm-stats/dqnv2-final-aggregate.json
```

- [ ] **Step 5.4: Run validation suite**

```bash
python3 scripts/validation/check_all_tiers.py /tmp/dqnv2-final-metrics.json --warmup-end 15
```

Exit code MUST be 0. If non-zero:

- Do NOT retry with different seeds or a different tag.
- Open the affected tier's failure mode as a separate task for investigation.
- This plan does not land until Tier 1 + Tier 2 + Tier 3 all PASS on the FIRST final run.

Per `feedback_stop_on_anomaly.md`: re-running the validation matrix hoping for a different outcome is anti-pattern — the failure indicates something mechanical is wrong.

- [ ] **Step 5.5: Run nsys regression check**

```bash
# Baseline captured in Task 3 Step 3.6
mc cp minio/foxhunt-training-artifacts/profiles/baseline-plan5/profile-seed0-fold0.nsys-rep /tmp/baseline.nsys-rep
mc cp minio/foxhunt-training-artifacts/profiles/$(git rev-parse --short HEAD)/profile-seed0-fold0.nsys-rep /tmp/current.nsys-rep

python3 scripts/compare-nsys-profiles.py /tmp/baseline.nsys-rep /tmp/current.nsys-rep
```

Exit 0 required.

- [ ] **Step 5.6: Document the final run**

Create `docs/dqn-v2-final-validation.md`:

```markdown
# DQN v2 — Final Validation Run

**Date:** <DATE>
**Commit SHA:** <SHA>
**Argo tag:** dqnv2-final
**Matrix:** 5 seeds × 6 folds = 30 training jobs
**GPU hours:** ~<H> hours L40S

## Tier 1 — Convergence discipline

[paste check_tier1.py output]

## Tier 2 — Behavioural parity

[paste check_tier2.py output, with val_trades_per_bar, val_active_frac, direction entropy values]

## Tier 3 — Profitability

[paste check_tier3.py output with val_sharpe, win_rate, profit_factor, std across seeds]

## nsys regression

[paste compare-nsys-profiles.py output]

## HEALTH_DIAG sample lines (last 3 epochs, seed 0, fold 5)

[paste 3 lines showing reward_split, temporal_reward, attn, aux, and controller values]

## Sign-off

DQN v2 unified integrated policy system — landed and validated.
All 9 invariants preserved across every commit in the rollout.
```

- [ ] **Step 5.7: Commit final validation**

```bash
git add docs/dqn-v2-final-validation.md
git commit -m "plan5(task5): DQN v2 final validation — Tier 1+2+3 exit PASS on first run"
```

---

## Task 6: Rollout close-out

**Files:**
- Modify: `docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md` — add status footer
- Modify: `MEMORY.md` — add retrospective pointers
- Archive: move all `docs/dqn-v2-plan-N-validation.md` docs into `docs/archive/dqn-v2-rollout/`
- Retrospective: short `docs/dqn-v2-rollout-retrospective.md` summarising lessons

- [ ] **Step 6.1: Mark spec as complete**

In `docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md`, change the header:

```markdown
**Date:** 2026-04-24
**Status:** LANDED (final validation passed <DATE>, commit <SHA>)
```

- [ ] **Step 6.2: Retrospective**

Create `docs/dqn-v2-rollout-retrospective.md`:

```markdown
# DQN v2 Rollout Retrospective

5 sequential plans, all landed, all 9 invariants preserved.

## What worked

- Plan 1's reset registry + AdaptiveController trait early.
  Every later controller landed faster because the protocol was defined.
- The "no feature flags" invariant forced every mechanism to be always-on,
  which eliminated the "config combinatorics" debugging class.
- Multi-seed harness found the val-Flat-trap's residual variance issue
  that single-seed runs had missed twice.

## What hurt

- [fill in from the actual implementation experience — drift from spec,
  surprises, areas where the plan was too optimistic, tasks that took
  3x the estimated time, etc.]

## What we'd do differently

- [from implementation experience]

## Open questions for v3

- [any new orphans or ambiguities discovered during implementation]
```

The retrospective is written AFTER Plans 1–5 are all landed, not before. It documents actual experience, not predicted.

- [ ] **Step 6.3: Update MEMORY.md**

Add:

```markdown
- [dqn-v2-rollout-retrospective.md](dqn-v2-rollout-retrospective.md) — what worked, what hurt, what to do differently for DQN v3
```

- [ ] **Step 6.4: Archive plan-level validation docs**

```bash
mkdir -p docs/archive/dqn-v2-rollout/
git mv docs/dqn-v2-plan-{1,2,3,4}-validation.md docs/archive/dqn-v2-rollout/
# Plan 5 final validation stays at docs/dqn-v2-final-validation.md (primary output).
```

- [ ] **Step 6.5: Commit close-out**

```bash
git add docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md \
        docs/dqn-v2-rollout-retrospective.md \
        MEMORY.md
git add docs/archive/dqn-v2-rollout/
git commit -m "plan5(task6): DQN v2 rollout close-out — spec marked LANDED, retrospective added"
```

---

## Plan 5 Exit Criteria Summary (final rollout gate)

- [ ] All 9 invariants preserved across every commit in this plan and in all of Plans 1–4.
- [ ] Multi-seed × multi-fold Argo harness wired, tested, operational.
- [ ] Regression detection hard-stops training on sustained error-band metrics.
- [ ] nsys profile harness produces `.nsys-rep` artifacts per run; baseline established.
- [ ] Tier 1 (convergence discipline) — PASS on final run.
- [ ] Tier 2 (behavioural parity) — PASS on final run.
- [ ] Tier 3 (profitability) — PASS on final run.
- [ ] No per-epoch regression > 20% vs. pre-spec baseline (nsys comparison).
- [ ] Spec marked LANDED.
- [ ] Retrospective written from actual implementation experience.
- [ ] All Plan-level validation docs archived; final validation doc published.
- [ ] Wire-up audit, ISV-slot registry, hot-path audit, named-dims registry — all current.

The DQN v2 unified integrated policy system is declared complete when all boxes in this summary are ticked.

---

## Post-rollout operational notes

- Multi-seed × multi-fold validation is the canonical pre-merge check for DQN changes going forward. Single-seed runs are dev-iteration only, not merge-blockers.
- `config/metric-bands.toml` is the living document; update bands via `scripts/populate-metric-bands-from-runs.py` after each successful baseline run.
- nsys baseline profile is re-captured on each quarterly refresh or major CUDA-toolchain update. Tag baselines with commit SHA + date.
- Part E audit doc is authoritative for "does the DQN consume concept X". Modifying the audit requires landing the wiring change in the same commit.
- The invariants from §3 of the spec become permanent pre-commit checks; no DQN-touching commit lands without them.

The next generation (DQN v3) starts by filing open questions from Plan 5's retrospective as specs, not by bolting new mechanisms onto the v2 substrate.
