docs(plan): rewrite for all-Dagster K8sRunLauncher + proper podman image build

Operator: go all-Dagster properly + build the image right. Diagnosis
corrected: Dagster ETL was healthy all along (23:30 run SUCCESS, 14 assets,
forward_nav fresh); the stale data is the SUSPENDED compare-measured cron that
writes bybit_sleeve_ret. That cron exists because the compute needs 6-8Gi and
OOM-killed the 4Gi daemon — so the correct all-Dagster fix is K8sRunLauncher
(per-run Jobs with own memory), not daemon-run assets. Plan: add dagster-k8s,
build+push via podman (Argo/Kaniko pipeline is dead), add Job-launch RBAC,
configure K8sRunLauncher + per-asset memory, fold sleeve-ret/backtest-refs into
the DAG, materialize+verify, then delete compare-measured+VRP crons and arm
execution. Also commits the harmless working_directory=/code/src fix.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docs/superpowers/plans/2026-07-19-production-scheduling-consistency.md

@@ -1,100 +1,104 @@
-# Production Scheduling Consistency — Dagster ETL + K8s Cron Execution
+# Production Scheduling — Consolidate onto Dagster (K8sRunLauncher) + build the proper image
 
-> **For the operator/agent:** execute inline, checkpoint per task. Live infra + real-money fund. This fixes the
-> half-suspended, split-brain scheduling left after the foxhunt→bizworx cutover so bizworx runs a CONSISTENT
-> production system (nothing silently stalled).
+> **For the operator/agent:** execute inline, checkpoint per task. Live infra + real-money fund. Operator
+> decision (2026-07-19): go ALL-DAGSTER properly — upgrade the Dagster infra so heavy compute runs in
+> per-run K8s Jobs (own memory), fold the cron precompute into the DAG, and build the image the right way
+> (podman → bizworx registry, bypassing the dead Argo/Kaniko/GitLab pipeline).
 
-**Goal:** One clear, working, documented two-layer schedule on bizworx — Dagster does the nightly ETL/precompute
-(7-day), K8s CronJobs do execution + post-processing. Fix the Dagster code-server crash (nothing has run since
-Sat), arm the correct crons, remove the dead VRP path, and document the architecture so it stops being confusing.
+**Goal:** One orchestrator (Dagster) owns the whole nightly ETL + precompute as an asset graph, with per-run
+memory isolation via `K8sRunLauncher` (heavy assets get 6-8Gi Jobs, the daemon stays 4Gi). Remove the
+`compare-measured-precompute` cron (folded into the DAG). Keep EXECUTION (rebalancers) in K8s CronJobs
+(deliberate money-exec blast-radius isolation). Delete the dead VRP. Result: consistent, documented, not-confusing.
 
-**Chosen architecture (operator, 2026-07-19): KEEP THE SPLIT, make it consistent.**
-- **Layer 1 — Dagster** `combined_book_daily` @ 23:30 UTC (7-day): `combined_book_forward_job` = ingest bybit
-  warehouse + futures, compute all `*_nav` forward tracks, `bybit_paper_book`, `cockpit_forward`. Purpose-built
-  for the asset DAG + deps + retries + UI. "replaces the fxhnt-forward cron" (per definitions.py docstring).
-- **Layer 2 — K8s CronJobs** staggered AFTER the 23:30 ingest: execution (rebalancers) + heavy post-processing
-  (compare-measured sleeve-ret writer, factory, backtest-refs).
+## Why this is the right end-state (and why it needs an infra upgrade)
+- The split felt "buggy" because `bybit_sleeve_ret` is written by a SUSPENDED own-memory cron, cross-timed to the
+  Dagster 23:30 ingest. Folding it into Dagster removes the timing coupling.
+- BUT the cron exists for a HARD reason: the sleeve-ret compute needs **6-8Gi** and OOM-killed the 4Gi daemon.
+  Naively making it a daemon-run asset re-breaks the nightly. The CORRECT fix = `K8sRunLauncher`: each Dagster
+  run launches as its own right-sized K8s Job; the daemon only schedules/launches (stays tiny). This is the
+  standard Dagster production pattern.
 
-## Root causes found (the "confusing and buggy" state)
-1. **Dagster code-server CRASH → schedule never fires.** `workspace.yaml` has `working_directory: /app` but the
-   git-sync'd code is at `/code/src` (PYTHONPATH=/code/src). The code server starts in the wrong dir → daemon
-   can't heartbeat it → "No heartbeat received in 20 seconds, shutting down" every 60s → the 23:30 job has NOT
-   run since the Sat 20:12 restart → bybit_sleeve_ret stale at run_date 2026-07-17, forward tracks stale.
-2. **All fund CronJobs SUSPENDED** from the cutover (only ucits-rebalancer armed) → even if Dagster ran, the
-   01:00 post-processing (sleeve-ret writer, factory, backtest-refs) wouldn't run.
-3. **Dead VRP cron still present** — `fxhnt-vrp-rebalancer` (weekly Mon). VRP is SHELVED/FALSIFIED
-   (project_fxhnt_diversifier_hunt_2026_07_15; vrp_nav already DE-WIRED from the Dagster nightly). The cron +
-   its NP are vestigial and must be REMOVED (operator: "vrp options can go, make it clear in memory + docs").
+## Current-state facts (verified 2026-07-19)
+- Dagster ETL HEALTHY: 23:30 run SUCCESS, all 14 assets materialized, forward_nav fresh 07-18. (working_directory
+  /app→/code/src change already applied — harmless, more-correct.)
+- `dagster_k8s` NOT in the image (orchestration extra = dagster/webserver/postgres only). RBAC: dagster SA
+  `tailscale-dagster` CANNOT create jobs. Run launcher: default in-process (4Gi daemon).
+- Image `rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest` (633MB) built via Dockerfile
+  `infra/docker/fxhnt.Dockerfile` (`pip install .[web,pg,data,factory,orchestration,poc]`). The Argo/Kaniko
+  build pipeline is DEAD (ci-compile-cpu pool gone + clones the killed GitLab).
+- Build capability: devbox has **podman 4.9.3, x86_64** = cluster amd64/linux. Registry live + reachable.
 
-## Target arm/suspend state (per manifest intent + operator)
-| Cron | Schedule | Cadence | ARM? | Why |
-|---|---|---|---|---|
-| compare-measured-precompute | `0 1 * * *` | daily 7d | **ARM** | sleeve-ret writer + backtest-refs; reads the 23:30 ingest |
-| factory | `0 1 * * *` | daily 7d | **ARM** | migrate-cockpit + factory-cycle |
-| ucits-volume-ibkr | `30 8 * * 1-5` | weekday | **ARM** | read-only ADV fetch, paper-safe |
-| ucits-rebalancer | `0 9 * * 1-5` | weekday | **ARM (already)** | the live PRIIPs-legal EU exec leg |
-| bybit-rebalancer | `30 0 * * *` | daily 7d | **SUSPEND** | no `bybit-testnet-credentials` secret (can't execute) |
-| ibkr-rebalancer | `35 14 * * 1-5` | weekday | **SUSPEND** | US-ETF PRIIPs/KID-rejected on the EU DU account |
-| vrp-rebalancer | `40 14 * * 1` | weekly | **DELETE** | shelved/falsified; no options perm anyway |
+## Task 1: Add dagster-k8s to deps + build the image RIGHT (podman → bizworx registry)
+**Files:** `pyproject.toml`, `infra/docker/fxhnt.Dockerfile` (verify), build via podman.
+- [ ] Add `dagster-k8s>=0.28,<0.29` to the `orchestration` extra in pyproject.toml (matches dagster 1.12 / the
+      0.28 dagster-postgres pin). `uv lock` to update uv.lock.
+- [ ] `podman login rg.fr-par.scw.cloud` with an scw secret-key (username `nologin`, password = SCW secret key;
+      NEVER print it). Verify login OK.
+- [ ] `podman build --platform linux/amd64 -f infra/docker/fxhnt.Dockerfile -t
+      rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:dagster-k8s-<sha> .` (tag with the git sha, NOT :latest yet).
+- [ ] `podman push` the sha-tag. VERIFY it lands in the registry (`scw registry image list`).
+- **EVAL:** image pushed; `podman run --rm <tag> python -c "import dagster_k8s; print(dagster_k8s.__version__)"`
+      succeeds. Do NOT retag :latest until the new image is proven on-cluster (Task 4). Roll back = keep :latest.
 
-Dagster `combined_book_daily` schedule: **ARM** (fix the code-server first, then confirm it ticks).
+## Task 2: RBAC — let the dagster SA launch Jobs + a Job SA with fund access
+**Files:** `infra/k8s/orchestration/dagster.yaml` (Role/RoleBinding + a run-job SA).
+- [ ] Add a Role granting the dagster SA: create/get/list/watch/delete on `jobs` (batch) + `pods`, get on
+      `pods/log`. Bind to `tailscale-dagster` (or a dedicated `dagster-runner` SA).
+- [ ] Define the SA the LAUNCHED Jobs use — needs the fund secrets (db-credentials, databento, tiingo),
+      git-sync key, and the same NP egress (postgres, gitea, apiserver, public HTTPS). Mirror the cron pods'
+      access. (Heavy assets that hit ib-gateway would need that NP too — but the nightly precompute does not.)
+- [ ] **VERIFY:** `kubectl auth can-i create jobs --as=system:serviceaccount:foxhunt:<sa>` = yes.
+- **EVAL:** RBAC in place, least-privilege (namespaced Role, not cluster-admin).
 
----
+## Task 3: Configure K8sRunLauncher + per-asset memory + make sleeve-ret/backtest-refs assets
+**Files:** `infra/k8s/orchestration/dagster.yaml` (dagster.yaml instance config), `src/fxhnt/adapters/
+orchestration/{assets.py,definitions.py}`.
+- [ ] dagster.yaml: add `run_launcher: module dagster_k8s class K8sRunLauncher` with job_namespace foxhunt,
+      the new image, image_pull_secrets scw-registry, the Job SA, and default resources. Add a `run_k8s_config`
+      / tag-based per-run memory (heavy assets → 6Gi req/8Gi limit, pinned to the large pool).
+- [ ] Wrap `persist_bybit_sleeve_returns` (already an import-light helper) as a Dagster `@asset(deps=[
+      bybit_warehouse_refresh])` with a k8s memory tag; add a `backtest_refs` asset (deps on the sleeve-ret +
+      nav assets) wrapping the `backtest-refs` logic. Add both to combined_book_job selection + defs.
+- [ ] Keep the plain helper functions intact (unit-testable); the assets just call them (no logic duplication).
+- [ ] Add/point tests: the existing sleeve-ret + backtest-refs unit tests still pass; a new test asserts the
+      assets are in the job selection.
+- **EVAL:** `uv run pytest` for the touched modules green; `python -c "import ...definitions"` lists the new
+      assets in combined_book_job. Do NOT deploy yet.
 
-## Task 1: Fix the Dagster code-server working_directory (root cause of the stall)
-**Files:** `infra/k8s/orchestration/dagster.yaml` (the `dagster-instance`/workspace ConfigMap).
-- [ ] Find the workspace.yaml source (ConfigMap `dagster-instance` or `dagster-cfg`) in dagster.yaml; the
-      init-container copies it to /dagster-home/workspace.yaml.
-- [ ] Change `working_directory: /app` → `working_directory: /code/src` (match PYTHONPATH + git-sync target).
-      (If the module is import-resolvable via PYTHONPATH alone, working_directory may just need to be a valid
-      dir; /code/src is correct + consistent.)
-- [ ] Apply the ConfigMap; `rollout restart deploy/dagster`.
-- [ ] **VERIFY:** daemon log NO LONGER shows "No heartbeat"; `dagster schedule list -w /dagster-home/workspace.yaml`
-      shows `combined_book_daily [RUNNING]`; a code-server stays up (grpc healthy).
-- **EVAL:** code server stable (no heartbeat-loss loop) for 3+ min. If still crashing → capture the actual
-      import/load error (systematic-debugging), do NOT proceed.
+## Task 4: Deploy the new image + config, materialize the nightly once, VERIFY end-to-end
+**Files:** none (deploy).
+- [ ] Point dagster deploy at the new sha-tagged image (edit dagster.yaml image or a kustomize overlay), apply,
+      rollout restart. git-sync pulls the new src (with the new assets).
+- [ ] Trigger `combined_book_forward_job` once. WATCH: the daemon LAUNCHES a K8s Job (not in-process); the Job
+      gets 6-8Gi on the large pool; it materializes ALL assets incl the new sleeve-ret + backtest-refs WITHOUT
+      OOM-killing the daemon.
+- [ ] **VERIFY:** `bybit_sleeve_ret` max(run_date) = today, src_updated = now; backtest_summary fresh; forward
+      tracks still fresh; cockpit reflects it; daemon did NOT restart/OOM. Check `kubectl get jobs -n foxhunt`
+      shows the dagster-launched run Job completed.
+- **EVAL:** full nightly runs in an isolated Job, data fresh, daemon healthy. THIS is the money shot. If OOM or
+      launcher fails → debug (RBAC/image/config), do NOT retag :latest. Roll back = old image + re-arm the cron.
+- [ ] Once proven: retag the sha image as `:latest` (or update the manifest to the sha) so it's the deployed default.
 
-## Task 2: Manually materialize the nightly once (catch up the stale data)
-**Files:** none.
-- [ ] Trigger `combined_book_forward_job` once now (via dagster CLI in the daemon pod:
-      `dagster job launch -w /dagster-home/workspace.yaml -j combined_book_forward_job` or the webserver UI over
-      the tailnet) so bybit + forward tracks catch up without waiting for 23:30.
-- [ ] **VERIFY:** `bybit_sleeve_ret` max(run_date) advances to today; `src_updated_at` = now; forward_nav max(date)
-      advances; cockpit reflects it.
-- **EVAL:** data fresh. If the job errors → debug the asset (likely surfer-data or bybit warehouse path), fix,
-      re-run. This is the real end-to-end proof the ETL layer works on bizworx.
-
-## Task 3: Arm the correct CronJobs
-**Files:** `infra/k8s/jobs/fxhnt-*.yaml` (suspend fields) + live patch.
-- [ ] Un-suspend on bizworx: `compare-measured-precompute`, `factory`, `ucits-volume-ibkr` (ucits-rebalancer
-      already armed). Patch live + set `suspend: false` in the manifests (source of truth).
-- [ ] Leave SUSPENDED: `bybit-rebalancer` (no creds), `ibkr-rebalancer` (PRIIPs). Keep `suspend: true` in manifests.
-- [ ] **VERIFY:** `kubectl get cronjobs` shows exactly {compare-measured, factory, ucits-volume-ibkr,
-      ucits-rebalancer} = false; {bybit, ibkr} = true; vrp = gone (Task 4).
-- **EVAL:** arm state matches the target table. Real-money note: only ucits-rebalancer places orders (paper);
-      the newly-armed ones are precompute/read-only/factory — safe.
-
-## Task 4: Remove the dead VRP cron + make its shelved status explicit
-**Files:** `infra/k8s/jobs/fxhnt-vrp-rebalancer.yaml` (delete), memory, docs.
-- [ ] `kubectl --context $BIZ -n foxhunt delete cronjob fxhnt-vrp-rebalancer` + delete its NetworkPolicy.
-- [ ] `git rm infra/k8s/jobs/fxhnt-vrp-rebalancer.yaml`.
-- [ ] Update memory `project_fxhnt_diversifier_hunt_2026_07_15.md` + `project_fxhnt_vrp_cvar_crash_research.md`:
-      note the VRP EXECUTION cron is now DELETED (shelved/falsified; vrp_nav de-wired from nightly; no options
-      perm). Add a one-liner to the platform roadmap / this consolidation doc.
-- [ ] **VERIFY:** vrp cron + NP gone on bizworx; manifest removed; memory updated.
-- **EVAL:** no dead VRP scheduling anywhere; documentation clear.
-
-## Task 5: Verify the full production cadence end-to-end + document
-**Files:** the consolidation progress anchor + a short architecture note.
-- [ ] Confirm the timing chain is coherent: Dagster 23:30 ingest → 01:00 compare-measured/factory (7-day) →
-      08:30 ucits-volume + 09:00 ucits-rebalancer (weekday). No cron reads data before Dagster writes it.
-- [ ] Document the two-layer architecture (Dagster ETL @23:30 7-day + K8s cron execution) in the progress anchor
-      and/or a CLAUDE-visible note so it's not confusing next time. Record which jobs are armed vs. suspended-with-reason.
-- [ ] **VERIFY:** one full cycle observed (or simulated by manual triggers) produces fresh data + a clean
-      cockpit; the armed cron next-run times are sane.
-- **EVAL:** consistent production system running, nothing silently stalled, architecture documented.
+## Task 5: Remove the now-redundant compare-measured cron + delete VRP + arm execution + document
+**Files:** `infra/k8s/jobs/*.yaml`, memory, progress anchor.
+- [ ] DELETE `fxhnt-compare-measured-precompute` cron + its NP (its work is now the Dagster assets). `git rm` the
+      manifest. (Keep `factory` cron — it's migrate-cockpit + factory-cycle, separate concern; ARM it.)
+- [ ] DELETE `fxhnt-vrp-rebalancer` cron + NP + manifest (shelved/falsified; vrp_nav already de-wired). Update
+      memory (project_fxhnt_diversifier_hunt_2026_07_15, project_fxhnt_vrp_cvar_crash_research): VRP exec cron DELETED.
+- [ ] ARM the EXECUTION crons that should run: `factory` (daily), `ucits-volume-ibkr` + `ucits-rebalancer`
+      (weekday, ucits already armed). Keep SUSPENDED: `bybit-rebalancer` (no creds), `ibkr-rebalancer` (PRIIPs).
+      Patch live + set suspend in manifests (source of truth).
+- [ ] **VERIFY:** cronjobs = {factory, ucits-volume-ibkr, ucits-rebalancer} armed; {bybit, ibkr} suspended;
+      {compare-measured, vrp} GONE. Dagster owns all ETL/precompute.
+- [ ] Document the final architecture in the progress anchor + a CLAUDE-visible note: Dagster (K8sRunLauncher,
+      per-run memory) = ALL nightly ETL+precompute; K8s CronJobs = execution only. Why exec stays crons
+      (blast-radius). Update reference_fxhnt_prod_deploy_mechanics with the podman build path (Argo build dead).
+- **EVAL:** consistent single-orchestrator ETL, execution isolated in crons, nothing suspended-by-accident,
+      VRP gone, architecture documented. Done.
 
 ## Global constraints
-- VERIFY-FIRST, per-task eval. Real-money: only ucits-rebalancer trades (paper) — don't arm bybit/ibkr exec.
-- Manifests are the source of truth: every live suspend/delete also changes the committed `infra/k8s/jobs/*.yaml`.
-- Don't re-introduce the dagster/cron overlap: Dagster owns ETL/precompute, crons own execution + post-processing.
+- VERIFY-FIRST, per-task eval. REAL MONEY: only ucits-rebalancer trades (paper). Never arm bybit/ibkr exec.
+- NEVER print the SCW secret key / registry password. podman login via stdin or env, not argv.
+- Image: tag with git sha, prove on-cluster BEFORE :latest. Old :latest stays as instant rollback.
+- Manifests are source of truth: every live change also edits the committed yaml.
+- Don't duplicate logic: assets call the existing import-light helpers; keep them unit-testable.

infra/k8s/orchestration/dagster.yaml

@@ -31,7 +31,7 @@ data:
     load_from:
       - python_module:
           module_name: fxhnt.adapters.orchestration.definitions
-          working_directory: /app
+          working_directory: /code/src   # git-sync target + PYTHONPATH; /app was the baked-image dir (mismatch -> code-server couldn't heartbeat -> 23:30 schedule never fired)
 ---
 apiVersion: v1
 kind: ConfigMap