# Production Scheduling — Consolidate onto Dagster (K8sRunLauncher) + build the proper image

> **For the operator/agent:** execute inline, checkpoint per task. Live infra + real-money fund. Operator
> decision (2026-07-19): go ALL-DAGSTER properly — upgrade the Dagster infra so heavy compute runs in
> per-run K8s Jobs (own memory), fold the cron precompute into the DAG, and build the image the right way
> (podman → bizworx registry, bypassing the dead Argo/Kaniko/GitLab pipeline).

**Goal:** One orchestrator (Dagster) owns the whole nightly ETL + precompute as an asset graph, with per-run
memory isolation via `K8sRunLauncher` (heavy assets get 6-8Gi Jobs, the daemon stays 4Gi). Remove the
`compare-measured-precompute` cron (folded into the DAG). Keep EXECUTION (rebalancers) in K8s CronJobs
(deliberate money-exec blast-radius isolation). Delete the dead VRP. Result: consistent, documented, not-confusing.

## Why this is the right end-state (and why it needs an infra upgrade)
- The split felt "buggy" because `bybit_sleeve_ret` is written by a SUSPENDED own-memory cron, cross-timed to the
  Dagster 23:30 ingest. Folding it into Dagster removes the timing coupling.
- BUT the cron exists for a HARD reason: the sleeve-ret compute needs **6-8Gi** and OOM-killed the 4Gi daemon.
  Naively making it a daemon-run asset re-breaks the nightly. The CORRECT fix = `K8sRunLauncher`: each Dagster
  run launches as its own right-sized K8s Job; the daemon only schedules/launches (stays tiny). This is the
  standard Dagster production pattern.

## Current-state facts (verified 2026-07-19)
- Dagster ETL HEALTHY: 23:30 run SUCCESS, all 14 assets materialized, forward_nav fresh 07-18. (working_directory
  /app→/code/src change already applied — harmless, more-correct.)
- `dagster_k8s` NOT in the image (orchestration extra = dagster/webserver/postgres only). RBAC: dagster SA
  `tailscale-dagster` CANNOT create jobs. Run launcher: default in-process (4Gi daemon).
- Image `rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest` (633MB) built via Dockerfile
  `infra/docker/fxhnt.Dockerfile` (`pip install .[web,pg,data,factory,orchestration,poc]`). The Argo/Kaniko
  build pipeline is DEAD (ci-compile-cpu pool gone + clones the killed GitLab).
- Build capability: devbox has **podman 4.9.3, x86_64** = cluster amd64/linux. Registry live + reachable.

## Task 1: Add dagster-k8s to deps + build the image RIGHT (podman → bizworx registry)
**Files:** `pyproject.toml`, `infra/docker/fxhnt.Dockerfile` (verify), build via podman.
- [ ] Add `dagster-k8s>=0.28,<0.29` to the `orchestration` extra in pyproject.toml (matches dagster 1.12 / the
      0.28 dagster-postgres pin). `uv lock` to update uv.lock.
- [ ] `podman login rg.fr-par.scw.cloud` with an scw secret-key (username `nologin`, password = SCW secret key;
      NEVER print it). Verify login OK.
- [ ] `podman build --platform linux/amd64 -f infra/docker/fxhnt.Dockerfile -t
      rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:dagster-k8s-<sha> .` (tag with the git sha, NOT :latest yet).
- [ ] `podman push` the sha-tag. VERIFY it lands in the registry (`scw registry image list`).
- **EVAL:** image pushed; `podman run --rm <tag> python -c "import dagster_k8s; print(dagster_k8s.__version__)"`
      succeeds. Do NOT retag :latest until the new image is proven on-cluster (Task 4). Roll back = keep :latest.

## Task 2: RBAC — let the dagster SA launch Jobs + a Job SA with fund access
**Files:** `infra/k8s/orchestration/dagster.yaml` (Role/RoleBinding + a run-job SA).
- [ ] Add a Role granting the dagster SA: create/get/list/watch/delete on `jobs` (batch) + `pods`, get on
      `pods/log`. Bind to `tailscale-dagster` (or a dedicated `dagster-runner` SA).
- [ ] Define the SA the LAUNCHED Jobs use — needs the fund secrets (db-credentials, databento, tiingo),
      git-sync key, and the same NP egress (postgres, gitea, apiserver, public HTTPS). Mirror the cron pods'
      access. (Heavy assets that hit ib-gateway would need that NP too — but the nightly precompute does not.)
- [ ] **VERIFY:** `kubectl auth can-i create jobs --as=system:serviceaccount:foxhunt:<sa>` = yes.
- **EVAL:** RBAC in place, least-privilege (namespaced Role, not cluster-admin).

## Task 3: Configure K8sRunLauncher + per-asset memory + make sleeve-ret/backtest-refs assets
**Files:** `infra/k8s/orchestration/dagster.yaml` (dagster.yaml instance config), `src/fxhnt/adapters/
orchestration/{assets.py,definitions.py}`.
- [ ] dagster.yaml: add `run_launcher: module dagster_k8s class K8sRunLauncher` with job_namespace foxhunt,
      the new image, image_pull_secrets scw-registry, the Job SA, and default resources. Add a `run_k8s_config`
      / tag-based per-run memory (heavy assets → 6Gi req/8Gi limit, pinned to the large pool).
- [ ] Wrap `persist_bybit_sleeve_returns` (already an import-light helper) as a Dagster `@asset(deps=[
      bybit_warehouse_refresh])` with a k8s memory tag; add a `backtest_refs` asset (deps on the sleeve-ret +
      nav assets) wrapping the `backtest-refs` logic. Add both to combined_book_job selection + defs.
- [ ] Keep the plain helper functions intact (unit-testable); the assets just call them (no logic duplication).
- [ ] Add/point tests: the existing sleeve-ret + backtest-refs unit tests still pass; a new test asserts the
      assets are in the job selection.
- **EVAL:** `uv run pytest` for the touched modules green; `python -c "import ...definitions"` lists the new
      assets in combined_book_job. Do NOT deploy yet.

## Task 4: Deploy the new image + config, materialize the nightly once, VERIFY end-to-end
**Files:** none (deploy).
- [ ] Point dagster deploy at the new sha-tagged image (edit dagster.yaml image or a kustomize overlay), apply,
      rollout restart. git-sync pulls the new src (with the new assets).
- [ ] Trigger `combined_book_forward_job` once. WATCH: the daemon LAUNCHES a K8s Job (not in-process); the Job
      gets 6-8Gi on the large pool; it materializes ALL assets incl the new sleeve-ret + backtest-refs WITHOUT
      OOM-killing the daemon.
- [ ] **VERIFY:** `bybit_sleeve_ret` max(run_date) = today, src_updated = now; backtest_summary fresh; forward
      tracks still fresh; cockpit reflects it; daemon did NOT restart/OOM. Check `kubectl get jobs -n foxhunt`
      shows the dagster-launched run Job completed.
- **EVAL:** full nightly runs in an isolated Job, data fresh, daemon healthy. THIS is the money shot. If OOM or
      launcher fails → debug (RBAC/image/config), do NOT retag :latest. Roll back = old image + re-arm the cron.
- [ ] Once proven: retag the sha image as `:latest` (or update the manifest to the sha) so it's the deployed default.

## Task 5: Remove the now-redundant compare-measured cron + delete VRP + arm execution + document
**Files:** `infra/k8s/jobs/*.yaml`, memory, progress anchor.
- [ ] DELETE `fxhnt-compare-measured-precompute` cron + its NP (its work is now the Dagster assets). `git rm` the
      manifest. (Keep `factory` cron — it's migrate-cockpit + factory-cycle, separate concern; ARM it.)
- [ ] DELETE `fxhnt-vrp-rebalancer` cron + NP + manifest (shelved/falsified; vrp_nav already de-wired). Update
      memory (project_fxhnt_diversifier_hunt_2026_07_15, project_fxhnt_vrp_cvar_crash_research): VRP exec cron DELETED.
- [ ] ARM the EXECUTION crons that should run: `factory` (daily), `ucits-volume-ibkr` + `ucits-rebalancer`
      (weekday, ucits already armed). Keep SUSPENDED: `bybit-rebalancer` (no creds), `ibkr-rebalancer` (PRIIPs).
      Patch live + set suspend in manifests (source of truth).
- [ ] **VERIFY:** cronjobs = {factory, ucits-volume-ibkr, ucits-rebalancer} armed; {bybit, ibkr} suspended;
      {compare-measured, vrp} GONE. Dagster owns all ETL/precompute.
- [ ] Document the final architecture in the progress anchor + a CLAUDE-visible note: Dagster (K8sRunLauncher,
      per-run memory) = ALL nightly ETL+precompute; K8s CronJobs = execution only. Why exec stays crons
      (blast-radius). Update reference_fxhnt_prod_deploy_mechanics with the podman build path (Argo build dead).
- **EVAL:** consistent single-orchestrator ETL, execution isolated in crons, nothing suspended-by-accident,
      VRP gone, architecture documented. Done.

## Global constraints
- VERIFY-FIRST, per-task eval. REAL MONEY: only ucits-rebalancer trades (paper). Never arm bybit/ibkr exec.
- NEVER print the SCW secret key / registry password. podman login via stdin or env, not argv.
- Image: tag with git sha, prove on-cluster BEFORE :latest. Old :latest stays as instant rollback.
- Manifests are source of truth: every live change also edits the committed yaml.
- Don't duplicate logic: assets call the existing import-light helpers; keep them unit-testable.
