diff --git a/docs/superpowers/plans/2026-07-19-production-scheduling-consistency.md b/docs/superpowers/plans/2026-07-19-production-scheduling-consistency.md index 2c973c6..2ead94b 100644 --- a/docs/superpowers/plans/2026-07-19-production-scheduling-consistency.md +++ b/docs/superpowers/plans/2026-07-19-production-scheduling-consistency.md @@ -1,100 +1,104 @@ -# Production Scheduling Consistency — Dagster ETL + K8s Cron Execution +# Production Scheduling — Consolidate onto Dagster (K8sRunLauncher) + build the proper image -> **For the operator/agent:** execute inline, checkpoint per task. Live infra + real-money fund. This fixes the -> half-suspended, split-brain scheduling left after the foxhunt→bizworx cutover so bizworx runs a CONSISTENT -> production system (nothing silently stalled). +> **For the operator/agent:** execute inline, checkpoint per task. Live infra + real-money fund. Operator +> decision (2026-07-19): go ALL-DAGSTER properly — upgrade the Dagster infra so heavy compute runs in +> per-run K8s Jobs (own memory), fold the cron precompute into the DAG, and build the image the right way +> (podman → bizworx registry, bypassing the dead Argo/Kaniko/GitLab pipeline). -**Goal:** One clear, working, documented two-layer schedule on bizworx — Dagster does the nightly ETL/precompute -(7-day), K8s CronJobs do execution + post-processing. Fix the Dagster code-server crash (nothing has run since -Sat), arm the correct crons, remove the dead VRP path, and document the architecture so it stops being confusing. +**Goal:** One orchestrator (Dagster) owns the whole nightly ETL + precompute as an asset graph, with per-run +memory isolation via `K8sRunLauncher` (heavy assets get 6-8Gi Jobs, the daemon stays 4Gi). Remove the +`compare-measured-precompute` cron (folded into the DAG). Keep EXECUTION (rebalancers) in K8s CronJobs +(deliberate money-exec blast-radius isolation). Delete the dead VRP. Result: consistent, documented, not-confusing. -**Chosen architecture (operator, 2026-07-19): KEEP THE SPLIT, make it consistent.** -- **Layer 1 — Dagster** `combined_book_daily` @ 23:30 UTC (7-day): `combined_book_forward_job` = ingest bybit - warehouse + futures, compute all `*_nav` forward tracks, `bybit_paper_book`, `cockpit_forward`. Purpose-built - for the asset DAG + deps + retries + UI. "replaces the fxhnt-forward cron" (per definitions.py docstring). -- **Layer 2 — K8s CronJobs** staggered AFTER the 23:30 ingest: execution (rebalancers) + heavy post-processing - (compare-measured sleeve-ret writer, factory, backtest-refs). +## Why this is the right end-state (and why it needs an infra upgrade) +- The split felt "buggy" because `bybit_sleeve_ret` is written by a SUSPENDED own-memory cron, cross-timed to the + Dagster 23:30 ingest. Folding it into Dagster removes the timing coupling. +- BUT the cron exists for a HARD reason: the sleeve-ret compute needs **6-8Gi** and OOM-killed the 4Gi daemon. + Naively making it a daemon-run asset re-breaks the nightly. The CORRECT fix = `K8sRunLauncher`: each Dagster + run launches as its own right-sized K8s Job; the daemon only schedules/launches (stays tiny). This is the + standard Dagster production pattern. -## Root causes found (the "confusing and buggy" state) -1. **Dagster code-server CRASH → schedule never fires.** `workspace.yaml` has `working_directory: /app` but the - git-sync'd code is at `/code/src` (PYTHONPATH=/code/src). The code server starts in the wrong dir → daemon - can't heartbeat it → "No heartbeat received in 20 seconds, shutting down" every 60s → the 23:30 job has NOT - run since the Sat 20:12 restart → bybit_sleeve_ret stale at run_date 2026-07-17, forward tracks stale. -2. **All fund CronJobs SUSPENDED** from the cutover (only ucits-rebalancer armed) → even if Dagster ran, the - 01:00 post-processing (sleeve-ret writer, factory, backtest-refs) wouldn't run. -3. **Dead VRP cron still present** — `fxhnt-vrp-rebalancer` (weekly Mon). VRP is SHELVED/FALSIFIED - (project_fxhnt_diversifier_hunt_2026_07_15; vrp_nav already DE-WIRED from the Dagster nightly). The cron + - its NP are vestigial and must be REMOVED (operator: "vrp options can go, make it clear in memory + docs"). +## Current-state facts (verified 2026-07-19) +- Dagster ETL HEALTHY: 23:30 run SUCCESS, all 14 assets materialized, forward_nav fresh 07-18. (working_directory + /app→/code/src change already applied — harmless, more-correct.) +- `dagster_k8s` NOT in the image (orchestration extra = dagster/webserver/postgres only). RBAC: dagster SA + `tailscale-dagster` CANNOT create jobs. Run launcher: default in-process (4Gi daemon). +- Image `rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest` (633MB) built via Dockerfile + `infra/docker/fxhnt.Dockerfile` (`pip install .[web,pg,data,factory,orchestration,poc]`). The Argo/Kaniko + build pipeline is DEAD (ci-compile-cpu pool gone + clones the killed GitLab). +- Build capability: devbox has **podman 4.9.3, x86_64** = cluster amd64/linux. Registry live + reachable. -## Target arm/suspend state (per manifest intent + operator) -| Cron | Schedule | Cadence | ARM? | Why | -|---|---|---|---|---| -| compare-measured-precompute | `0 1 * * *` | daily 7d | **ARM** | sleeve-ret writer + backtest-refs; reads the 23:30 ingest | -| factory | `0 1 * * *` | daily 7d | **ARM** | migrate-cockpit + factory-cycle | -| ucits-volume-ibkr | `30 8 * * 1-5` | weekday | **ARM** | read-only ADV fetch, paper-safe | -| ucits-rebalancer | `0 9 * * 1-5` | weekday | **ARM (already)** | the live PRIIPs-legal EU exec leg | -| bybit-rebalancer | `30 0 * * *` | daily 7d | **SUSPEND** | no `bybit-testnet-credentials` secret (can't execute) | -| ibkr-rebalancer | `35 14 * * 1-5` | weekday | **SUSPEND** | US-ETF PRIIPs/KID-rejected on the EU DU account | -| vrp-rebalancer | `40 14 * * 1` | weekly | **DELETE** | shelved/falsified; no options perm anyway | +## Task 1: Add dagster-k8s to deps + build the image RIGHT (podman → bizworx registry) +**Files:** `pyproject.toml`, `infra/docker/fxhnt.Dockerfile` (verify), build via podman. +- [ ] Add `dagster-k8s>=0.28,<0.29` to the `orchestration` extra in pyproject.toml (matches dagster 1.12 / the + 0.28 dagster-postgres pin). `uv lock` to update uv.lock. +- [ ] `podman login rg.fr-par.scw.cloud` with an scw secret-key (username `nologin`, password = SCW secret key; + NEVER print it). Verify login OK. +- [ ] `podman build --platform linux/amd64 -f infra/docker/fxhnt.Dockerfile -t + rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:dagster-k8s- .` (tag with the git sha, NOT :latest yet). +- [ ] `podman push` the sha-tag. VERIFY it lands in the registry (`scw registry image list`). +- **EVAL:** image pushed; `podman run --rm python -c "import dagster_k8s; print(dagster_k8s.__version__)"` + succeeds. Do NOT retag :latest until the new image is proven on-cluster (Task 4). Roll back = keep :latest. -Dagster `combined_book_daily` schedule: **ARM** (fix the code-server first, then confirm it ticks). +## Task 2: RBAC — let the dagster SA launch Jobs + a Job SA with fund access +**Files:** `infra/k8s/orchestration/dagster.yaml` (Role/RoleBinding + a run-job SA). +- [ ] Add a Role granting the dagster SA: create/get/list/watch/delete on `jobs` (batch) + `pods`, get on + `pods/log`. Bind to `tailscale-dagster` (or a dedicated `dagster-runner` SA). +- [ ] Define the SA the LAUNCHED Jobs use — needs the fund secrets (db-credentials, databento, tiingo), + git-sync key, and the same NP egress (postgres, gitea, apiserver, public HTTPS). Mirror the cron pods' + access. (Heavy assets that hit ib-gateway would need that NP too — but the nightly precompute does not.) +- [ ] **VERIFY:** `kubectl auth can-i create jobs --as=system:serviceaccount:foxhunt:` = yes. +- **EVAL:** RBAC in place, least-privilege (namespaced Role, not cluster-admin). ---- +## Task 3: Configure K8sRunLauncher + per-asset memory + make sleeve-ret/backtest-refs assets +**Files:** `infra/k8s/orchestration/dagster.yaml` (dagster.yaml instance config), `src/fxhnt/adapters/ +orchestration/{assets.py,definitions.py}`. +- [ ] dagster.yaml: add `run_launcher: module dagster_k8s class K8sRunLauncher` with job_namespace foxhunt, + the new image, image_pull_secrets scw-registry, the Job SA, and default resources. Add a `run_k8s_config` + / tag-based per-run memory (heavy assets → 6Gi req/8Gi limit, pinned to the large pool). +- [ ] Wrap `persist_bybit_sleeve_returns` (already an import-light helper) as a Dagster `@asset(deps=[ + bybit_warehouse_refresh])` with a k8s memory tag; add a `backtest_refs` asset (deps on the sleeve-ret + + nav assets) wrapping the `backtest-refs` logic. Add both to combined_book_job selection + defs. +- [ ] Keep the plain helper functions intact (unit-testable); the assets just call them (no logic duplication). +- [ ] Add/point tests: the existing sleeve-ret + backtest-refs unit tests still pass; a new test asserts the + assets are in the job selection. +- **EVAL:** `uv run pytest` for the touched modules green; `python -c "import ...definitions"` lists the new + assets in combined_book_job. Do NOT deploy yet. -## Task 1: Fix the Dagster code-server working_directory (root cause of the stall) -**Files:** `infra/k8s/orchestration/dagster.yaml` (the `dagster-instance`/workspace ConfigMap). -- [ ] Find the workspace.yaml source (ConfigMap `dagster-instance` or `dagster-cfg`) in dagster.yaml; the - init-container copies it to /dagster-home/workspace.yaml. -- [ ] Change `working_directory: /app` → `working_directory: /code/src` (match PYTHONPATH + git-sync target). - (If the module is import-resolvable via PYTHONPATH alone, working_directory may just need to be a valid - dir; /code/src is correct + consistent.) -- [ ] Apply the ConfigMap; `rollout restart deploy/dagster`. -- [ ] **VERIFY:** daemon log NO LONGER shows "No heartbeat"; `dagster schedule list -w /dagster-home/workspace.yaml` - shows `combined_book_daily [RUNNING]`; a code-server stays up (grpc healthy). -- **EVAL:** code server stable (no heartbeat-loss loop) for 3+ min. If still crashing → capture the actual - import/load error (systematic-debugging), do NOT proceed. +## Task 4: Deploy the new image + config, materialize the nightly once, VERIFY end-to-end +**Files:** none (deploy). +- [ ] Point dagster deploy at the new sha-tagged image (edit dagster.yaml image or a kustomize overlay), apply, + rollout restart. git-sync pulls the new src (with the new assets). +- [ ] Trigger `combined_book_forward_job` once. WATCH: the daemon LAUNCHES a K8s Job (not in-process); the Job + gets 6-8Gi on the large pool; it materializes ALL assets incl the new sleeve-ret + backtest-refs WITHOUT + OOM-killing the daemon. +- [ ] **VERIFY:** `bybit_sleeve_ret` max(run_date) = today, src_updated = now; backtest_summary fresh; forward + tracks still fresh; cockpit reflects it; daemon did NOT restart/OOM. Check `kubectl get jobs -n foxhunt` + shows the dagster-launched run Job completed. +- **EVAL:** full nightly runs in an isolated Job, data fresh, daemon healthy. THIS is the money shot. If OOM or + launcher fails → debug (RBAC/image/config), do NOT retag :latest. Roll back = old image + re-arm the cron. +- [ ] Once proven: retag the sha image as `:latest` (or update the manifest to the sha) so it's the deployed default. -## Task 2: Manually materialize the nightly once (catch up the stale data) -**Files:** none. -- [ ] Trigger `combined_book_forward_job` once now (via dagster CLI in the daemon pod: - `dagster job launch -w /dagster-home/workspace.yaml -j combined_book_forward_job` or the webserver UI over - the tailnet) so bybit + forward tracks catch up without waiting for 23:30. -- [ ] **VERIFY:** `bybit_sleeve_ret` max(run_date) advances to today; `src_updated_at` = now; forward_nav max(date) - advances; cockpit reflects it. -- **EVAL:** data fresh. If the job errors → debug the asset (likely surfer-data or bybit warehouse path), fix, - re-run. This is the real end-to-end proof the ETL layer works on bizworx. - -## Task 3: Arm the correct CronJobs -**Files:** `infra/k8s/jobs/fxhnt-*.yaml` (suspend fields) + live patch. -- [ ] Un-suspend on bizworx: `compare-measured-precompute`, `factory`, `ucits-volume-ibkr` (ucits-rebalancer - already armed). Patch live + set `suspend: false` in the manifests (source of truth). -- [ ] Leave SUSPENDED: `bybit-rebalancer` (no creds), `ibkr-rebalancer` (PRIIPs). Keep `suspend: true` in manifests. -- [ ] **VERIFY:** `kubectl get cronjobs` shows exactly {compare-measured, factory, ucits-volume-ibkr, - ucits-rebalancer} = false; {bybit, ibkr} = true; vrp = gone (Task 4). -- **EVAL:** arm state matches the target table. Real-money note: only ucits-rebalancer places orders (paper); - the newly-armed ones are precompute/read-only/factory — safe. - -## Task 4: Remove the dead VRP cron + make its shelved status explicit -**Files:** `infra/k8s/jobs/fxhnt-vrp-rebalancer.yaml` (delete), memory, docs. -- [ ] `kubectl --context $BIZ -n foxhunt delete cronjob fxhnt-vrp-rebalancer` + delete its NetworkPolicy. -- [ ] `git rm infra/k8s/jobs/fxhnt-vrp-rebalancer.yaml`. -- [ ] Update memory `project_fxhnt_diversifier_hunt_2026_07_15.md` + `project_fxhnt_vrp_cvar_crash_research.md`: - note the VRP EXECUTION cron is now DELETED (shelved/falsified; vrp_nav de-wired from nightly; no options - perm). Add a one-liner to the platform roadmap / this consolidation doc. -- [ ] **VERIFY:** vrp cron + NP gone on bizworx; manifest removed; memory updated. -- **EVAL:** no dead VRP scheduling anywhere; documentation clear. - -## Task 5: Verify the full production cadence end-to-end + document -**Files:** the consolidation progress anchor + a short architecture note. -- [ ] Confirm the timing chain is coherent: Dagster 23:30 ingest → 01:00 compare-measured/factory (7-day) → - 08:30 ucits-volume + 09:00 ucits-rebalancer (weekday). No cron reads data before Dagster writes it. -- [ ] Document the two-layer architecture (Dagster ETL @23:30 7-day + K8s cron execution) in the progress anchor - and/or a CLAUDE-visible note so it's not confusing next time. Record which jobs are armed vs. suspended-with-reason. -- [ ] **VERIFY:** one full cycle observed (or simulated by manual triggers) produces fresh data + a clean - cockpit; the armed cron next-run times are sane. -- **EVAL:** consistent production system running, nothing silently stalled, architecture documented. +## Task 5: Remove the now-redundant compare-measured cron + delete VRP + arm execution + document +**Files:** `infra/k8s/jobs/*.yaml`, memory, progress anchor. +- [ ] DELETE `fxhnt-compare-measured-precompute` cron + its NP (its work is now the Dagster assets). `git rm` the + manifest. (Keep `factory` cron — it's migrate-cockpit + factory-cycle, separate concern; ARM it.) +- [ ] DELETE `fxhnt-vrp-rebalancer` cron + NP + manifest (shelved/falsified; vrp_nav already de-wired). Update + memory (project_fxhnt_diversifier_hunt_2026_07_15, project_fxhnt_vrp_cvar_crash_research): VRP exec cron DELETED. +- [ ] ARM the EXECUTION crons that should run: `factory` (daily), `ucits-volume-ibkr` + `ucits-rebalancer` + (weekday, ucits already armed). Keep SUSPENDED: `bybit-rebalancer` (no creds), `ibkr-rebalancer` (PRIIPs). + Patch live + set suspend in manifests (source of truth). +- [ ] **VERIFY:** cronjobs = {factory, ucits-volume-ibkr, ucits-rebalancer} armed; {bybit, ibkr} suspended; + {compare-measured, vrp} GONE. Dagster owns all ETL/precompute. +- [ ] Document the final architecture in the progress anchor + a CLAUDE-visible note: Dagster (K8sRunLauncher, + per-run memory) = ALL nightly ETL+precompute; K8s CronJobs = execution only. Why exec stays crons + (blast-radius). Update reference_fxhnt_prod_deploy_mechanics with the podman build path (Argo build dead). +- **EVAL:** consistent single-orchestrator ETL, execution isolated in crons, nothing suspended-by-accident, + VRP gone, architecture documented. Done. ## Global constraints -- VERIFY-FIRST, per-task eval. Real-money: only ucits-rebalancer trades (paper) — don't arm bybit/ibkr exec. -- Manifests are the source of truth: every live suspend/delete also changes the committed `infra/k8s/jobs/*.yaml`. -- Don't re-introduce the dagster/cron overlap: Dagster owns ETL/precompute, crons own execution + post-processing. +- VERIFY-FIRST, per-task eval. REAL MONEY: only ucits-rebalancer trades (paper). Never arm bybit/ibkr exec. +- NEVER print the SCW secret key / registry password. podman login via stdin or env, not argv. +- Image: tag with git sha, prove on-cluster BEFORE :latest. Old :latest stays as instant rollback. +- Manifests are source of truth: every live change also edits the committed yaml. +- Don't duplicate logic: assets call the existing import-light helpers; keep them unit-testable. diff --git a/infra/k8s/orchestration/dagster.yaml b/infra/k8s/orchestration/dagster.yaml index ee7ae3d..470bb27 100644 --- a/infra/k8s/orchestration/dagster.yaml +++ b/infra/k8s/orchestration/dagster.yaml @@ -31,7 +31,7 @@ data: load_from: - python_module: module_name: fxhnt.adapters.orchestration.definitions - working_directory: /app + working_directory: /code/src # git-sync target + PYTHONPATH; /app was the baked-image dir (mismatch -> code-server couldn't heartbeat -> 23:30 schedule never fired) --- apiVersion: v1 kind: ConfigMap