From 0a126754ebb339a0608b5676e5e9ca6aae8881bf Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Fri, 10 Jul 2026 23:57:12 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20spec=20=E2=80=94=20paper-validation=20e?= =?UTF-8?q?nvelope=20+=20real=20exec-venue=20recording=20(C3)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 --- .gitignore | 4 + ...alidation-envelope-exec-venue-c3-design.md | 207 ++++++++++++++++++ 2 files changed, 211 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-10-paper-validation-envelope-exec-venue-c3-design.md diff --git a/.gitignore b/.gitignore index d717b21..8eb115c 100644 --- a/.gitignore +++ b/.gitignore @@ -24,3 +24,7 @@ dist/ # downloaded 1m klines (binance.vision) + the worst_basis side-store — data, never git klines_cache/ *.parquet + +# local paper-trading API keys (never commit) +alpacca-paper-keys.txt +*-paper-keys.txt diff --git a/docs/superpowers/specs/2026-07-10-paper-validation-envelope-exec-venue-c3-design.md b/docs/superpowers/specs/2026-07-10-paper-validation-envelope-exec-venue-c3-design.md new file mode 100644 index 0000000..9212290 --- /dev/null +++ b/docs/superpowers/specs/2026-07-10-paper-validation-envelope-exec-venue-c3-design.md @@ -0,0 +1,207 @@ +# C3 — Paper-Validation Execution Envelope + Real Exec-Venue Recording + +**Date:** 2026-07-10 +**Status:** Approved (design) +**Depends on:** A (deterministic forward state), B2a (allocation), B3 (funding), C1 (Alpaca exec leg), C2 (Bybit exec leg) — all merged. + +## Problem + +Two gaps surfaced when the first executed multistrat run landed on the cockpit: + +1. **The exec track cannot run continuously.** The paper execution book (`multistrat_exec`, + `bybit_4edge_exec`) is funded by B2a's real-capital allocation + (`envelope = min(current_allocation[sid], nlv)`). B2a correctly sets `multistrat = $0` + (edge has <60 forward days), so on the next `--execute` the book books one real return day + and then **flattens** (`envelope = min($0, nlv) = $0`). We cannot validate execution over + time because the deterministic real-capital policy — correctly — refuses to fund an immature + edge. Paper execution *validation* is a different concern from real-capital *allocation*, but + they currently share one envelope. + +2. **The executed-reality badge lies about the venue.** The cockpit's "Executed reality" block + renders `d.venue` — the sleeve's *configured* venue (`alpaca-paper`). The first run actually + executed on **IBKR paper** (Alpaca keys were unavailable). Since IBKR whole-share and Alpaca + fractional have very different execution friction, and the divergence readout exists precisely + to surface that friction, the badge must reflect **where the fills actually happened**. + +## Goals + +- A **separate paper-validation funding envelope** for exec tracks that is NOT gated by B2a's + 60-day / $35k real-capital policy, so the paper book runs continuously — while B2a continues to + govern **real capital** unchanged. +- Record the **actual execution venue** (`alpaca-paper`, `ibkr-paper`, `bybit-testnet`) on each + exec track and surface it in the cockpit badge, falling back to the configured venue when no + executed run has been recorded yet. +- Arm the continuous Alpaca paper experiment. + +## Non-Goals / Hard Constraints + +- **Real money errs false-WAIT.** The paper envelope MUST be structurally impossible to apply + against a live/real account. It never reads or writes B2a's `strategy_allocation` table, so the + real-capital policy is untouched by construction. +- No new execution engine, no new return store, no schema migration (venue rides in existing JSONB + book-state). +- Out of scope: the deferred crypto re-entry-cost accounting noted in `bybit_exec_record.py` + (still deferred; not this sub-project despite the "C3" reference in that comment). + +## Design + +### Component 1 — Paper-validation envelope + +**Config (single source of the amount):** add `PaperValidationSettings` to `config.py` +(env prefix `FXHNT_PAPER_VALIDATION_`), one field: + +```python +class PaperValidationSettings(BaseSettings): + model_config = SettingsConfigDict(env_prefix="FXHNT_PAPER_VALIDATION_") + envelope: float = 0.0 # 0.0 = OFF (real-capital B2a behavior everywhere). >0 = paper $ per exec track. +``` + +Exposed as `settings.paper_validation`. + +**multistrat (`execute-multistrat`, CLI):** new option +`--paper-envelope ` (default `0.0`). Resolution: +`paper_envelope = paper_envelope_flag if paper_envelope_flag > 0 else settings.paper_validation.envelope`. +The CronJob passes `--paper-envelope 1000000` explicitly (self-documenting). + +**bybit (`bybit_testnet_reconcile`, Dagster asset — no CLI):** reads +`settings.paper_validation.envelope` directly (assets have no CLI flag surface). + +**Envelope selection** — `plan_and_record` (`multistrat_exec_record.py`) and the bybit asset +(`assets.py:1208`, via a new `crypto_envelope` param or a sibling helper): + +``` +if paper_envelope > 0: # paper-validation mode — bypasses B2a entirely + if not account_is_paper: # SAFETY GATE (see below) + REFUSE: raise / block, place NO orders + envelope = min(paper_envelope, nlv) # nlv = alpaca NLV / bybit testnet equity +else: # real-capital mode — UNCHANGED + envelope = min(current_allocation[sid], nlv) +``` + +In paper mode `current_allocation` is never read → B2a real-capital policy is structurally +untouched. + +**Safety gate (the load-bearing invariant):** paper mode requires a paper/testnet account. +`AccountState.is_paper` is already populated correctly by every adapter: +- `AlpacaBroker`: `is_paper = "paper-api" in base_url` +- `IbkrBroker`: `is_paper = account.startswith("DU")` +- `BybitBroker`: testnet → paper. + +The exec path already reads `account_state()` for NLV, so the gate reuses that read. If +`paper_envelope > 0` and `account_state().is_paper` is False (or an `allow_live` escape is set), +the command/asset **hard-refuses**: raises before any order is placed, exits non-zero (CLI) / +fails the asset (Dagster). A live account can never be sized by the paper envelope. + +For multistrat this gate lives in `plan_and_record` (it already has `nlv`/account context via the +caller); the caller passes `account_is_paper` alongside `nlv`. For bybit the asset checks testnet +before calling the envelope helper. + +### Component 2 — Real exec-venue recording + +**Derive** the venue string at the execution site from the broker: +`venue = f"{broker.name}-{'paper' if account_state().is_paper else 'live'}"` +→ `alpaca-paper`, `ibkr-paper`. Bybit's asset records the literal `bybit-testnet` +(its broker runs testnet-only). + +**Persist** it in the existing exec-position book-state (JSONB, no migration): +- `build_pos_state(positions, prices, envelope, venue)` adds `"venue": str(venue)`. +- `build_crypto_pos_state(positions, marks, envelope, funding_cursor_ts, venue)` adds + `"venue": str(venue)`. + +`record_exec_track` / `record_bybit_exec_track` gain a `venue: str` parameter, threaded from the +CLI/asset (which hold the live broker). The venue is written on **every** exec run, including the +inception run (the book-state is written even when no return is booked), so the badge is correct +from the first run. + +### Component 3 — Cockpit surfacing + +- `DashboardService._exec_pair(sid, summaries)` additionally reads + `self._repo.get_book_state(f"{twin}_pos")` and extracts `.get("venue")` → + new field `exec_venue: str | None` on the returned dict. +- `ports/dashboard.py`: `StrategyDetail` gains `exec_venue: str | None = None`. + (FleetRow does not need it — the badge only appears on the detail page.) +- `strategy.html` exec-tag: render `{{ d.exec_venue or d.venue }}` — recorded venue wins, configured + venue is the fallback when no executed run exists yet. + +### Component 4 — Wiring & deploy + +- `infra/k8s/jobs/fxhnt-alpaca-rebalancer.yaml`: append `--paper-envelope 1000000` to the + `execute-multistrat --execute` args; flip `suspend: false` to arm the continuous Alpaca paper + experiment. (`alpaca-credentials` secret is already populated with paper keys.) +- **Revert the `$1M` `strategy_allocation[multistrat]` override** in prod Postgres so the cockpit + ALLOCATION shows the true B2a state (`$0`, building). The paper experiment funds via the flag, + not the override → the "$1M silently reverts" confusion is resolved: policy display is honest, + experiment is flag-driven. +- **Cancel the queued IBKR paper orders** on DU9600528 (the Monday-queued ETF rebalance + IBIT + flatten) via a one-off `ib_async` job, so the experiment lives solely on Alpaca. + +## Data Flow + +``` +CronJob (alpaca-rebalancer, --paper-envelope 1000000) + → execute-multistrat + → AlpacaBroker.account_state() → nlv, is_paper=True + → plan_and_record(paper_envelope=1e6, account_is_paper=True, ...) + → is_paper True ✓ → envelope = min(1e6, nlv) [B2a NOT read] + → exec_svc.rebalance_weights(...) places fills + → compute_exec_return(prior_book) → observed return + → record_exec_track(..., venue="alpaca-paper") + → run_track("multistrat_exec", return) [A's forward store] + → set_book_state("multistrat_exec_pos", {positions, prices, envelope, venue}) + +Cockpit (dashboard.fxhnt.ai/strategy/multistrat) + → DashboardService.detail("multistrat") + → _exec_pair → exec_return_pct, exec_divergence_pct, exec_venue="alpaca-paper" + → strategy.html badge renders "alpaca-paper" +``` + +## Error Handling + +- **Paper envelope on a live/non-paper account** → hard refuse before any order; non-zero exit / + asset failure. Test explicitly. +- **`paper_envelope = 0`** → identical behavior to today (B2a-gated). Regression-tested. +- **Missing `venue` in an older book-state** (pre-C3 anchor, e.g. today's IBKR inception) → + `_exec_pair` `.get("venue")` returns `None` → badge falls back to configured `d.venue`. No crash. +- **Paper envelope > NLV** → `min(paper_envelope, nlv)` clamps to NLV (cannot deploy more than the + account holds). Existing behavior, retained. + +## Testing + +Unit (pytest): +- `plan_and_record`: `paper_envelope>0` + `is_paper=True` → envelope = `min(paper, nlv)`, + `current_allocation` NOT consulted. +- `plan_and_record`: `paper_envelope>0` + `is_paper=False` → refuses, no orders placed. +- `plan_and_record`: `paper_envelope=0` → envelope = `min(alloc, nlv)` (unchanged). +- bybit envelope helper: same three cases against testnet equity. +- venue string derivation: `alpaca`+paper → `alpaca-paper`; `ibkr`+paper → `ibkr-paper`. +- `build_pos_state` / `build_crypto_pos_state` persist `venue`. +- `_exec_pair`: reads `venue` from book-state → `exec_venue`; missing venue → `None` (fallback). + +Local cockpit verify (per `feedback_verify_cockpit_locally_not_prod`): +- Seed a `multistrat_exec` book-state with `venue="alpaca-paper"` and a `bybit_4edge_exec` with + `venue="bybit-testnet"` in the local dev DB. +- `scripts/dev-cockpit-local.sh`, drive with Playwright, screenshot `/strategy/multistrat` and + `/strategy/bybit_4edge`, **read** that each badge shows the real venue. + +## File Touchpoints + +| File | Change | +|---|---| +| `src/fxhnt/config.py` | `PaperValidationSettings` + `settings.paper_validation` | +| `src/fxhnt/application/multistrat_exec_record.py` | paper-envelope branch + safety gate in `plan_and_record`; `venue` in `build_pos_state` + `record_exec_track` | +| `src/fxhnt/application/bybit_exec_record.py` | paper-envelope helper + `venue` in `build_crypto_pos_state` + `record_bybit_exec_track` | +| `src/fxhnt/adapters/orchestration/assets.py` | bybit asset reads `settings.paper_validation.envelope`, testnet gate, passes `venue="bybit-testnet"` | +| `src/fxhnt/cli.py` | `execute-multistrat --paper-envelope` flag + threading `account_is_paper`/venue | +| `src/fxhnt/application/dashboard_service.py` | `_exec_pair` reads book-state `venue` → `exec_venue` | +| `src/fxhnt/ports/dashboard.py` | `StrategyDetail.exec_venue` | +| `src/fxhnt/adapters/web/templates/strategy.html` | exec-tag → `{{ d.exec_venue or d.venue }}` | +| `infra/k8s/jobs/fxhnt-alpaca-rebalancer.yaml` | `--paper-envelope 1000000`, `suspend: false` | + +## Deploy Sequence (post-merge) + +1. Push master → `argo-deploy-cockpit.sh` (build + deploy). +2. `migrate-forward-anchors` — not required (no schema change), but harmless. +3. Revert `strategy_allocation[multistrat]` override → `$0` (or let the nightly do it). +4. Cancel queued IBKR paper orders on DU9600528. +5. Arm `fxhnt-alpaca-rebalancer` (`suspend: false`). +6. Verify cockpit LOCALLY, then confirm on `dashboard.fxhnt.ai`.