# Deterministic Forward State (Sub-project A) — Design

**Date:** 2026-07-08
**Status:** Approved (brainstorming) — ready for implementation plan
**Sub-project:** A of the "Unified data-driven pipeline" vision (foundation layer)

## North-star context (the target the sub-project serves)

The whole system converges on ONE data-driven loop:

```
market data → warehouse (SSOT) → deterministic strategy compute
   → gate/health (policy evaluation) → allocation (capital policy, human-ceilinged)
   → execution engine (orders/fills) → fills back as data → loop
```

Five planes: (1) Data, (2) Compute, (3) **Deterministic state ← THIS sub-project**,
(4) Policy (gate/health/promotion/allocation, declarative), (5) Execution (the coming
trading engine, capital human-gated). The guiding principle: **data-driven mechanics,
human-gated capital** — automate the pipeline up to the point of committing real money;
gate the capital with tested policy + a human ceiling + killswitch. Sub-project A does
NOT touch capital; it makes the mechanics deterministic so B and C can build on solid
ground.

**The problem A solves.** Today each paper-forward tracker accumulates a path-dependent
ledger in a mutable JSON on the PVC (`*_state.json`: `inception`/T0, `last_mark_day`,
`positions`, `equity`, `log`). Operations are hand-driven: re-inception = manually delete
the JSON + delete DB rows + re-persist the backtest ref; a two-store drift exists (JSON =
truth, DB = display). Every operational error observed recently (hollow gates, empty fleet
table, stale display name, the manual BTC-drop re-inception dance) traces to this mutable,
hand-editable state that drifts from what a clean deterministic recompute would produce.

**North-star for A:** system state = `f(committed policy, observed data)`. With no
hand-editable accumulated state, there is nothing to hand-drive or corrupt.

## Goal

Replace the mutable, accumulated tracker state with a **recompute-from-anchor** model:
persist only an immutable anchor `(T0, definition_hash)` per track; recompute the full
forward record over `[T0, today]` from point-in-time warehouse data on every run;
auto-re-inception when the declared definition changes. Idempotent, self-healing, single
source of truth.

## Core mechanism: recompute-from-anchor (chosen over event-sourced ledger / JSON+guards)

The paper strategies are already (or can be refactored to be) **pure functions of the
warehouse data**. So nothing needs to be accumulated: given a frozen `T0` and the current
data, `forward_series(data, T0)` IS the forward record. The anchor is the only persistent
state; everything else is derived and cacheable.

Rejected alternatives:
- **Event-sourced ledger** — append-only, no hand-edits, but still *accumulated* (a
  missed/wrong day stays wrong unless history is rewritten) and doesn't leverage the
  determinism we already have. More machinery for less guarantee.
- **JSON + validation guards** — keeps the mutable state (the root cause); a band-aid that
  conflicts with the single-SSOT / wire-or-delete discipline.

## Components

### 1. The anchor (new SSOT)

A new Postgres table `forward_anchor`:

| column | type | notes |
|---|---|---|
| `strategy_id` | text | part of PK |
| `t0` | date | frozen inception; the OOS clock start |
| `definition_hash` | text | sha256 of the declared definition (see §2) |
| `definition_version` | int | the dev-controlled escape-hatch component |
| `params_json` | jsonb | the return-affecting params that were hashed (audit) |
| `created_at` | timestamptz | when this anchor was set |
| `archived_at` | timestamptz null | set when superseded by a re-inception |

PK is `(strategy_id, created_at)` so re-inception archives the old row and inserts a new
one (full lineage kept, nothing deleted). The **active** anchor for a strategy is the row
with `archived_at IS NULL`. This is the ONLY persistent per-track state. It **retires the
PVC `*_state.json` files** and collapses the two-store drift: `forward_nav` /
`forward_summary` become pure derived caches of the recompute, not an independent store.

### 2. The declared definition + hash

Each tracked strategy declares, in the registry (the existing SSOT), a `definition`:

- the **return-affecting params**: instrument list, `cost_bps`, universe key, sleeve set,
  and any key numeric constants that change the return series;
- a `definition_version: int` — an explicit escape-hatch the dev bumps for pure-code
  changes (e.g., a formula change inside `book_series`) that the params don't capture;
- a `record_mode: "recompute" | "observed"` (see §3b) — a static property of the track.

`definition_hash = sha256(canonical_json(sorted(params)) + str(definition_version))`.

This catches param changes automatically (the BTC-drop — `FUND_INSTRUMENTS` 6→5 — would
have auto-triggered re-inception), never resets on comments/refactors, and keeps the human
in control of "did the return-generating logic change." Erring slightly toward extra
re-inception is the safe direction for real money (a changed strategy must re-prove itself
OOS = false-WAIT-safe).

### 3. The recompute engine (replaces accumulation)

`_run_paper_tracker` changes from *accumulate-and-write* to *recompute-from-anchor*:

1. Read the active anchor for `strategy_id`; compute the current `definition_hash`.
2. **No active anchor** → inception: insert anchor `(T0=today, current hash)`.
3. **Hash changed** → auto-re-inception: set `archived_at` on the old anchor, insert a new
   one `(T0=today, new hash)`.
4. **Hash unchanged** → produce the forward record over `[T0, today]` per the track's
   **record mode** (see §4): `recompute` tracks recompute the full series from PIT data;
   `observed` tracks append today's booked row to their immutable log. Write the derived
   `forward_nav`/`forward_summary` (a cache/projection of the record).

For `recompute` tracks, no accumulated equity/positions/log is written anywhere.

### 3b. Two record modes (recompute + observed)

Not every tracked strategy is recomputable. Each track declares a `record_mode`:

| mode | tracks | how the record is produced |
|---|---|---|
| `recompute` | multistrat, sixtyforty, crypto_tstrend, bybit_4edge, bybit_4edge_levered, positioning, stablecoin, unlock | anchor + PIT data → recompute `[T0, today]` each run (stateless except anchor; self-healing) |
| `observed` | xsfunding, combined | anchor + append-only immutable booked log + minimal carried book-state, all in DB |

**`observed` tracks** book off a live snapshot that has no reproducible full-history series
(e.g. xsfunding books today's live funding snapshot; `persist_backtest_ref=False`). They
cannot be recomputed. So their forward record is a genuinely *observed* series: each run
appends today's booked `(date, ret)` to an append-only, immutable log in the DB, plus the
minimal carried book-state the strategy needs to book the next day (e.g. the current
delta-neutral positions). This removes the mutable, hand-editable PVC JSON (append-only in
DB ≠ hand-editable) but does NOT provide the recompute/self-heal guarantee — which these
tracks fundamentally cannot have. `observed` is the *same* machinery sub-project C reuses
for execution fills (fills are observed events too).

`combined` currently carries a latching kill/re-enter drawdown state; it starts as
`observed` and MAY later move to `recompute` if the kill overlay is refactored to a pure
fold over the recomputed equity path (out of scope for A — a `definition_version` bump when
it happens).

### 4. Determinism contract + the point-in-time (PIT) requirement

Each `recompute`-mode strategy exposes a pure full-history series with no hidden state.
The primitive already exists: `build_strategy().advance(None, {})` returns the full
inception series at the exact basis the forward books (this is what
`_persist_track_backtest_ref` uses). Recompute-from-anchor is therefore
`advance(None, {})` filtered to `date >= T0`. Path-dependent pieces (e.g., a latching
killswitch) stay pure because they are a deterministic function of the — itself
deterministic — return path. `observed`-mode tracks (§3b) are exempt from this contract.

**Critical correctness requirement:** recompute is only stable if *all* return-affecting
inputs are **point-in-time (PIT)** in the warehouse. Crypto (`bybit_features`) already is
(bars are immutable). **Yahoo (multistrat) is not**: adjusted closes are revised
retroactively (dividends/splits), so a live re-fetch would silently shift the record.
Therefore **A includes persisting the Yahoo daily closes as PIT snapshots into the
warehouse**, and the multistrat recompute reads those snapshots — never a live re-fetch.
Without this, the anchor model would only *appear* deterministic while drifting at the data
layer (the very drift we are killing, displaced downward).

### 5. Boundary to sub-project C (fills are out of scope)

Actual execution fills (Alpaca / Bybit paper) are **append-only observed data**, not
recomputable — they belong to the trading-engine sub-project (C). A covers only the
*modeled* forward tracks (the gate's subject). The fills log is untouched by A. This is a
clean seam: modeled tracks are `f(anchor, data)`; fills are observed events.

### 6. Migration (preserve T0 + reconcile) — one-time

For each existing live track:

1. Read the current `*_state.json` inception → seed the anchor `T0`. Compute + store the
   current `definition_hash` and the `record_mode`.
2. **`recompute` tracks:** recompute `[T0, today]` and compare to the accumulated
   `forward_nav` within tolerance.
   - **Match** → seamless: the honestly-accrued clock is preserved (e.g. `bybit_4edge`
     keeps its ~15 days).
   - **Diverge beyond tolerance** → flag it (log + a migration-report row). Divergence *is*
     evidence the old accumulated record was drifting; the deterministic recompute is the
     honest truth and the gate rules on it henceforth. **Never silently overwrite** without
     surfacing the divergence.
3. **`observed` tracks:** import the accumulated booked `(date, ret)` series from the JSON
   into the append-only log verbatim (it is the only record of those observations — it
   cannot be recomputed), and carry the current book-state (positions) into the DB. No
   reconciliation is possible or attempted; the accrued clock is preserved as-is.
4. After migration is verified, retire the `state_reader` JSON path — the DB anchor +
   record are SSOT.

### 7. Self-healing + idempotency

A missed nightly run self-heals (the next run recomputes `[T0, today]` in full). A
corrupted or hand-edited record cannot persist (nothing is accumulated beyond the
immutable anchor). Re-running an asset is byte-identical (deterministic). The manual
BTC-drop dance collapses to: change the declared params → the nightly detects the hash
change → it re-inceptions itself.

## Error handling

- **Insufficient data** for `[T0, today]` → the recompute returns an empty/partial series →
  the gate stays **WAIT** (never fabricates a PASS). Consistent with the real-money rule:
  err toward false-WAIT, never false-PASS.
- **Migration divergence** → flagged in a report row + log, not silently applied.
- **Non-deterministic track** (fails the `forward_series` contract, or a required input is
  not PIT in the warehouse) → migration/build fails loudly. Never silently fall back to
  accumulation.

## Testing

- **Determinism:** recompute twice → byte-identical (property test).
- **Auto-re-inception:** change a declared param → hash changes → new active anchor, old
  archived, T0 resets. Change a comment/refactor → hash unchanged → clock preserved.
- **Migration reconcile:** seed an anchor from a known accumulated record; a matching
  recompute preserves the clock; an injected drift is flagged (not silently overwritten).
- **Self-heal:** skip a run → the next run's record equals the uninterrupted record.
- **Gate honesty:** recompute with insufficient data → WAIT.
- **PIT Yahoo:** a retroactive adjusted-close revision does not change an already-anchored
  record (it reads the PIT snapshot, not a live re-fetch).
- **Observed mode:** an `observed` track appends today's booked row to the immutable log
  and carries its book-state; re-running the same day does not double-append (idempotent
  on `(strategy_id, date)`); a param change re-inceptions and starts a fresh log.
- **Fills boundary:** the fills log is untouched by A's code paths.

## Out of scope (YAGNI)

No event log; no fills / execution (sub-project C); no policy-layer changes (sub-project
B); no new UI beyond surfacing the anchor. Capital is untouched.

## Affected code (initial map — the plan refines this)

- New: `forward_anchor` table + repo methods (`adapters/persistence/forward_nav.py` or a
  new `forward_anchor.py`); `cockpit_models.py` row.
- New: `ForwardDefinition` + hash helper; `definition` entries in `registry.py`.
- New: `forward_record` append-only log table (for `observed`-mode tracks + reused by C)
  keyed `(strategy_id, date)`; repo append/read methods.
- Modify: `_run_paper_tracker` (`adapters/orchestration/assets.py`) → recompute-from-anchor
  for `recompute` tracks; append-to-log for `observed` tracks (branch on `record_mode`).
- Modify: each tracked strategy to expose the pure `forward_series(data, t0)` contract
  (`application/paper_strategies.py`, `domain/strategies/*`).
- New: Yahoo PIT close-snapshot persistence into the warehouse + a PIT-reading client for
  the multistrat recompute.
- New: one-time migration (seed anchors from existing `*_state.json`, reconcile, report).
- Retire: `adapters/persistence/state_reader.py` JSON path once migration is verified.
```
