# Ongoing diversification-aware edge-factory — design

**Date:** 2026-06-14
**Status:** Design approved (research-grounded), ready for implementation plan
**Parent:** the agentic-hedge-fund thesis — the machine that *produces* uncorrelated edges continuously, so the blended Sharpe (and the trajectory) rises over time. Builds on the existing fxhnt factory skeleton (DiscoveryService, FleetOrchestrator, FactoryRuntime, ForwardValidationService, gauntlet, StrategyStore) and the cockpit.

## Problem

We have a research engine that found one real edge and a deployable book. But a single edge is a dead end — its capacity is finite and edges decay. The compounding power of a quant fund is the *factory*: a closed loop that continuously **generates** hypotheses across many domains, **filters** them through a rigorous anti-overfitting gauntlet, admits only those **uncorrelated** to what's already deployed (especially in the left tail), forward-validates them as the only ground truth, **promotes** survivors to capital autonomously, allocates the book by clustering, and **retires** decayed edges — recycling capital. The blended Sharpe ≈ s·√N for N uncorrelated edges, so the objective is *uncorrelatedness*, not raw edge.

## Goals

- A continuous, in-cluster, **autonomous** factory loop: generate → gauntlet → diversification-gate → paper-forward → promote → HRP-allocate → monitor-decay → retire → recycle, visible in the cockpit.
- **Diversification-aware selection + allocation**: a candidate is admitted on its *marginal* contribution and its *left-tail* correlation to the deployed book; the book is allocated by **Hierarchical Risk Parity**, re-estimated rolling.
- A **domain-analyst fleet** (specialist LLM agents) that widens the idea funnel across asset × mechanism × horizon × regime, reasoning over the feature warehouse.
- A **rigorous immune system** so continuous search does not become a false-discovery machine: the Deflated-Sharpe benchmark scales with the honest global trial count (False Strategy Theorem), FDR across the search, forward-only truth.
- Autonomous promotion to capital, bounded by governance (kill-switch, caps).

## Non-goals

- Not a new alpha by itself — it *finds and combines* alphas. The edges come from the hunters + analyst fleet.
- No live order routing here — promotion sets target weights consumed by the **execution layer** (separate, already built).
- Not replacing the gauntlet's statistics — it *feeds them the honest global N* and adds the diversification dimension.
- Deep feature-warehouse streaming of the foxhunt Rust ml-features is a deepening follow-on; v1 analysts reason over the data/features already available.

## Research-grounded design principles (the immune system)

Verified against the literature; these are non-negotiable for a continuous factory:

1. **Deflated-Sharpe benchmark = False Strategy Theorem expected-max** (Bailey & López de Prado 2014). The significance bar `SR*` is not a fixed floor; it is the *expected maximum Sharpe under the null* given `N` trials and the variance of the trial Sharpes: `SR* ≈ √(var(trial SRs)) · E[max of N standard normals]`. As the factory runs, the honest global `N` grows → `SR*` rises automatically. The existing `gauntlet.effective_trials` (N_eff correcting for inter-trial correlation) is exactly the right input; the factory must feed it the cumulative global trial count.
2. **Factor-zoo discipline** (Harvey, Liu & Zhu 2016: "most claimed findings are likely false"; a new factor needs t≈3, not 2). Every hypothesis — grid- AND LLM-generated, including abandoned ones — is counted in `N` and deduped via `SeenCandidate`. The analyst fleet's productivity is a *liability* without this; the bar must scale with the search.
3. **Left-tail correlation, not full-sample** (T. Rowe Price "When Diversification Fails"; Barron's "when markets crash, everything's correlated, even factors"). The diversification gate measures correlation on the deployed book's *worst* days (downside / left tail), where correlations spike. A sleeve uncorrelated full-sample but co-moving in selloffs is rejected.
4. **HRP base, re-estimated rolling** (López de Prado 2016; CFA "static portfolios fail when regimes change"). Hierarchical Risk Parity (no matrix inversion, no concentrated weights) on a rolling correlation estimate; **HERC** (Raffinot 2018, equal-risk-contribution within clusters) is a noted calibration refinement.
5. **Forward is the only ground truth.** Backtest + gauntlet open the door only to paper-forward; only the forward gate (plus the diversification gate) opens the door to capital.

## Architecture (hexagonal; reuses the existing factory skeleton)

```
domain/diversification/                 ← THE CORE (pure, money-grade tested)
  correlation.py    full-sample + LEFT-TAIL (book-worst-decile) correlation between return streams
  hrp.py            HRP: corr → distance → hierarchical clustering → quasi-diagonalization →
                    recursive-bisection weights (+ HERC variant flagged for calibration)
  marginal.py       candidate's marginal contribution to book Sharpe; admit/reject inputs
  gate.py           diversification gate: left-tail ρ < ρ_max AND marginal contribution > floor
domain/gauntlet/                        ← EXISTING; verify/upgrade SR* to FST expected-max(N)
application/factory/
  analysts/         domain specialists (momentum·carry·mean_reversion·microstructure·vrp·seasonality·funding)
  analyst_fleet.py  runs specialists → proposals (Territory/StrategySpec) → dedup vs seen → DiscoveryService
  factory_loop.py   the full daily cycle (see data flow) + autonomous promotion + HRP re-allocation + retirement
  governance.py     kill-switch · gross/per-sleeve caps · max new-deploys per cycle
application/  (EXISTING, reused) DiscoveryService · FleetOrchestrator · ForwardValidationService · StrategyStore
adapters/warehouse/  feature store (EXISTING; analysts read features here)
adapters/persistence/  factory-funnel + book-allocation tables (cockpit DB) for mission-control
adapters/web/  cockpit funnel view (EXISTING cockpit extended)
infra/argo + k8s/  daily in-cluster CronJob runs factory_loop
cli.py:  factory-cycle (--dry-run | --run) · factory-flatten (kill-switch) · factory-status
```

## Data flow (the conveyor belt — daily)

```
analyst fleet proposes hypotheses (asset×mechanism×horizon×regime, from the feature warehouse)
  → dedup vs SeenCandidate · increment the honest GLOBAL trial count
  → DiscoveryService.search → GAUNTLET (DSR with SR*=FST-expected-max(global N) · PBO · FDR · regime-strat)
  → survivors → StrategyStatus.FORWARD (paper conveyor)
  → ForwardValidationService: _mark daily · CUSUM/Page-Hinkley edge-decay monitor
  → forward gate GO?  →  DIVERSIFICATION GATE (left-tail ρ vs DEPLOYED book + marginal contribution)
       both green → AUTONOMOUS PROMOTION → StrategyStatus.DEPLOYED  (governance: caps · kill-switch · max/cycle)
  → HRP RE-ALLOCATION across all deployed sleeves (rolling corr) → target weights → (execution layer consumes)
  → edge-decay monitor on deployed sleeves → decayed → RETIRE → recycle capital
  → persist the whole funnel + allocation + correlation matrix to the cockpit DB (mission-control)
```

## The diversification core

- **Left-tail correlation** (`correlation.py`): given each sleeve's return series and the deployed book's series, compute ρ conditioned on the book's worst-decile days (configurable tail fraction). This is the crisis-relevant correlation. Full-sample ρ is also reported for context.
- **HRP** (`hrp.py`): correlation → distance `d = √(½(1−ρ))` → hierarchical (single/ward) linkage → quasi-diagonalize (recursive tree order) → recursive bisection allocating inverse-variance within/between clusters. Pure: returns matrix → weights. HERC variant (equal risk contribution per cluster) behind a flag.
- **Marginal contribution** (`marginal.py`): ΔSharpe of the book from adding the candidate at its HRP weight (or the closed-form `s·√(1−ρ²)` proxy). The selection objective.
- **Gate** (`gate.py`): admit a forward-survivor to DEPLOYED iff `left_tail_ρ < ρ_max` AND `marginal_contribution > floor`. Pure, table-tested.

## Domain-analyst fleet

A set of specialist analyst agents (one per mechanism family), each: reads the relevant features/markets from the warehouse, proposes a bounded set of `Territory`/`StrategySpec` hypotheses in its specialty (with an empirical prior to avoid absurd parameters), and returns them. `analyst_fleet.py` runs all specialists, dedups vs `SeenCandidate`, increments the global trial count, and hands the proposals to `DiscoveryService.search`. **No special treatment**: LLM proposals clear the same gauntlet. LLM-agnostic via the existing `build_chat_model` (local Ollama / API). A fake/echo analyst is used in tests (no LLM dependency in CI).

## Governance & autonomous promotion

Promotion to capital is autonomous (the forward gate + diversification gate ARE the authority), bounded by `governance.py`:
- **Kill-switch** (`factory-flatten`): halt all new promotions; optionally signal the execution layer to flatten.
- **Caps**: max total deployed sleeves, max gross, max per-sleeve weight, **max new deploys per cycle** (so a burst can't over-commit).
- **Retirement**: a deployed sleeve whose edge-decay monitor fires (sustained, not transient) is demoted out of DEPLOYED; its capital is recycled on the next HRP re-allocation.
- Every promotion/retirement is recorded and shown in the cockpit.

## Cockpit extension (mission-control)

The cockpit gains a **funnel** view: proposed → tested → gauntlet-survived → forward → deployed → retired, the honest global trial count, the deployed book's **correlation matrix + HRP weights**, and per-sleeve edge-decay θ + days-in-forward. So you "just follow" how the factory drives the blended Sharpe.

## Error handling

- An analyst that errors or returns junk is skipped (logged); the fleet continues (one bad specialist ≠ no proposals).
- A candidate with insufficient data for the gauntlet/correlation is skipped, not admitted (fail-safe = don't deploy).
- The diversification gate fails safe: if the left-tail correlation can't be estimated (too few book-tail days), the candidate stays in FORWARD (not promoted).
- HRP on a degenerate/singular correlation falls back to inverse-variance weights (documented), never crashes the cycle.
- Kill-switch or any cap breach halts promotion for the cycle (logged), existing positions untouched.

## Testing (TDD, money-grade)

- Pure core offline: `hrp` (known correlation block-structure → expected clustering + weights; degenerate fallback), `correlation` (constructed tail days → left-tail ρ ≠ full-sample), `marginal` (ΔSharpe sign/magnitude), `gate` (admit/reject boundaries incl. unestimable-tail fail-safe).
- Immune system: the FST `SR*` rises with N (a test that the bar increases as the global trial count grows); FDR over a search.
- `analyst_fleet` with a fake analyst (no LLM): proposals deduped, global count incremented.
- `factory_loop` with a seeded store + fake data: a clearly-uncorrelated survivor promotes; a correlated one does not; a decayed deployed sleeve retires; caps/kill-switch block promotion. Autonomous-promotion governance fully covered.

## Milestones

- **M1 — Diversification core (pure):** `correlation` (left-tail) · `hrp` · `marginal` · `gate`. + verify/upgrade the gauntlet `SR*` to the FST expected-max scaled by the honest global N.
- **M2 — Factory loop:** `factory_loop` + `governance` (autonomous promotion on forward+diversification gates, HRP re-allocation, edge-decay retirement, caps, kill-switch) over the EXISTING grid/agent hunters; persist funnel + allocation to the cockpit DB. CLI `factory-cycle`/`factory-flatten`/`factory-status`.
- **M3 — Domain-analyst fleet:** specialist analyst agents + `analyst_fleet` over the feature warehouse, feeding DiscoveryService through the same gauntlet (fake-analyst tests).
- **M4 — Cockpit funnel view:** funnel + correlation-matrix + HRP-weights + per-sleeve θ in the dashboard.
- **M5 — In-cluster scheduling:** daily Argo CronJob running `factory-cycle`, mirroring the cockpit/exec deploy pattern; cutover.

## Open follow-ups (post-v1)

- HERC / DHRP allocation calibration vs vanilla HRP on realized data.
- Deep feature-warehouse streaming of the foxhunt Rust ml-features (microprice/OFI) → richer analyst substrate (the structural data-moat edges).
- Regime-conditional allocation (allocate differently per detected regime).
- Capacity-aware promotion (don't deploy an edge whose capacity is below the capital it would receive).
