gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(sp15): trader discipline and recovery design spec

Browse Source 
Design spec for SP15 — addresses the train-dd4xl downward spiral
post-mortem (8 epochs, trades 131k→64k, sharpe 79→44 with monotone
descent across active_frac, dir_entropy) and the walk-forward audit
finding (test_start..test_end slice generated but never consumed,
val IS the selection set, no sealed test).

Six phases over ~13-21 days:
- Phase 0: SP14 EGF ISV-driven retune (gate1=closed forever fix)
- Phase 1: honest numbers — unified sharpe kernel, cost-net (commission
   + spread + OFI-impact, dev/prod parity), drawdown reporting,
   8 counterfactual baselines, dd_pct as foundational state input,
   --holdout-quarters/--dev-quarters CLI, consume abandoned test slice
- Phase 2: 21 behavioral tests on dev RTX 3050 Ti, pre-commit hook
   gates argo-train.sh
- Phase 3: 5 trader teachings (r_quality/r_discipline split, explicit
   cost, quadratic DD, regret, confidence-aware Hold floor)
- Phase 3.5: 4 recovery mechanisms (asymmetric reward under DD,
   cooldown gate, plasticity injection, recovery curriculum in PER)
- Phase 4: L40S walk-forward Q1-Q7, Q8 final dev, sealed Q9 OOS

Cross-cutting discipline rule: no new pearl/controller/kernel/ISV
slot/reward term ships without a Phase 2 behavioral test that proves
the intended trader behavior. Hard rule, no exceptions.

Decisions captured from 7 clarifying questions answered 2026-05-06.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-06 08:32:37 +02:00
parent 0275a25d9f
commit fb5f2394d0
1 changed files with 684 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

684
docs/superpowers/specs/2026-05-06-trader-discipline-and-recovery-design.md Normal file
View File

@@ -0,0 +1,684 @@
# SP15: Trader Discipline and Recovery — Design
> **Status**: spec
> **Date**: 2026-05-06
> **Predecessors**: SP11/12/13/14 chain merged into main at `0275a25d9` (2026-05-06)
> **Implementation branch**: `sp15-trader-discipline-recovery` (forks from main post-spec-commit)
> **Authoritative path**: `docs/superpowers/specs/2026-05-06-trader-discipline-and-recovery-design.md`
## TL;DR
Take the system from "stable, gradient-flowing, but trades like a panicked toddler" to "disciplined trader with measurable competence on a sealed quarter." Six phases:
1. **Phase 0** — Retune SP14 EGF (gate1=closed forever) so its anchors are ISV-driven, not hardcoded.
2. **Phase 1** — Honest numbers: unified sharpe kernel, cost-net sharpe (commission + spread + OFI-impact, dev/prod parity), drawdown reporting, 8 counterfactual baselines, `dd_pct` as foundational state input, `--holdout-quarters` + `--dev-quarters` CLI flags, consume abandoned walk-forward test slice.
3. **Phase 2** — 21-test behavioral suite on dev RTX 3050 Ti. Pre-commit hook blocks `argo-train.sh` if any test fails. CI gate before L40S.
4. **Phase 3** — 5 trader teachings (reward + action selection): r_quality/r_discipline split, explicit cost, quadratic DD penalty, regret signal, confidence-aware Hold floor.
5. **Phase 3.5** — 4 recovery mechanisms (3.5.1 already in Phase 1): asymmetric reward under DD, cooldown gate, plasticity injection, recovery curriculum in PER.
6. **Phase 4** — L40S walk-forward on Q1-Q7, Q8 final dev, sealed Q9 OOS — produces report card with deltas vs all 8 baselines.
**Cross-cutting discipline rule**: no new pearl, controller, kernel, ISV slot, or reward term ships without a Phase 2 behavioral test that proves the intended trader behavior. Hard rule alongside `feedback_no_partial_refactor`.
**Estimated scope**: ~4900 LOC, 13-21 days serial / 10-15 days with parallel dispatch (Approach B).
---
## 1. Premise
> "we know it's stable, we know it learns, but it has behavioural issues as well as mechanical issues. The model doesn't understand its role as trader. Drives itself into a downward spiral, very hard to recover from."
>
> — User, 2026-05-06, post-`train-dd4xl` termination
The architecture works. Gradients flow. Smoke tests pass. But:
- Each iteration we identify a gaming behaviour and add an adaptive controller. Controllers control controllers. We end up with a 50-controller toddler, not a trader.
- The model has no architectural mechanism to escape its own losses. Once it starts losing, it learns "trade less" — which reduces gradient signal — which slows recovery — which compounds.
- We measure on the same window we tune controllers against. "Out-of-sample" sharpe on val is a fantasy: val IS the selection set.
This spec is the alternative path: fix measurement, validate behaviour locally, teach role explicitly.
---
## 2. Failure evidence
### 2.1 The spiral curve (`train-dd4xl`, 8 epochs, terminated 2026-05-06)
Commit `c0fc28e45` (full SP14 Layer A+B chain, bus fix, warmup gate delete) on L40S walk-forward:
```
trades active sharpe pf dir_ent
ep1: 131,805 0.482 79.45 1.195 1.03
ep2: 127,463 0.446 70.46 1.197 1.00
ep3: 116,817 0.381 65.87 1.214 0.93
ep4: 98,835 0.296 69.52 1.252 0.81
ep5: 98,781 0.295 63.92 1.279 0.81
ep6: 91,883 0.268 60.40 1.293 0.76
ep8: 63,837 0.171 43.63 1.225 0.57 ← terminated
```
Monotone descent across `trade_count` (−51%), `active_frac` (−65%), `dir_entropy` (−45%). Profit factor briefly **rises** (1.195 → 1.293) — fake win, surviving trades are higher-confidence — then drops as the policy collapses to non-trading.
### 2.2 HEALTH_DIAG[8] terminal state
```
val_dir_dist [short=0.07 hold=0.69 long=0.10 flat=0.14] ← 69% Hold
mag_stats wr q=0.429 h=0.412 f=0.414 ← all ~ random
mag_stats var q=4.6e-10 h=4.2e-9 f=1.1e-8 ← magnitudes flat
eval_dist eq=0.885 eh=0.115 ef=0.000 ← 88.5% Quarter only
c51_budget mag 0.0004 ← magnitude over-suppressed
aux_dir_acc short=0.518 long=0.487 ← 50/50 = no signal
aux_moe util one expert at 1.000, rest at 0 ← MoE collapse
improvement_z -0.850 ← controller says "worse"
gate1 closed ← EGF never opens
sp9_kelly floor=0.0000 ← Kelly literally cannot trade
```
The model isn't learning to trade. It's learning to look stable on the window it tunes against, while collapsing to "Hold most bars, pick Quarter mag, ignore direction."
### 2.3 Walk-forward audit finding
`crates/ml/src/cuda_pipeline/gpu_walk_forward.rs:122-165` defines proper expanding-window walk-forward with `train_start..train_end / val_start..val_end / test_start..test_end` per fold.
`crates/ml/src/trainers/dqn/trainer/mod.rs:1155` consumes only `train` and `val`:
```rust
self.set_training_range(fold.train_start, fold.train_end, fold.val_start, fold.val_end);
self.set_val_data_from_slices(&features[fold.val_start..fold.val_end], ...)
```
`grep -rn "fold\.test_start"` outside the generator returns **zero hits**. The reported `sharpe_annualised=64-79` is on the **val window** — same window the early-stop callback, Kelly warmup cap, and controller anchors all read.
What this means in practice:
| Signal | Sees val window? |
|---|---|
| Parameter gradients | No (train only) ✅ |
| Early-stop callback | **Yes** ❌ |
| Kelly warmup cap | **Yes** ❌ |
| Controller anchors / EMAs | **Yes** ❌ |
| Reported "OOS" sharpe | **Yes** (it's the same window) ❌ |
Val is temporally clean for parameter learning, but it's the selection set, not a sealed test set. Sharpe=80 reported means "best on selection set we tuned against," not "robust on unseen quarter."
---
## 3. Decisions reference (clarifying questions, 2026-05-06)
| # | Question | Decision |
|---|---|---|
| Q1 | DD-state architectural placement | **Foundational (Phase 1)**: ISV slot + producer kernel + trunk concat. Layout fingerprint break OK (greenfield). Eval-time policy SEES dd context. |
| Q2 | SP14 EGF cut vs retune | **Keep, retune to ISV-driven**. No hardcoded thresholds; Schmitt thresholds, variance gate refs, steepness anchors all become ISV-tracked. |
| Q3 | Cost model dev/prod parity | **commission + spread + OFI-impact, everywhere**. Dev synthetic markets generate synthetic spread + OFI. Prod reads from MBP10 fxcache. Same kernel. |
| Q4 | Counterfactual baselines | **All 8**: buyhold, hold-only, random_dir + same Kelly, naive momentum, aux-only, mag-Quarter-fixed, trail-only, naive reversion. |
| Q5 | Phase 2 test set | **All 21 tests**: 7 trading behavior + 10 state/arch grounding + 4 recovery dynamics. |
| Q6 | OOS framework | **Train/dev/test split**: Q1-Q7 walk-forward train, Q8 final dev, Q9 sealed final test. |
| Q7 | Phase 3.5.4 plasticity injection | **Land all 4 recovery mechanisms** including plasticity. No defer. |
---
## 4. Architecture overview
### 4.1 Phase chain
```
Phase 0: SP14 EGF ISV-driven retune ─┐
Phase 1: Honest numbers + dd_pct foundation ─┼─ parallel sub-worktrees (Approach B)
Phase 2A: behavioral test scaffolding ─┘
│
▼ (Phase 2B individual tests land as Phase 1 capabilities ship)
Phase 2B: Groups 1+2 (17 tests, gates Phase 3 entry)
│
▼
Phase 3: 5 trader teachings (atomic per teaching, each gated on anchor 2.X test)
│
▼
Phase 3.5: 4 recovery mechanisms (3.5.1 already in Phase 1.5)
│ ↑ Phase 2C (Group 3 tests) interleaved with each teaching
▼
Phase 4: L40S walk-forward (Q1-Q7) + Q8 final dev + sealed Q9 OOS
```
### 4.2 Branch and worktree structure
- **Branch**: `sp15-trader-discipline-recovery` forks from main (`0275a25d9` post-SP11-merge)
- **Sub-worktrees** for parallel dispatch:
- `sp15-phase0-egf-retune` — SP14 EGF retune (Phase 0)
- `sp15-phase1-honest-numbers` — Phase 1 sub-fixes (parallel to Phase 0)
- `sp15-phase2a-test-scaffold` — synthetic market generators + harness (parallel to Phase 0+1)
- **Merge model**: each phase merges back to `sp15-trader-discipline-recovery` atomically. SP15 branch merges to main only after Phase 4 production-track gate passes.
### 4.3 Cross-cutting discipline rule
**No new pearl, controller, kernel, ISV slot, or reward term ships without:**
1. **Phase 2 behavioral test** that proves the intended trader behavior (the test commit and the implementation commit ship together).
2. **Wire-up audit entry** in `docs/dqn-wire-up-audit.md`.
3. **Memory pearl** in `~/.claude/projects/-home-jgrusewski-Work-foxhunt/memory/` if the technique generalises beyond this spec.
This adds to `feedback_no_partial_refactor`, `feedback_no_stubs`, `feedback_isv_for_adaptive_bounds`, `feedback_wire_everything_up` as a hard discipline rule. Violations mean the change is reverted, not merged-with-followup.
---
## 5. Phase 0 — SP14 EGF ISV-Driven Retune (~1-2 days, atomic per fix)
### 5.1 Goal
Make EGF actually fire under target conditions. Eliminate every hardcoded threshold per `feedback_isv_for_adaptive_bounds`.
### 5.2 Phase 0.A — Diagnose
`HEALTH_DIAG[7]/[8]` from `train-dd4xl` showed:
- `var_aux=0.00021` against threshold `0.01` → 50× below trip point
- `var_q=0.00879` against threshold `0.05` → 6× below trip point
- `α_smoothed=0.0008`, `α_raw=0.0006` → mathematically consistent with closed gate
Either the thresholds are wrong (signal magnitudes are inherently smaller than expected when calibrated on synthetic-distribution training) or the signals being read into `var_aux`/`var_q` aren't the right ones (e.g., reading post-clipped values that have been collapsed). Diagnostic step:
1. Add temporary HEALTH_DIAG emission of the raw aux/Q variance signals **before** EGF reads them.
2. Run 5-epoch L40S smoke against post-Phase-0.A HEAD.
3. Compare emitted variances to the EGF-internal `var_aux`/`var_q` values. Mismatch → upstream signal corruption. Match → thresholds need ISV-driven dynamic anchoring.
### 5.3 Phase 0.B — Retune
Every hardcoded anchor becomes ISV-driven:
| Anchor (current value) | New ISV slot | Producer |
|---|---|---|
| Schmitt thresholds (±0.03 hardcoded) | `EGF_SCHMITT_HI_INDEX`, `EGF_SCHMITT_LO_INDEX` | rolling p99/p1 of α_raw over fold |
| Variance gate ref aux (0.01) | `EGF_VAR_AUX_REF_INDEX` | running mean of var_aux over fold |
| Variance gate ref Q (0.05) | `EGF_VAR_Q_REF_INDEX` | running mean of var_q over fold |
| Steepness normaliser (k_aux=20/(1+var_aux/0.01) hardcoded `0.01`) | reuses `EGF_VAR_AUX_REF_INDEX` | (above) |
| Steepness normaliser k_q (0.05 hardcoded) | reuses `EGF_VAR_Q_REF_INDEX` | (above) |
Total: 4 new ISV slots + 2 new producer kernels. EGF kernels read these instead of constants.
### 5.4 Phase 0 anchor test (Phase 2.21 — `egf_gate_opens_under_disagreement`)
Synthetic batch with deliberately disagreeing aux head (output = +1.0) vs Q head (output = −1.0) → gate1 must open within K=100 steps. If it doesn't, EGF is broken regardless of how clean the kernels look.
This test is authored alongside Phase 0.B and gates the commit. EGF retune cannot land without test 2.21 going green.
---
## 6. Phase 1 — Honest Numbers (~2-3 days, 7 atomic sub-fixes)
### 6.1 Sub 1.1 — Unified sharpe kernel
Single GPU kernel `compute_sharpe_per_bar_kernel.cu`. Inputs: `pnl_array[T]`. Output: `(mean_pnl, std_pnl, sharpe_per_bar)`. Train and val use identical formula and identical window definition.
Eliminates `sharpe_ema` (per-batch EMA with smoothing constant divergence) vs `sharpe_annualised` (val × sqrt(525,600)) confusion. Reports both `sharpe_per_bar` and `sharpe_annualised = sharpe_per_bar × sqrt(N_bars_per_year)` from the same kernel.
### 6.2 Sub 1.2 — Cost-net sharpe
`cost_t = commission + spread_t × position_size + ofi_impact(spread_t, ofi_t)`
- `commission`: $1.50/RT default, `--commission` CLI overridable
- `spread_t`: real bid-ask from MBP10 fxcache (prod) OR synthetic spread from market generator (dev)
- `ofi_impact()`: linear-impact model `λ × |position_size| × |ofi_t|` where λ is ISV-tracked impact coefficient (initial: empirical fit from MES historical)
- `position_size`: contracts traded (1 for Quarter, 2 for Half, 4 for Full at base Kelly cap)
`sharpe_net = (mean_pnl − cost_per_bar_avg) / std_pnl_with_costs`
The cost kernel reads `spread_t` and `ofi_t` as input arrays. Dev synthetic markets generate these as part of fixtures. This is the same kernel in both environments — `feedback_isv_for_adaptive_bounds` extends to costs.
### 6.3 Sub 1.3 — Drawdown reporting
New ISV slots:
- `DD_CURRENT_INDEX`: peak-to-trough percentage of running cumulative PnL
- `DD_MAX_INDEX`: maximum DD seen this fold
- `DD_RECOVERY_BARS_INDEX`: bars since last DD peak (0 = currently at ATH)
- `DD_PERSISTENCE_INDEX`: bars in current drawdown ≥ threshold (used by 3.5.4)
HEALTH_DIAG emits `max_dd_pct`, `current_dd`, `dd_recovery_bars`, `Calmar = mean_pnl / max(max_dd_pct, 1e-4)`. Denominator floor (1e-4) replaces the current Calmar=100 saturation artifact.
### 6.4 Sub 1.4 — 8 counterfactual baselines
Each baseline is a pure CUDA kernel that replays the val/test window with a fixed action policy and computes sharpe_net + max_dd + trade_count. All 8 run in parallel kernels per fold per epoch:
| Baseline | Definition |
|---|---|
| `buyhold` | always Long, mag = Quarter (smallest), no exits |
| `hold_only` | always Hold, never enter |
| `random_dir_kelly` | random direction at random bars (seeded), mag pinned to policy's Kelly cap output |
| `naive_momentum` | direction = sign(last bar return), mag = Quarter |
| `aux_only` | direction = sign(aux head's next-bar prediction), mag = Quarter |
| `mag_quarter_fixed` | policy's direction, mag pinned to Quarter |
| `trail_only` | random entries (seeded), exits via policy's trail-stop logic |
| `naive_reversion` | direction = -sign(last bar return), mag = Quarter |
HEALTH_DIAG emits per-fold deltas: `vs_buyhold = sharpe_net − buyhold_sharpe_net`, etc.
### 6.5 Sub 1.5 — `dd_pct` as foundational state input
This is the layout fingerprint break.
- New ISV slot: `DD_PCT_INDEX = current_dd / max(dd_budget, 1e-4)`. `dd_budget` is config (e.g., 0.20 = 20% drawdown limit). Producer kernel reads `DD_CURRENT_INDEX` from sub 1.3.
- **Trunk concat extension**: trunk forward input grows by 1 dim. Layout fingerprint seed extended with `DD_PCT_TRUNK_INPUT`. All trunk-input consumers migrate atomically (per `feedback_no_partial_refactor`): batched_forward.rs, batched_backward.rs, gpu_dqn_trainer scratch buffer sizing, eval-mode forward path.
Eval-time policy now SEES dd_pct on every forward. Phase 3 teachings condition on it naturally.
### 6.6 Sub 1.6 — `--holdout-quarters` and `--dev-quarters` CLI flags
In `bin/fxt/` and `services/ml-training/`:
- `--holdout-quarters N` (default 1): slice last N quarters off `training_data` BEFORE walk-forward generation. This is the sealed final test set.
- `--dev-quarters M` (default 1): slice next-N-from-tail quarters off, after holdout. This is the final dev set (used for hyperparameter tuning, controller anchor calibration).
- Walk-forward generates folds on `Q1..Q(9-N-M)`. With defaults: walk-forward on Q1-Q7 (3 folds at 0.5/0.125/0.125/0.125), Q8 = final dev, Q9 = sealed test.
### 6.7 Sub 1.7 — Consume the abandoned test slice
`set_test_data_from_slices` method on trainer + post-fold `test_sharpe_net` emit. The 12.5% `test_start..test_end` slice generated by walk-forward is now actually used: per-fold test slice metric reported separately from val. Sharpe gap between val and per-fold-test surfaces the selection-vs-test difference quantitatively for each fold.
### 6.8 Phase 1 sequencing
Sub 1.1 → 1.2 → 1.3 → 1.4 → 1.5 → 1.6 → 1.7. Each is one atomic commit. 1.5 (dd_pct) is the layout-fingerprint-break commit; explicitly tagged in the commit message so checkpoints from before this point are known-incompatible.
---
## 7. Phase 2 — Behavioral Test Suite (~3-5 days, ~2000 LOC)
### 7.1 Goal
Every behavior the policy must exhibit gets a deterministic test on dev RTX 3050 Ti. Suite runs <10 min total. Pre-commit hook blocks `argo-train.sh` if any test fails. CI gate before L40S.
### 7.2 Phase 2A — Test infrastructure (parallel to Phase 0+1)
- **Synthetic market generators** in `crates/ml/tests/behavioral/synthetic_markets.rs`:
- `flat_market(noise_σ)` — flat price + Gaussian noise + parameterized bid-ask spread
- `drift_market(μ, σ, spread_pattern)` — geometric drift + spread parameterization
- `OU_market(θ, σ_eq, μ)` — Ornstein-Uhlenbeck process for mean-reversion
- `regime_switch_market(N_regimes, transition_matrix)` — multi-regime Markov chain
- All generators produce `(price_t, spread_t, ofi_t)` triples — feed unified cost kernel from Phase 1.2
- **Oracle policies** — known-correct action sequences per market type. Used both as reference for test assertions and as initial-weight seeds for "policy starts informed" tests.
- **Evaluator harness** in `crates/ml/tests/behavioral/harness.rs`: runs policy on synthetic for K bars, captures action distribution + sharpe_net + max_dd + behavioral metrics, compares against test assertions.
- **Pre-commit hook** in `.claude/helpers/pre_commit_behavioral_suite.sh`: runs `cargo test -p ml --test behavioral_suite -- --nocapture`. Fail → block commit.
- **CI integration**: `argo-train.sh` invokes the same hook before submitting workflow. Local fail = no L40S deploy.
### 7.3 21 tests by group
#### Group 1: Trading behavior (7 tests, lands as Phase 1 cost + market gen ships)
| # | Test | Pass criterion |
|---|---|---|
| 2.1 | `flat_market_holds` | ≥80% Hold over last 5k bars on flat synthetic |
| 2.2 | `trend_market_directional` | drift>0 → ≥70% Long; drift<0 → ≥70% Short |
| 2.3 | `mean_revert_reverses` | OU process → reverses at ±2σ within 5 bars |
| 2.4 | `cost_sensitivity` | trade freq drops ≥40% at high spread vs low spread |
| 2.6 | `regime_silences` | high-noise no-edge regime → trade freq ≤20% |
| 2.7 | `stop_discipline` | adverse excursion past trail dist → close within 1 bar |
| 2.18 | `action_latency_test` | step-function +5σ signal → policy switches Hold→Long within 5 bars |
#### Group 2: State/architecture grounding (10 tests, lands as Phase 1 state + Phase 0 EGF ships)
| # | Test | Pass criterion |
|---|---|---|
| 2.8 | `hold_vs_flat_semantics` | Hold preserves pre_pos; Flat zeros it |
| 2.9 | `eval_fold_isolation` | KL(shuffled-future, unshuffled-future) < 0.01 — no temporal leak in policy state |
| 2.13 | `eval_train_consistency` | KL(train-mode, eval-mode action dists) < 0.5 |
| 2.14 | `magnitude_uses_all_buckets` | each mag bucket > 5% of trade events |
| 2.15 | `fold_transition_stability` | sharpe collapse at fold boundary < 50% |
| 2.16 | `q_value_bounded_under_replay` | 1k replay updates → Q-values stay in [V_MIN, V_MAX] |
| 2.17 | `gradient_flow_to_all_heads` | non-zero grad reaches every head (dir/mag/ord/urg/aux/CQL/IQN/C51/ens) |
| 2.19 | `kelly_calibration_test` | kelly_f monotonically decreasing in Q-variance |
| 2.20 | `state_input_grounding_test` | KL(uptrend-context, downtrend-context action dists) > 0.5 |
| 2.21 | `egf_gate_opens` (Phase 0 anchor) | aux/Q deliberate disagreement → gate1 opens within 100 steps |
#### Group 3: Recovery dynamics (4 tests, anchors Phase 3.5 teachings)
| # | Test | Pass criterion |
|---|---|---|
| 2.5 | `drawdown_de_risks` | inject 10-bar loss streak → Kelly cap ≤0.5×base over next 100 bars |
| 2.10 | `recovery_after_streak` | inject 10R loss streak → recover ≥5R over next 200 bars (NOT collapse to all-Hold) |
| 2.11 | `dd_state_aware_sizing` | KL(0% DD, 50% dd_budget action dists) > 0.1 — policy conditions sizing on DD |
| 2.12 | `cooldown_engagement` | K=5 losses → cooldown fires within 1 bar; disengages cleanly after M=20 bars |
### 7.4 Sequencing within Phase 2
Three atomic commit batches:
- **2A: scaffolding** (synthetic gens + oracle + harness) — first
- **2B: Groups 1+2** (17 tests) — once Phase 0+1 ships; gates Phase 3 entry
- **2C: Group 3** (4 tests) — written alongside Phase 3.5 teachings; each test commit paired with its teaching commit per the discipline rule
### 7.5 Discipline gate enforcement
Every Phase 3 / 3.5 commit cites which Group 1/2/3 test it makes green. If teaching X lands but `2.X` doesn't go green, teaching reverts. No "we'll fix the test later" exceptions.
---
## 8. Phase 3 — Trader Teaching (~3-5 days, atomic per teaching, sequential)
### 8.1 Goal
Reward shape and action selection teach the trader role, not just optimize a number.
**SP12 v3 (per-trade event-driven reward) stays.** Phase 3 builds ON SP12, doesn't replace it. SP12's per-event asymmetric pos/neg cap (5.0 / -10.0) and zero per-bar dense shaping are preserved.
### 8.2 The 5 teachings
#### 3.1 — r_quality + r_discipline split
Architectural framing, no separate code beyond the convex combination:
```
r_total = α × r_quality_per_event + (1 − α) × r_discipline_per_bar
```
- `r_quality_per_event` = SP12 per-trade event-driven reward (UNCHANGED)
- `r_discipline_per_bar` = 3.4 regret signal (NEW)
- α is a fixed config constant (default 0.7), NOT learned, NOT controller-modulated. Per `feedback_no_feature_flags`: no `enable_*` toggle.
**Anchor tests**: 2.4 (cost_sensitivity), 2.6 (regime_silences). Both should be made simultaneously achievable by the split.
#### 3.2 — Explicit cost in r_quality
```
r_quality_per_event −= cost_t × trade_attempt_indicator
```
`cost_t` is the same Phase 1.2 unified kernel (commission + spread + OFI-impact). Model SEES the bill in the gradient signal during training, not just in eval-time metrics.
**Anchor test**: 2.4 (cost_sensitivity). Policy must trade less under high-spread regime.
#### 3.3 — Quadratic drawdown penalty
```
r −= λ_dd × max(0, current_dd − dd_threshold)²
```
- Asymmetric quadratic: zero penalty until dd crosses threshold, then quadratic growth. Encodes loss aversion per `pearl_audit_unboundedness_for_implicit_asymmetry`.
- `dd_threshold`: ISV-driven from running variance of dd (initial: 0.05 = 5% of capital, retunes to per-fold scale).
- `λ_dd`: ISV-driven from grad-balance ratio between r_total and r_quality (so DD term doesn't dominate gradient).
**Anchor test**: 2.5 (drawdown_de_risks). Policy must reduce position size after loss streak.
#### 3.4 — Regret signal = r_discipline content
When the policy holds and a trade WOULD have been profitable past `cost_t + trail_dist`, log `regret_bar = ideal_pnl_missed − cost_t`.
```
r_discipline_per_bar = −λ_regret × regret_ema
```
- `regret_ema`: rolling EMA of `regret_bar` (Wiener-α adaptive smoothing per `pearl_wiener_optimal_adaptive_alpha`).
- `λ_regret`: ISV-driven from a target ratio between regret-derived gradient and r_quality gradient.
Teaches "Hold when no edge: fine. Hold when edge exists: bad."
**Anchor test**: 2.6 (regime_silences). Policy must trade in high-edge regimes, NOT silence in them.
#### 3.5 — Confidence-aware Hold floor
Action-selection level (NOT reward shaping):
```
hold_floor = monotonic_inv_func(entropy(dir_Q_distribution))
```
When entropy is high (low confidence) → bias toward Hold by adding `+hold_floor` to the Hold-Q before argmax/Thompson selection. Hold becomes uncertainty expression, not distributional default.
`monotonic_inv_func` parameters (slope, anchor) are ISV-driven from running entropy distribution.
**Anchor tests**: 2.1 (flat_market_holds), 2.6 (regime_silences). Hold must dominate when dir entropy is high.
### 8.3 Phase 3 sequencing
Sequential. Each teaching's commit asserts (a) its anchor test goes green AND (b) prior anchor tests still pass. If 3.3 breaks 2.4, 3.3 reverts.
---
## 9. Phase 3.5 — Recovery Dynamics (~2-3 days, atomic per mechanism, sequential)
### 9.1 Goal
Model can escape its own losses — break the spiral observed in train-dd4xl.
**3.5.1 (dd_pct as state input) already landed in Phase 1.5** per Q1. Phase 3.5 has 4 remaining mechanisms.
### 9.2 The 4 mechanisms
#### 3.5.2 — Asymmetric reward under DD
```
For gains: r_adjusted = r × (1 + λ × dd_pct)
For losses: r_adjusted = r (unchanged)
```
- `λ` ≈ 0.5 (ISV-tracked from observed DD distribution)
- Layered on top of SP12 asymmetric pos/neg caps. Composes cleanly because SP12 is event-level value cap; this is gain-side multiplier conditioned on DD state.
Teaches "a winning trade out of DD is worth more than the same trade at ATH." Matches actual trading P&L psychology — recovering from drawdown is harder than maintaining ATH, so recovery gains deserve more reward signal.
**Anchor test**: 2.10 (recovery_after_streak).
#### 3.5.3 — Cooldown gate
After K consecutive losing trades, force Hold for M bars. Both K and M are ISV-driven:
- K: tracked from running variance of trade outcomes (cooldown trips when streak exceeds typical loss-cluster length)
- M: tracked from time-to-mean-reversion of vol_normalizer signal (cooldown lasts until regime stabilises)
**Cooldown counter is part of state** — model can reason about it, not work around it. New ISV slot `COOLDOWN_BARS_REMAINING_INDEX`. Producer decrements per bar; consumer (action selection) forces Hold while > 0.
**Anchor test**: 2.12 (cooldown_engagement).
#### 3.5.4 — Plasticity injection during persistent drawdown
When `dd_persistence_bars > threshold` (Phase 1.3 ISV slot), reset last 10% of advantage-head weights to Xavier-initialised values to escape local minima.
- Trigger fires at most once per fold (debounced via additional ISV slot `PLASTICITY_INJECTION_FIRED_INDEX`).
- Threshold is ISV-driven from running mean of `dd_persistence_bars` over training history.
- Reset target: last 10% of advantage-head weights (NOT shared trunk, NOT direction Q-head fully). Affects only the magnitude/ordinal/urgency advantage tails where local minima are most likely.
Mimics real-trader "step away & restart" — the most aggressive recovery mechanism. Most research-y; gated strictly on 2.10 staying green.
**Anchor test**: 2.10 (recovery_after_streak) AND must NOT regress 2.16 (q_value_bounded_under_replay) — plasticity reset cannot create unbounded Q values.
#### 3.5.5 — Recovery curriculum in PER
When sampling from PER, **oversample episodes where policy started in drawdown and recovered to flat** (`dd > 0.05` at start, `dd < 0.01` at end). Provides positive recovery examples for credit assignment.
- Oversample factor: ISV-driven from current-DD context (more DD now → more weight on recovery episodes).
- Currently PER oversamples loss states (high TD-error), which compounds the spiral. This counter-balances by giving "I survived a similar drawdown" examples more weight.
**Anchor test**: 2.10 (recovery_after_streak).
### 9.3 Phase 3.5 sequencing
Per Q7: **all 4 mechanisms land**. No defer of 3.5.4.
```
Phase 2B green → 3.5.2 → 3.5.3 → 3.5.4 → 3.5.5
↓ each gated on anchor test
```
3.5.4's gate is strict: must not regress 2.10 OR 2.16. If it does, reverts and we ship 3.5 with only 3.5.2 + 3.5.3 + 3.5.5.
---
## 10. Phase 4 — Final Validation (~2-3 days)
### 10.1 Goal
Produce trustworthy "this trained system is/isn't a trader" signal. Phase 4 is the only phase that touches the sealed Q9.
### 10.2 Sub 4.1 — Pre-flight (~½ day)
- All 21 Phase 2 tests green on dev (RTX 3050 Ti). Pre-commit hook blocks if not.
- Per-pearl audit doc updated, MEMORY.md index extended for any new pearls landed.
- New worktree `sp15-final-validation` for the deployment artifacts.
### 10.3 Sub 4.2 — Walk-forward training (~1 day wall)
```
argo-train.sh \
--branch sp15-final-validation \
--commit <sp15-merged-sha> \
--holdout-quarters 1 \
--dev-quarters 1 \
--seeds 3 \
--epochs 30
```
- 3 seeds × 3 walk-forward folds on Q1-Q7. Q8 used as final dev (early-stop, Kelly warmup, controller anchors legitimately read from Q8 — it IS the dev set now, not the test set).
- HEALTH_DIAG emits `sharpe_net` + `Calmar` + 8 baseline deltas per fold per epoch.
- **Hard kill criterion** (per `feedback_kill_runs_on_anomaly_quickly`): if any fold sharpe_net trends down 3 epochs straight AND below all 8 baselines, terminate workflow + diagnose which Group 1/2/3 test failed in the wild.
- Best checkpoint per seed/fold saved to `trained-models-pvc` (task #30 dependency — needs landing first or as Phase 4 sub-step).
### 10.4 Sub 4.3 — Sealed Q9 OOS evaluation (~½ day)
- **Separate Argo workflow** `scripts/argo-eval-final.sh` — model-loaded eval-only, never touches training. Cannot accidentally use Q9 during a misconfigured training run.
- 3 seeds × Q9 evaluation. Report mean ± std for each metric.
- **Q9 burn policy**: each Q9 run generates `docs/runs/YYYY-MM-DD-sp15-q9-<commit>.md` audit file. Q9 is "burned" when run; subsequent runs against meaningfully-different architecture are allowed but each is logged. Clearly NOT to be used for hyperparameter tuning.
### 10.5 Sub 4.4 — Report card
Generated automatically by Phase 4.3 workflow:
```
SP15 Q9 Final OOS Report — commit <sha>
─────────────────────────────────────────
Per-fold (Q1-Q7 walk-forward, Q8 final dev):
fold 0: sharpe_net=X Calmar=Y trades=N dd_max=Z
fold 1: ...
fold 2: ...
Q8 dev: ...
Q9 Sealed OOS (3-seed mean ± std):
sharpe_net = X.X ± σ
Calmar = Y.Y ± σ
trade_count = N ± σ
max_dd_pct = Z.Z ± σ
recovery_bars = R ± σ
Deltas vs 8 baselines (Q9):
vs buyhold : +X.X (need > 0)
vs hold-only : +X.X (need > 0)
vs random_dir : +X.X (need > 0)
vs naive_momentum : +X.X (need > 0)
vs aux-only : +X.X (need > 0)
vs mag-Q-fixed : +X.X (need > 0)
vs trail-only : +X.X (need > 0)
vs naive_revert : +X.X (need > 0)
Behavioral test status (final, on dev):
21/21 passing
Pearls landed: <list>
Architectural changes: dd_pct as state, cost-net unified, 8 baselines, Q9 sealed
```
### 10.6 Sub 4.5 — Production-track gate
**Pass criteria**:
1. Q9 `sharpe_net > 1.0`
2. Q9 `Calmar > 1.0`
3. Q9 beats ALL 8 baselines on net sharpe (delta > 0 for each)
4. All 21 Phase 2 tests still pass post-deployment
**Fail mode → behavioral test mapping**:
| Failure | Targeted test | Action |
|---|---|---|
| sharpe_net low | 2.4, 2.6 | Cost-aware trading / regime appropriateness broken |
| Calmar low | 2.5, 2.10 | Drawdown management or recovery broken |
| loses to hold-only | 2.6 | Model overtrading garbage |
| loses to aux-only | architecture | Q-network not earning keep over aux head — consider simplification |
| loses to mag-Q-fixed | 2.14 | Magnitude head dead silicon |
| loses to trail-only | architecture | Entry logic not adding value over random+disciplined exit |
NOT a generic re-train. Each failure axis has a specific behavioral test to target.
---
## 11. Cumulative scope
| Phase | LOC | Days serial | Key artifacts |
|---|---|---|---|
| 0 | ~400 | 1-2 | EGF retune kernels, 4 ISV slots, test 2.21 |
| 1 | ~700 | 2-3 | sharpe kernel, cost kernel, 8 baselines, dd_pct trunk concat, CLI flags |
| 2 | ~2000 | 3-5 | Synthetic markets, harness, 21 tests, pre-commit hook |
| 3 | ~800 | 3-5 | 5 teachings (reward + Hold floor) |
| 3.5 | ~700 | 2-3 | 4 recovery mechanisms |
| 4 | ~300 | 2-3 | argo-eval-final.sh, report card automation |
| **Total** | **~4900** | **13-21** | |
With Approach B parallel dispatch on Phases 0+1+2A: ~10-15 days realistic.
---
## 12. Risks and open questions
### 12.1 3.5.4 plasticity injection is research-y
No published results in trading RL for "reset weights mid-training to escape local minima." Risk: mistuned trigger destroys good learning. Mitigation:
- Strict gate on 2.10 staying green
- Strict gate on 2.16 (q_value_bounded) NOT regressing — plasticity must not create unbounded values
- Debounced fire (at most once per fold)
### 12.2 8 baselines compute cost
Each baseline runs as a separate kernel against the val/test window. 8× the eval cost per fold per epoch. Mitigation:
- Baselines are pure forward (no backprop), much cheaper than training step
- Run baselines only once per fold per epoch (not per step)
- Estimated overhead: ~5-10% of total eval cost
### 12.3 Q9 burn policy
"Each Q9 run generates audit file" relies on discipline, not enforcement. A motivated developer could run Q9 eval N times and only commit the best. Mitigation:
- Q9 run requires a tagged commit — `argo-eval-final.sh` rejects un-tagged or amended commits
- Audit log includes counter incrementing per architectural milestone — anomalous gaps surface gaming
### 12.4 Synthetic LOB fidelity for Phase 2
Phase 2 synthetic markets generate synthetic spread + OFI to match the Phase 1.2 unified cost kernel's signature. Risk: synthetic LOB doesn't match real LOB statistics → Phase 2 tests pass but L40S behavior diverges.
Mitigation:
- Calibrate synthetic LOB to match MES historical first-moments (mean spread, OFI distribution) on Q1-Q7 training data
- Add Phase 2 test "cost_consistency_check": run policy on (synthetic LOB, real LOB) — KL divergence < 0.3
- This is the same kind of dev/prod parity discipline as the unified cost kernel itself
### 12.5 Discipline rule enforcement
"No new pearl without behavioral test" relies on commit-time review. Mitigation:
- Pre-commit hook checks new ISV slots against `docs/dqn-wire-up-audit.md` entries
- Plan reviews in PRs reference the discipline rule explicitly
- If violation slips through, it's reverted, not patched
### 12.6 Phase 3.1 "architectural split" is fuzzy
3.1 is described as "no separate code beyond the convex combination." Risk: it doesn't land as a distinguishable commit, gets merged into 3.4. Mitigation:
- 3.1 lands as a single explicit commit that introduces the `α × r_quality + (1-α) × r_discipline` combine point with `r_discipline = 0.0` default. 3.4 then populates `r_discipline` with regret content.
- If 3.1 ends up indistinguishable from 3.4 in implementation, accept that and merge into a single 3.1+3.4 atomic commit. The discipline rule still applies (anchor tests 2.4, 2.6 must go green).
### 12.7 Layout fingerprint break in Phase 1.5
`dd_pct` trunk concat invalidates pre-SP15 checkpoints. Per Q1 ("greenfield, checkpoint compatibility doesn't matter") this is acceptable — but it does mean any in-flight L40S work prior to SP15 cannot resume from those checkpoints.
Mitigation:
- Phase 1.5 commit message tagged with explicit warning
- `argo-train.sh` rejects loading pre-SP15 checkpoints (layout fingerprint mismatch hard error)
---
## 13. Anticipated memory pearls
To be authored alongside the implementation phases:
| Pearl | Origin |
|---|---|
| `pearl_dd_pct_foundational_state` | Phase 1.5 — DD-context as eval-time state input vs reward-only modulation |
| `pearl_unified_cost_kernel_dev_prod_parity` | Phase 1.2 — same cost kernel reads spread/OFI from synthetic (dev) or real (prod) LOB |
| `pearl_behavioral_test_gates_l40s_deploy` | Phase 2 — pre-commit hook + dev-runtime test suite is the L40S filter |
| `pearl_recovery_curriculum_in_per` | Phase 3.5.5 — PER weighting can encode "recovery exists" signal explicitly |
| `pearl_plasticity_injection_for_rl_local_minima` | Phase 3.5.4 — weight-reset mechanism for breaking spiral attractors (if 2.10 verifies) |
| `pearl_8_counterfactual_baselines_arch_specific` | Phase 1.4 — domain baselines that test specific subsystems we built (aux-only, mag-Q-fixed, trail-only) |
| `pearl_egf_isv_driven_anchors` | Phase 0.B — Schmitt thresholds and variance refs as ISV slots, not constants |
---
## 14. Discipline rule (re-stated for posterity)
**No new pearl, controller, kernel, ISV slot, or reward term ships without:**
1. A Phase 2 behavioral test that proves the intended trader behavior
2. A wire-up audit entry
3. A memory pearl if the technique generalises
Violations mean the change is reverted, not merged-with-followup.
This rule applies to every commit on the `sp15-trader-discipline-recovery` branch and every commit that merges back to main. It supersedes prior tolerance for "land it now, behavioral validation later." There is no later.