From 5417e27567938c23ce534bd5a79f524156497c59 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Wed, 6 May 2026 08:58:28 +0200 Subject: [PATCH] =?UTF-8?q?docs(sp15):=20amend=20spec=20v2=20=E2=80=94=201?= =?UTF-8?q?0=20fixes=20from=20second=20critical=20review?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Second critical review pass found 10 more issues, 3 critical. All addressed: CRITICAL: - §8.2 (3.1): ALPHA_SPLIT cold-start was unspecified — formula produces 0/0=0, not the claimed 0.5 sentinel. Now: ISV slot initialized DIRECTLY to 0.5 in trainer constructor; formula takes over only after both grad-norm EMAs accumulate N_warm non-zero observations - §9.2 (3.5.4): plasticity now performs TWO-STEP recovery: (1) Flat all positions at fire bar (close current losing trade), (2) engage warmup cooldown forcing Hold for M_warm bars. Without step 1, forced Hold preserved the losing position that drove drawdown for the entire 200-bar warmup - §12.2: stale "5-10%" baseline cost estimate updated to "15-25%" matching §6.4 (was contradicting earlier amendment) IMPORTANT: - §9.2 (3.5.5): DD_TRAJECTORY_DECREASING threshold 0.02 hardcoded → ISV-driven via new slot DD_TRAJECTORY_FLOOR (slot 441, 25th percentile of running dd_pct distribution) - §8.2 (3.5): HOLD_FLOOR_ALPHA tracked from rolling 95th percentile of |Q_dir| (NOT running max — was outlier-ratchet vulnerable) - §9.2 (3.5.3): MEDIAN_STREAK_LENGTH formerly undefined in cooldown K formula → ISV-driven via new slot 442, running median of observed loss-streak lengths via two-heap algorithm - §9.2 (3.5.2): asymmetric reward × α split compound interaction explicitly stated as intentional with POS_CAP as binding ceiling NIT: - §7.4: "2C: Group 3 (4 tests)" → "(5 tests)" (was off-by-one after 2.22 added) - §9.2 (3.5.4): Xavier → Kaiming-He init for advantage head reset (architecturally appropriate for ReLU-gated activation chain) ISV_TOTAL_DIM: 441 → 443 post-SP15 (added DD_TRAJECTORY_FLOOR and MEDIAN_STREAK_LENGTH at slots [441..443)). 46 SP15 slots total. File: 799 → 811 lines. Spec is now consistent end-to-end with no contradictions between sections, no hardcoded values violating feedback_isv_for_adaptive_ bounds, and no underspecified load-bearing parameters. Co-Authored-By: Claude Opus 4.7 (1M context) --- ...6-trader-discipline-and-recovery-design.md | 36 ++++++++++++------- 1 file changed, 24 insertions(+), 12 deletions(-) diff --git a/docs/superpowers/specs/2026-05-06-trader-discipline-and-recovery-design.md b/docs/superpowers/specs/2026-05-06-trader-discipline-and-recovery-design.md index 25348c206..46e0e832d 100644 --- a/docs/superpowers/specs/2026-05-06-trader-discipline-and-recovery-design.md +++ b/docs/superpowers/specs/2026-05-06-trader-discipline-and-recovery-design.md @@ -164,7 +164,8 @@ Approach B parallel dispatch requires that Phase 0 / Phase 1 / Phase 2A sub-work | `[433..436)` | Phase 3.5.3 (cooldown) | `COOLDOWN_K_THRESHOLD` (ISV from running mean of per-trade PnL), `COOLDOWN_M_BARS`, `COOLDOWN_BARS_REMAINING` | 3 | | `[436..439)` | Phase 3.5.4 (plasticity) | `PLASTICITY_FIRED_THIS_FOLD`, `PLASTICITY_PERSISTENCE_THRESHOLD`, `PLASTICITY_WARM_BARS_REMAINING` (force cooldown engagement post-fire) | 3 | | `[439..441)` | Phase 3.5.5 (recovery curriculum) | `DD_TRAJECTORY_DECREASING` (per-step proxy for "in recovery"), `RECOVERY_OVERSAMPLE_WEIGHT` | 2 | -| **Total** | | **`ISV_TOTAL_DIM = 441` post-SP15** | 44 | +| `[441..443)` | Phase 3.5 deferred ISV anchors (resolves second-review #5, #7) | `DD_TRAJECTORY_FLOOR` (25th percentile of `dd_pct` distribution — replaces hardcoded 0.02), `MEDIAN_STREAK_LENGTH` (running median of consecutive-loss streak lengths — used by 3.5.3 cooldown K formula) | 2 | +| **Total** | | **`ISV_TOTAL_DIM = 443` post-SP15** | 46 | Each phase's first commit lands `sp15_isv_slots.rs` extension covering its allocated range. Layout fingerprint seed extended atomically per phase. Sub-worktrees check the allocation map before assigning indices. @@ -396,7 +397,7 @@ Every behavior the policy must exhibit gets a deterministic test on dev RTX 3050 Three atomic commit batches: - **2A: scaffolding** (synthetic gens + oracle + harness) — first - **2B: Groups 1+2** (17 tests) — once Phase 0+1 ships; gates Phase 3 entry -- **2C: Group 3** (4 tests) — written alongside Phase 3.5 teachings; each test commit paired with its teaching commit per the discipline rule +- **2C: Group 3** (5 tests) — written alongside Phase 3.5 teachings; each test commit paired with its teaching commit per the discipline rule ### 7.5 Discipline gate enforcement @@ -424,8 +425,10 @@ r_total = α × r_quality_per_event + (1 − α) × r_discipline_per_bar - `r_quality_per_event` = SP12 per-trade event-driven reward (UNCHANGED) - `r_discipline_per_bar` = 3.4 regret signal (NEW) -- **α is ISV-driven** at slot `ALPHA_SPLIT` (slot 417 per §4.3). Producer reads `GRAD_NORM_QUALITY` (slot 418, EMA of |∇r_quality|) and `GRAD_NORM_DISCIPLINE` (slot 419, EMA of |∇r_discipline|), computes `α = grad_norm_q / (grad_norm_q + grad_norm_d + ε)` so the two signals contribute equal gradient pressure regardless of fold-scale. -- Per `feedback_isv_for_adaptive_bounds` — α tracks the actual gradient distribution rather than being a hardcoded 0.7 guess. (Initial sentinel α = 0.5 until Welford warm-up completes; first-observation bootstrap per `pearl_first_observation_bootstrap`.) +- **α is ISV-driven** at slot `ALPHA_SPLIT` (slot 417 per §4.3). Producer reads `GRAD_NORM_QUALITY` (slot 418, EMA of |∇r_quality|) and `GRAD_NORM_DISCIPLINE` (slot 419, EMA of |∇r_discipline|). +- **Cold-start init (resolves second-review #2)**: `ALPHA_SPLIT` slot is initialized **directly to 0.5 in the trainer constructor** (NOT derived from formula at boot). The formula `α = grad_norm_q / (grad_norm_q + grad_norm_d + ε)` only takes over once BOTH grad-norm slots have accumulated `≥ N_warm` non-zero observations (Welford warm-up; `N_warm = 100` initial sentinel, ISV-tracked from per-fold convergence pattern). Until then, the slot remains at the constructor-written 0.5. +- The `pearl_first_observation_bootstrap` semantics (sentinel = 0; first observation replaces directly) apply to the grad-norm EMAs themselves — first observation of `|∇r_quality|` becomes `GRAD_NORM_QUALITY[0]`, etc. But the dependent slot `ALPHA_SPLIT` is NOT computed from these until both EMAs are warmed up. +- Per `feedback_isv_for_adaptive_bounds` — once warm, α tracks the actual gradient distribution rather than being a hardcoded 0.7 guess. **Anchor tests**: 2.4 (cost_sensitivity), 2.6 (regime_silences). Both should be made simultaneously achievable by the split. @@ -476,7 +479,7 @@ hold_floor = α × σ(k × (entropy − ε₀)) Where: - `entropy` = Shannon entropy of dir_Q_distribution (range `[0, log(K_dir)]`, K_dir=4 → max ≈ 1.386) -- `α` = scale factor in `HOLD_FLOOR_ALPHA` (slot 426, ISV-tracked from running max of |Q_dir| so the bias is comparable to Q magnitudes) +- `α` = scale factor in `HOLD_FLOOR_ALPHA` (slot 426, ISV-tracked from rolling 95th percentile of |Q_dir| over fold — NOT running max, which is one-way ratchet sensitive to outliers. 95th-percentile is robust to early-training noise spikes while still tracking actual Q magnitude scale) - `k` = sigmoid steepness in `HOLD_FLOOR_K` (slot 427, ISV-tracked from running variance of entropy — wider entropy distribution → gentler slope) - `ε₀` = entropy threshold in `HOLD_FLOOR_EPS0` (slot 428, ISV-tracked at 75th percentile of running entropy distribution `ENTROPY_DIST_REF` slot 429 — biases Hold only when policy is in upper quartile of uncertainty) - `σ` = standard logistic sigmoid @@ -526,6 +529,7 @@ r_quality_post_cap = clamp(intermediate, NEG_CAP, POS_CAP) [SP12's cap stays fi - For `dd_pct = 0` (at ATH): `(1 + 0) = 1` → no-op, SP12 cap unchanged. - For `dd_pct > 0` AND `r_quality_pre_cap × (1 + λ × dd_pct) > POS_CAP`: the multiplier saturates the cap. Behaviorally correct — recovery trades hit ceiling sooner, encouraging frequent small recoveries over rare-large. - SP12 NEG_CAP (loss side) still applies asymmetrically; this teaching doesn't touch loss multiplier. +- **Compound interaction with α split (resolves second-review #8)**: `r_quality_post_cap` then feeds `r_total = α × r_quality_post_cap + (1 − α) × r_discipline`. The compound multiplier × α weight of recovery-trade r_quality is **intentional and bounded by POS_CAP**. The POS_CAP IS the binding ceiling — implementers should NOT "fix" the apparent double-amplification by removing α-weighting or splitting the cap. The semantics: amplifier in DD → cap in DD → α-weighted contribution to total. Cap is the safety; α is the balance signal. `λ` is ISV-tracked at `DD_ASYMMETRY_LAMBDA` (slot 430). Producer reads `DD_DIST_VAR` (slot 432, running variance of dd_pct over fold) — wider DD distribution → smaller λ (less aggressive multiplier when DD is volatile and unstable). Initial λ ≈ 0.5 sentinel. @@ -536,7 +540,8 @@ Teaches "a winning trade out of DD is worth more than the same trade at ATH." Ma #### 3.5.3 — Cooldown gate After K consecutive losing trades, force Hold for M bars. Both K and M are ISV-driven: -- **K (loss-streak threshold)**: at slot `COOLDOWN_K_THRESHOLD` (slot 433). Producer reads running mean of per-trade PnL (NOT variance — variance is LOW during a streak when all losses cluster, which would delay cooldown precisely when needed). When running mean drops below a fold-percentile threshold (e.g., 25th percentile), K is reduced — cooldown trips sooner. Concretely: `K = max(2, floor(median_streak_length × clamp(mean_pnl_recent / mean_pnl_baseline, 0.5, 2.0)))`. This makes K small (cooldown sensitive) when recent trades are below baseline. +- **K (loss-streak threshold)**: at slot `COOLDOWN_K_THRESHOLD` (slot 433). Producer reads running mean of per-trade PnL (NOT variance — variance is LOW during a streak when all losses cluster, which would delay cooldown precisely when needed). When running mean drops below a fold-percentile threshold (e.g., 25th percentile), K is reduced — cooldown trips sooner. Concretely: `K = max(2, floor(MEDIAN_STREAK_LENGTH × clamp(mean_pnl_recent / mean_pnl_baseline, 0.5, 2.0)))`. +- `MEDIAN_STREAK_LENGTH` (slot 442) is **ISV-driven** — running median of observed consecutive-loss streak lengths within fold. Producer kernel maintains a streak counter (incremented on losing trade, reset on winning trade), updates median via two-heap algorithm or running quantile sketch on each trade-close event. Per `feedback_isv_for_adaptive_bounds`: K threshold derives from observed streak distribution, not a guessed constant. - **M (cooldown duration)**: at slot `COOLDOWN_M_BARS` (slot 434). Tracked from time-to-mean-reversion of vol_normalizer signal — cooldown lasts until regime stabilises. **Cooldown counter as state**: ISV slot `COOLDOWN_BARS_REMAINING` (slot 435). Producer decrements per bar; consumer (action selection) forces Hold while > 0. Model SEES the counter on every forward — can reason about it, not work around it. @@ -545,12 +550,17 @@ After K consecutive losing trades, force Hold for M bars. Both K and M are ISV-d #### 3.5.4 — Plasticity injection during persistent drawdown -When `DD_PERSISTENCE` (slot 404 per §4.3) exceeds `PLASTICITY_PERSISTENCE_THRESHOLD` (slot 437), reset last 10% of advantage-head weights to Xavier-initialised values to escape local minima. +When `DD_PERSISTENCE` (slot 404 per §4.3) exceeds `PLASTICITY_PERSISTENCE_THRESHOLD` (slot 437), reset last 10% of advantage-head weights (initialization scheme specified below) to escape local minima. - **Debounced**: fires at most once per fold via `PLASTICITY_FIRED_THIS_FOLD` (slot 436). - **Threshold ISV-driven**: `PLASTICITY_PERSISTENCE_THRESHOLD` (slot 437) tracks running mean of `dd_persistence_bars` over training history. Set to mean + 2σ so only persistent (statistically unusual) drawdowns trigger. - **Reset target**: last 10% of advantage-head weights (NOT shared trunk, NOT direction Q-head fully). Affects only the magnitude/ordinal/urgency advantage tails where local minima are most likely. -- **Mandatory cooldown interlock (resolves Critical #3)**: when plasticity fires, the kernel ALSO writes `M_warm` to `PLASTICITY_WARM_BARS_REMAINING` (slot 438). This slot routes into the 3.5.3 cooldown consumer via OR-gate: `effective_cooldown = max(COOLDOWN_BARS_REMAINING, PLASTICITY_WARM_BARS_REMAINING)`. While `PLASTICITY_WARM_BARS_REMAINING > 0`, action selection forces Hold (same path as 3.5.3 cooldown). Prevents the random-tail-output → loss-streak → re-trigger loop. M_warm = 200 bars (initial sentinel; ISV-tracked post-fold from observed Adam re-equilibration time). +- **Reset initialization**: **Kaiming-He** (`mode=fan_in`, `gain` matched to activation). The trunk uses GRN blocks with GLU activations and the advantage heads chain through ReLU; Kaiming-He is calibrated for half-rectified activations and is the architecturally-appropriate default. Xavier (sometimes proposed) assumes symmetric linear activations and would underestimate variance for the actual activation chain. (Resolves second-review #10.) +- **Mandatory cooldown interlock (resolves Critical #3)**: when plasticity fires, the kernel performs a **two-step recovery sequence** (resolves second-review #3): + 1. **Flat all positions** at the fire bar (action = Flat, force-overrides policy output for that bar). The position that drove persistent drawdown is closed; the policy is no longer holding a known-bad position while its advantage tail is randomised. + 2. **Engage warmup cooldown** by writing `M_warm` to `PLASTICITY_WARM_BARS_REMAINING` (slot 438). This routes into the 3.5.3 cooldown consumer via OR-gate: `effective_cooldown = max(COOLDOWN_BARS_REMAINING, PLASTICITY_WARM_BARS_REMAINING)`. While `PLASTICITY_WARM_BARS_REMAINING > 0`, action selection forces Hold (now safe — no position to bleed since step 1 closed it). +- M_warm = 200 bars (initial sentinel; ISV-tracked post-fold from observed Adam re-equilibration time). +- Prevents the random-tail-output → loss-streak → re-trigger loop AND the open-losing-position-during-warmup leak. Symmetric: plasticity = "step away & restart" requires both closing current trade AND not re-entering during warmup, matching real-trader behavior. Mimics real-trader "step away & restart" — the most aggressive recovery mechanism. Most research-y; gated strictly on 2.10 staying green AND 2.22 (NEW interlock test) green. @@ -566,7 +576,8 @@ Mimics real-trader "step away & restart" — the most aggressive recovery mechan PER stores transitions `(s_t, a_t, r_t, s_{t+1})` with TD-error priority. Adding episode-level "started in DD, recovered to flat" tags requires structural PER changes (episode boundaries, episode-level indices) which are out of scope. **Per-transition proxy**: -- New per-step ISV signal `DD_TRAJECTORY_DECREASING` (slot 439): boolean (or smoothed scalar in `[0, 1]`) indicating `dd_pct(t) < dd_pct(t-1) AND dd_pct(t-1) > 0.02`. True when the transition is part of a recovery (DD shrinking from a non-trivial drawdown). +- New per-step ISV signal `DD_TRAJECTORY_DECREASING` (slot 439): boolean (or smoothed scalar in `[0, 1]`) indicating `dd_pct(t) < dd_pct(t-1) AND dd_pct(t-1) > DD_TRAJECTORY_FLOOR`. True when the transition is part of a recovery (DD shrinking from a non-trivial drawdown). +- `DD_TRAJECTORY_FLOOR` (slot 441) is **ISV-driven** — tracks 25th percentile of running `dd_pct` distribution. Per `feedback_isv_for_adaptive_bounds`: "non-trivial drawdown" scales with the fold's own DD regime, not a guessed `0.02` constant. - This signal is COMPUTED PER-BAR by the existing dd_pct producer (sub 1.5) — no PER restructure, no episode tracking, no replay buffer schema change. - When PER samples a batch, weight transitions by `1.0 + RECOVERY_OVERSAMPLE_WEIGHT × DD_TRAJECTORY_DECREASING_t` (slot 440). Transitions on a recovery path get amplified sampling weight. - `RECOVERY_OVERSAMPLE_WEIGHT` is ISV-driven from current `dd_pct`: more DD now → higher weight (current trajectory needs more recovery examples). @@ -719,10 +730,11 @@ No published results in trading RL for "reset weights mid-training to escape loc ### 12.2 8 baselines compute cost -Each baseline runs as a separate kernel against the val/test window. 8× the eval cost per fold per epoch. Mitigation: -- Baselines are pure forward (no backprop), much cheaper than training step +Each baseline runs against the val/test window. 8 baselines × N_bars per fold per epoch. Mitigation per §6.4: +- Baselines that need policy-trained heads (3 of 8: aux_only, mag_quarter_fixed, random_dir_kelly) share the trunk forward pass with main policy eval — no redundant trunk computation +- 5 of 8 are constant or last-bar-only (buyhold, hold_only, naive_momentum, trail_only, naive_reversion) — pure kernels with negligible cost - Run baselines only once per fold per epoch (not per step) -- Estimated overhead: ~5-10% of total eval cost +- **Realistic overhead estimate: ~15-25% of total eval cost** (per §6.4 — revised from earlier 5-10% which underestimated trunk-fusion-required baselines) ### 12.3 Q9 burn policy is honor-system