diff --git a/docs/superpowers/specs/2026-04-21-unified-train-val-env-design.md b/docs/superpowers/specs/2026-04-21-unified-train-val-env-design.md new file mode 100644 index 000000000..e6d19123f --- /dev/null +++ b/docs/superpowers/specs/2026-04-21-unified-train-val-env-design.md @@ -0,0 +1,254 @@ +# Unified Training and Validation Environments — Design + +## Problem + +The DQN training loop produces a model whose training Sharpe and validation +Sharpe diverge catastrophically: + +| Metric | Training | Validation | +|---|---|---| +| `Sharpe_raw` (per-bar) | +0.03 to +0.07 | -0.31 to -0.39 | +| `WinRate` | 15% – 28% (oscillating) | 26% – 32% (stable) | +| `MaxDD` | 0.002% – 0.006% | 10% – 15% | +| Sharpe (annualized) | -13 to +20 (noisy) | -97 to -150 (flat) | + +Measured over a 50-epoch E1 smoke test run on 2026-04-20 after the +distillation-collapse fixes landed (commit `24f96ab78`). + +**Same network, same weights, same data — drastically different outcomes.** +Validation metrics are *stable* across 50 epochs (the network converges to a +deterministic losing policy) while training metrics oscillate (shaped reward +varies with exploration noise). This is not a bug in any single component; +it is the product of two environments that have drifted apart over many +incremental design decisions. + +### Evidence that it's architectural, not a single bug + +A reward-shaping ablation (2026-04-20, E1 smoke) closed ~30% of the gap: + +- Dropping `order_credit_weight`, `urgency_credit_weight`, + `risk_efficiency_weight`, `exit_timing_weight`, `ofi_reward_weight`, + `kelly_sizing_weight`, `reward_noise_scale`, `cea_weight` +- Result: val `Sharpe_raw` -0.39 → -0.31, val WinRate 30% → 41%, val MaxDD + 11% → 8% + +The remaining ~70% gap is architectural: the environments execute the same +weights against different trade physics. + +### Concrete divergences between the two environments + +| Aspect | `experience_kernels.cu` (training) | `backtest_env_kernel.cu` (validation) | +|---|---|---| +| Spread scaling | `0.5× – 2.0×` via CUSUM regime proxy (line 1395) | Fixed (no scaling; `-1.0f` override) | +| Fill model | Saboteur: per-episode `(spread_mult, fill_prob, slippage_mult)` adversarial noise | Always fill at close price | +| Transaction cost | `tx_cost_multiplier` scaled by saboteur + curriculum | Fixed `tx_cost_bps` | +| Reward composition | 18 shaping terms (DSR, DD, churn, micro-reward, credits, entropy, HER, counterfactuals, plan params, OFI, Kelly…) | Pure P&L + inventory penalty + churn penalty | +| Action selection | ε-greedy with `ε ≈ 0.02` + noisy-net parameter noise + plan-constrained | Deterministic argmax (no noise) | +| Position dynamics | `plan_params` influence target position + counterfactual flips (N4) | Pure action → target position mapping | +| Trailing stop | Fixed parameters (aligned per comment: "removed regime scaling to eliminate mismatch") | Same fixed parameters | +| Capital floor | 25% drawdown liquidation | 25% drawdown liquidation (matched) | + +The trailing-stop comment at `experience_kernels.cu:1418` tells the story: +*"Regime-adaptive scaling removed to eliminate train/eval mismatch."* +Someone was here before. They aligned *some* things. Many more remain +unaligned. + +## Design goals + +1. **Training and validation must produce comparable scalar P&L**, up to the + noise floor introduced by intentional randomization (exploration). +2. **Validation must be an honest measure of deployed-model behavior** — + adding shaping to the validation env to close the gap is the wrong + direction. +3. **Training must still be stable** — the shaping terms were added to + address real pathologies (churn, over-trading, drawdown); stripping them + without replacement risks regressing training dynamics. +4. **The fix must be testable within the existing smoke-test budget** (~30s + per run on RTX 3050 Ti; 20 epochs). Otherwise iteration becomes too slow + to converge on the right answer. + +## Options considered + +### Option A — Backtest matches training + +Make the backtest kernel execute the full shaping stack: DSR reward, churn, +credits, saboteur noise, counterfactuals, plan params. + +- **Pro**: closes the gap immediately — val Sharpe would track training + Sharpe by construction. +- **Con**: val is no longer an honest measure of deployed P&L. Training + metric "looks good" but model still bleeds money in the real world. This + is strictly worse than the current state — the gap was informing us of a + real issue, and we'd be hiding it. +- **Verdict**: **Rejected**. Measurement degradation. + +### Option B — Training matches backtest (strip shaping) + +Remove all shaping terms. Training reward = trading P&L + minimal safety +(drawdown penalty only). + +- **Pro**: pure alignment. One source of truth for "good model." +- **Con**: the shaping terms exist because naive P&L training exhibits + pathologies — churning, over-trading, gambling for variance, collapse in + sparse-reward regimes. Stripping without replacement regresses to those + failure modes. +- **Verdict**: **Rejected**. Throws the baby out with the bathwater. + +### Option C — Unified environment with conditional shaping layer (RECOMMENDED) + +One environment kernel that implements trade physics identically for +training and validation. Reward is a *layered* computation: a core P&L term +(identical in both modes) plus a tunable *shaping contribution* that is +**additive, P&L-aligned, and documented in units of "dollars per bar."** + +Key design principles: + +1. **Core reward = per-step change in portfolio equity.** Same in training + and validation. This is the ground truth. +2. **Shaping terms must be P&L-units additive.** Each shaping term's + contribution is measured in the same units as the core reward (dollars, + or normalized portfolio fraction). The sum of `core + sum(shaping)` IS + the training signal; the validation signal is `core` alone. The gap + between training and validation Sharpe becomes a *computable* quantity: + `E[shaping] / σ(core)` — and can be bounded by caps on shaping + magnitudes. +3. **Non-P&L-aligned shaping is forbidden.** Terms like + `order_credit_weight` (reward for picking a certain order-type branch + regardless of outcome) violate this and are deleted, not disabled — + per `feedback_no_feature_flags.md`. +4. **Adversarial/exploration noise (saboteur, ε-greedy, plan nudges) only + modulates the core state transition, not the reward.** The reward is + whatever actually happened — fills, slippage, costs all reflected. + Exploration doesn't get "credit" for trying; it gets credit only for + what its trials actually produced. +5. **Validation = training with exploration = 0 AND shaping = 0.** + Single flag at the env entry point (a scalar mode multiplier, not a + boolean: `exploration_scale: f32`, `shaping_scale: f32`). In training, + both are 1.0; in validation, both are 0.0. The env kernel itself is + *identical code* — no branches on mode. + +This is the same architectural move that fixed the collapse-prevention +distillation: compute the varying signal at the source (GPU, per-step) +from a scalar mode factor baked via pinned device-mapped memory, rather +than maintaining two parallel code paths. + +### Option D — Keep both environments; use both signals meaningfully + +Accept the two environments as-is. Use training Sharpe for hyperparameter +search and validation Sharpe for deployment gating. Report the gap as a +model-quality metric. + +- **Pro**: no code changes. Documents reality. +- **Con**: doesn't *solve* anything. Leaves the divergent environment + structure as permanent technical debt. Future changes to one env will + continue to drift from the other. +- **Verdict**: **Fallback only**. Choose this only if C turns out to be too + expensive to implement. + +## Recommended path: Option C + +### Phase 1 — Inventory shaping terms (≤1 day) + +Document every term currently summed into training reward. For each: + +- What was it added for? (find the commit) +- Is its contribution P&L-aligned? (does it reward outcomes or behaviors?) +- Can it be re-expressed as an additive dollar-cost term? + +Expected output: two lists. +- **Keepers**: drawdown penalty, transaction-cost penalty, inventory + penalty, capital-floor liquidation cost — all in P&L units. +- **Removers**: credit weights, entropy bonuses, DSR, micro-reward, + counterfactual reward scaling — all behavioral. + +Store the inventory as a commit-tracked doc. This is the one-time +reckoning. + +### Phase 2 — Rewrite env as a single kernel (≤3 days) + +New file: `crates/ml/src/cuda_pipeline/unified_env_kernel.cu`. One +`__global__ void unified_env_step(...)` that: + +- Takes the exploration/shaping scale scalars as `const float*` device + pointers (pinned device-mapped, same pattern as distillation alpha). +- Computes trade physics once. +- Computes reward = `pnl_per_step + shaping_scale × shaping_bundle`. +- Writes `step_return = pnl_per_step` to a separate output buffer — this + is the **authentic** P&L signal used by validation regardless of + training/validation mode. + +Both training and validation call the same kernel. The only difference is +the scalar values at the pinned pointers. + +### Phase 3 — Deprecate the two old kernels (≤1 day) + +Delete `experience_env_step_batch` and `backtest_env_step_batch`. Rewire +all callers to the unified kernel. Remove any dead utility code that was +specific to one path. + +### Phase 4 — Re-tune adaptive-learning mechanisms (≤2 days) + +The distillation, collapse-prevention, and health-EMA machinery was tuned +against the old training reward magnitudes. With shaping removed or +re-scaled, the health-signal thresholds (`SNAPSHOT_HEALTH_THRESHOLD`, +`DISTILL_HEALTH_THRESHOLD`, etc.) may need recalibration. The E1 smoke +test is the canary. + +### Phase 5 — Verify on L40S full-scale run (≤1 day) + +Production 50-epoch run. Expected signatures of success: + +- Training `Sharpe_raw` ≥ 0 by epoch 10; ≥ +0.03 by epoch 30 +- Validation `Sharpe_raw` ≥ training `Sharpe_raw` − 0.05 +- Validation `MaxDD` ≤ 3× training `MaxDD` +- Validation `WinRate` tracks training `WinRate` ± 5 percentage points + +If any of these fail, the analysis is wrong and we revisit Phase 1. + +## Risks + +1. **Collapse-prevention regresses.** The distillation mechanism was tuned + with the existing shaping weights in the loop. Pure-P&L training may + cause the network to find attractor states the current mechanisms don't + cover. Mitigation: Phase 4 recalibration, E1 smoke test as the gate. +2. **Training becomes too noisy.** Sparse P&L reward may have insufficient + gradient signal for the Q-network to learn. Mitigation: the counterfactual + curriculum (N4) should provide dense gradient signal even with pure P&L + reward — verify this empirically. +3. **Historical reward-shaping decisions are load-bearing in ways we don't + see.** Some terms may have been added to fix subtle numerical instability + (e.g., `reward_noise_scale = 0.05` — label smoothing proxy). Mitigation: + Phase 1's per-term inventory must include the provenance commit, not just + the semantic purpose. + +## Out of scope + +- Changing the underlying state representation (96-dim layout, OFI + integration, MTF). +- Changing the network architecture (4-branch factored DQN, ensemble heads, + IQN/C51). +- Re-tuning transaction cost parameters (`tx_cost_bps`, `spread_cost`). + These are calibrated to actual broker execution and should remain as-is. + +## Success criteria + +A pull request that: + +1. Deletes `experience_env_step` and `backtest_env_step` kernels in favor + of a single unified env step. +2. Deletes the non-P&L-aligned shaping terms (from Phase 1 inventory). +3. Passes the E1 collapse-prevention smoke test with final + `epoch_q_gap > 0.05`. +4. Shows validation `Sharpe_raw` within 0.05 of training `Sharpe_raw` by + epoch 30 on the L40S 50-epoch production run. +5. Documents the reward semantics in a single comment block at the + unified kernel site. + +## References + +- Current distillation root-cause analysis: commit `24f96ab78` +- Smoke test run showing divergence: 2026-04-20 E1 trajectory +- Reward shaping ablation result: 30% gap closure from disabling 8 credit + weights +- Existing hint at the problem: `experience_kernels.cu:1418` comment + *"Regime-adaptive scaling removed to eliminate train/eval mismatch"*