From 4a32b9f8b0a00d70fd3aa38552ab4c5799480a2f Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Thu, 16 Apr 2026 21:38:06 +0200 Subject: [PATCH] =?UTF-8?q?spec:=20Training=20Environment=20Alignment=20?= =?UTF-8?q?=E2=80=94=20single=20path=20to=20close=20the=2045=C3=97=20gap?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Root cause: training environment differs from validation backtest in 3 ways: 1. 100-bar episodes force exits (val runs continuous) → position-gated done 2. Rank normalization destroys sparse trade-level reward → remove it 3. Different Sharpe computation (per-trade vs per-bar, different annualization) Single execution path: Phase 1: Episode alignment (100→5000 bars, position-gated done, soft reset) Phase 2: Reward alignment (remove rank norm, raw trade P&L to replay) Phase 3: Metrics alignment (un-annualized per-trade Sharpe, both paths) Expected: training Sharpe within 2× of validation with same weights. Removes 5 unnecessary shaping components that fought the broken signal. Co-Authored-By: Claude Opus 4.6 (1M context) --- ...6-training-environment-alignment-design.md | 220 ++++++++++++++++++ 1 file changed, 220 insertions(+) create mode 100644 docs/superpowers/specs/2026-04-16-training-environment-alignment-design.md diff --git a/docs/superpowers/specs/2026-04-16-training-environment-alignment-design.md b/docs/superpowers/specs/2026-04-16-training-environment-alignment-design.md new file mode 100644 index 000000000..988c7733f --- /dev/null +++ b/docs/superpowers/specs/2026-04-16-training-environment-alignment-design.md @@ -0,0 +1,220 @@ +# Training Environment Alignment — Design Spec + +**Goal:** Align the training experience collector with the validation backtest evaluator so that training Sharpe reflects REAL model performance. Currently val_Sharpe=25 and training_Sharpe=0.19 with the SAME model weights — a 45× gap caused by structural differences between the two environments, not model quality. + +**Root Cause Analysis (this session):** + +Three bugs found and their status: + +| Bug | Impact | Status | +|-----|--------|--------| +| `raw_next` (future price) in reward path | Actions misaligned with outcomes by 1 bar | **FIXED** (commit ae5743ef4) | +| 100-bar episode length on H100 | Forces exits every 100 bars, kills long trades | **NOT FIXED** | +| Different Sharpe computation (per-trade vs per-bar, different annualization) | 45× gap is partly measurement artifact | **NOT FIXED** | + +Additional issues found: + +| Issue | Impact | +|-------|--------| +| Rank normalization on sparse trade-level reward | 95% of rewards are 0 (holding), ranking zeros is meaningless | +| Forced flat at episode boundary | Model can't hold positions > 100 bars | +| Portfolio state reset every 100 bars | Equity/position history lost, Mamba2 context lost | +| Metrics use different units | Training: per-trade annualized. Validation: per-bar annualized. Incomparable. | + +**Architecture:** ONE unified fix that makes the training environment match the validation backtest. Not 6 separate fixes — one coherent change with clear execution order. + +--- + +## The Single Path + +### Principle: Training = Validation + Exploration + +The training environment should be IDENTICAL to the validation backtest except for epsilon-greedy exploration. If a greedy policy produces Sharpe X in validation, the same policy with 1% exploration should produce ~0.99X in training. Currently it produces 0.007X (45× gap). + +### Changes (in execution order): + +#### 1. Episode Length: 100 → Position-Gated (no timer) + +**What:** Remove the fixed 100-bar episode timer. Episodes end ONLY when the model transitions to Flat after being positioned (natural trade completion). + +**Why:** The validation backtest runs continuously over 800K bars with no forced exits. The training experience collector forces exits every 100 bars. A trade that needs 200 bars gets killed at bar 100. The model learns to trade within 100-bar windows instead of learning the optimal holding period. + +**How (config/gpu/h100.toml):** +```toml +# OLD: gpu_timesteps_per_episode = 100 +# NEW: large enough to never be the binding constraint +gpu_timesteps_per_episode = 5000 +``` + +**How (experience_kernels.cu env_step done flag):** +```cuda +// OLD: done at fixed timer +int time_done = (step + 1) >= episode_length; + +// NEW: done ONLY when model naturally completes a trade cycle +// (was positioned → chose Flat, completing a trade) +int trade_complete = (fabsf(pre_trade_position) > 0.001f && fabsf(position) < 0.001f); +int data_done = (bar_idx >= total_bars - 1); +int done = trade_complete || data_done; +``` + +**V(flat) = 0 is correct:** When the model has no position, its expected unrealized P&L is zero. This gives the Bellman backup a mathematically correct terminal anchor without artificial resets. + +**VRAM:** `1024 episodes × 5000 timesteps × 80 dim × 4 bytes = 1.6 GB`. Fits easily in H100 80GB. Reduce n_episodes from 4096 to 1024 to compensate for longer episodes. Total experience per epoch: `1024 × 5000 = 5.12M steps` (was `4096 × 100 = 409.6K`). 12× more experience per epoch. + +**Portfolio state at done:** When `trade_complete=1`, clear position and unrealized P&L but KEEP equity (no artificial cash reset). The model starts the next "episode" with its accumulated gains/losses. + +```cuda +if (trade_complete) { + // Soft reset: clear position state, keep equity + ps[0] = 0.0f; // position = 0 (already flat) + // ps[9] = equity stays (no artificial reset) + // Cash stays at current value + // Entry price = 0 (no active trade) + entry_price = 0.0f; +} +``` + +#### 2. Remove Rank Normalization for Trade-Level Reward + +**What:** Disable `reward_rank_normalize` when using trade-level reward. + +**Why:** With trade-level reward, 95% of bars have reward ≈ 0 (holding cost) or exactly 0 (flat). Rank normalizing a distribution that's 95% zeros produces uniform noise — the $500 winner and the $5 winner get similar ranks. The model can't learn that bigger trades are better. + +**How (gpu_experience_collector.rs):** +```rust +// OLD: always rank normalize +self.shape_rewards(&mut rewards_dst, n, reward_std, reward_mean)?; + +// NEW: skip rank normalization — pass raw trade P&L directly +// The C51 distributional head handles the scale naturally. +// No normalization needed when rewards are sparse trade-level P&L. +``` + +The C51 atoms with adaptive positions already model the return distribution. Rank normalization was needed for dense per-bar noise — with sparse trade-level P&L, it destroys the signal. + +#### 3. Align Sharpe Computation + +**What:** Both training and validation compute Sharpe the SAME way — per-trade, same annualization. + +**Why:** Currently training uses per-trade Sharpe with `sqrt(trades_per_year)` annualization = 121×. Validation uses per-bar Sharpe with `sqrt(bars_per_year)` annualization = 313×. These are incomparable. val_Sharpe=25 and training_Sharpe=0.57 might be SIMILAR performance measured differently. + +**How (two options, implement BOTH for comparison):** + +Option A — Change validation to per-trade Sharpe (match training): +```rust +// In backtest_metrics_kernel.cu or the Rust metrics aggregation: +// Compute Sharpe from per-TRADE returns instead of per-BAR returns. +// This requires tracking trade entries/exits in the backtest step loop. +``` + +Option B — Report un-annualized Sharpe alongside annualized: +```rust +// In training_loop.rs epoch logging: +info!("Epoch {}: Sharpe_annualized={:.2} Sharpe_raw={:.4} (per-trade, un-annualized)", + epoch, financials.sharpe, financials.sharpe / trade_annualization); + +// In validation logging: +info!("val_Sharpe_annualized={:.2} val_Sharpe_raw={:.4} (per-bar, un-annualized)", + val_sharpe, val_sharpe / bar_annualization); +``` + +**Primary metric going forward:** Un-annualized per-trade Sharpe (mean_trade_return / std_trade_return). No annualization factor. Directly comparable between training and validation. + +#### 4. Pass Raw Trade P&L to Replay Buffer + +**What:** The reward written to the replay buffer should be the RAW trade P&L, not the shaped reward. + +**Why:** The shaped reward (after DSR, commitment penalty, entropy bonus, label smoothing, drawdown penalty, capital floor) is a heavily transformed version of the actual trade outcome. The model optimizes for shaped reward but is EVALUATED on raw P&L. This creates a systematic disconnect. + +**How:** +```cuda +// In experience_env_step, at the reward write: +// OLD: out_rewards[out_off] = reward; // heavily shaped +// NEW: out_rewards[out_off] = trade_level_pnl; // raw trade P&L + +// The shaping components (DSR, drawdown penalty, etc.) should +// modify the BELLMAN TARGET, not the reward. Or be removed entirely +// and replaced by the self-improving enrichment loop which adapts +// from actual eval performance. +``` + +This is the most aggressive change. Alternative: keep shaping but ALSO write raw P&L to a separate buffer for accurate metrics. + +--- + +## What This Removes + +The following components become UNNECESSARY with this alignment: + +| Component | Why unnecessary | +|-----------|----------------| +| Adaptive DSR reward shaping (G2) | Raw P&L is the reward — no shaping needed | +| Rank normalization (G13) | Sparse trade-level P&L shouldn't be ranked | +| Commitment penalty (G15) | Natural trade lifecycle replaces artificial penalty | +| Cost curriculum sigmoid | Costs are in the raw P&L from the portfolio sim | +| Q-mean drift penalty | Q-values should align with raw P&L naturally | + +These components were FIGHTING the training environment's broken signal. With a correct signal, they become noise. + +**Keep:** +- AdamW weight decay (G1) — regularization is always needed +- L1 sparse input (G8) — feature selection is always needed +- Ensemble heads — epistemic uncertainty is always useful +- Mamba2 temporal context — temporal features are always useful +- Risk branch — learned risk management is always useful +- Self-improving enrichment loop — eval-as-training is always useful + +--- + +## Implementation Order + +**PHASE 1: Episode alignment (the 45× fix)** +1. Change `config/gpu/h100.toml`: `gpu_timesteps_per_episode = 5000`, `gpu_n_episodes = 1024` +2. Modify done flag in `experience_env_step`: position-gated instead of timer +3. Modify portfolio reset at done: soft reset (keep equity, clear position) + +**PHASE 2: Reward alignment** +4. Disable rank normalization for trade-level reward +5. Write raw trade P&L to replay buffer (remove or bypass reward shaping) + +**PHASE 3: Metrics alignment** +6. Add un-annualized per-trade Sharpe logging to both training and validation +7. Report both side-by-side for direct comparison + +--- + +## Expected Outcome + +If the ONLY difference between training and validation is 1% exploration: + +``` +Current: + val_Sharpe (annualized per-bar): 25.67 + training_Sharpe (annualized per-trade): 0.57 + Gap: 45× + +After alignment: + val_Sharpe (un-annualized per-trade): ~0.21 + training_Sharpe (un-annualized per-trade): ~0.19 (99% of greedy) + Gap: ~1.1× (explained by 1% exploration) + + OR if episode length was the binding constraint: + training_Sharpe (un-annualized per-trade): ~0.15-0.21 + Gap: < 2× (close to greedy) +``` + +**Success criteria:** training un-annualized per-trade Sharpe within 2× of validation un-annualized per-trade Sharpe with the same model weights. + +--- + +## Risks + +| Risk | Mitigation | +|------|-----------| +| 5000-bar episodes exhaust VRAM | Reduce n_episodes from 4096 to 1024. Total VRAM: 1.6 GB. | +| Position-gated done produces very long episodes (model never goes flat) | Fallback: hard cap at 5000 bars. Also: the model's risk branch should learn to exit. | +| Removing rank normalization causes reward scale explosion | C51 v_min/v_max auto-scales (existing adaptive atoms). Trade P&L is bounded by position size × price range. | +| Removing reward shaping removes useful training signal | The self-improving enrichment loop provides ALL the adaptive signals (epsilon, gamma, LR, curriculum). Shaping is redundant. | +| Raw P&L has high variance → slow learning | C51 distributional head is DESIGNED for high-variance returns. 51 atoms model the distribution. | +| 12× more experience per epoch → longer epoch time | Experience collection is ~0.3s currently. At 12× more steps: ~3.6s. Still small vs 88s training. |