spec: Training Environment Alignment — single path to close the 45× gap

Root cause: training environment differs from validation backtest in 3 ways:
1. 100-bar episodes force exits (val runs continuous) → position-gated done
2. Rank normalization destroys sparse trade-level reward → remove it
3. Different Sharpe computation (per-trade vs per-bar, different annualization)

Single execution path:
Phase 1: Episode alignment (100→5000 bars, position-gated done, soft reset)
Phase 2: Reward alignment (remove rank norm, raw trade P&L to replay)
Phase 3: Metrics alignment (un-annualized per-trade Sharpe, both paths)

Expected: training Sharpe within 2× of validation with same weights.
Removes 5 unnecessary shaping components that fought the broken signal.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

docs/superpowers/specs/2026-04-16-training-environment-alignment-design.md

@@ -0,0 +1,220 @@
+# Training Environment Alignment — Design Spec
+
+**Goal:** Align the training experience collector with the validation backtest evaluator so that training Sharpe reflects REAL model performance. Currently val_Sharpe=25 and training_Sharpe=0.19 with the SAME model weights — a 45× gap caused by structural differences between the two environments, not model quality.
+
+**Root Cause Analysis (this session):**
+
+Three bugs found and their status:
+
+| Bug | Impact | Status |
+|-----|--------|--------|
+| `raw_next` (future price) in reward path | Actions misaligned with outcomes by 1 bar | **FIXED** (commit ae5743ef4) |
+| 100-bar episode length on H100 | Forces exits every 100 bars, kills long trades | **NOT FIXED** |
+| Different Sharpe computation (per-trade vs per-bar, different annualization) | 45× gap is partly measurement artifact | **NOT FIXED** |
+
+Additional issues found:
+
+| Issue | Impact |
+|-------|--------|
+| Rank normalization on sparse trade-level reward | 95% of rewards are 0 (holding), ranking zeros is meaningless |
+| Forced flat at episode boundary | Model can't hold positions > 100 bars |
+| Portfolio state reset every 100 bars | Equity/position history lost, Mamba2 context lost |
+| Metrics use different units | Training: per-trade annualized. Validation: per-bar annualized. Incomparable. |
+
+**Architecture:** ONE unified fix that makes the training environment match the validation backtest. Not 6 separate fixes — one coherent change with clear execution order.
+
+---
+
+## The Single Path
+
+### Principle: Training = Validation + Exploration
+
+The training environment should be IDENTICAL to the validation backtest except for epsilon-greedy exploration. If a greedy policy produces Sharpe X in validation, the same policy with 1% exploration should produce ~0.99X in training. Currently it produces 0.007X (45× gap).
+
+### Changes (in execution order):
+
+#### 1. Episode Length: 100 → Position-Gated (no timer)
+
+**What:** Remove the fixed 100-bar episode timer. Episodes end ONLY when the model transitions to Flat after being positioned (natural trade completion).
+
+**Why:** The validation backtest runs continuously over 800K bars with no forced exits. The training experience collector forces exits every 100 bars. A trade that needs 200 bars gets killed at bar 100. The model learns to trade within 100-bar windows instead of learning the optimal holding period.
+
+**How (config/gpu/h100.toml):**
+```toml
+# OLD: gpu_timesteps_per_episode = 100
+# NEW: large enough to never be the binding constraint
+gpu_timesteps_per_episode = 5000
+```
+
+**How (experience_kernels.cu env_step done flag):**
+```cuda
+// OLD: done at fixed timer
+int time_done = (step + 1) >= episode_length;
+
+// NEW: done ONLY when model naturally completes a trade cycle
+// (was positioned → chose Flat, completing a trade)
+int trade_complete = (fabsf(pre_trade_position) > 0.001f && fabsf(position) < 0.001f);
+int data_done = (bar_idx >= total_bars - 1);
+int done = trade_complete || data_done;
+```
+
+**V(flat) = 0 is correct:** When the model has no position, its expected unrealized P&L is zero. This gives the Bellman backup a mathematically correct terminal anchor without artificial resets.
+
+**VRAM:** `1024 episodes × 5000 timesteps × 80 dim × 4 bytes = 1.6 GB`. Fits easily in H100 80GB. Reduce n_episodes from 4096 to 1024 to compensate for longer episodes. Total experience per epoch: `1024 × 5000 = 5.12M steps` (was `4096 × 100 = 409.6K`). 12× more experience per epoch.
+
+**Portfolio state at done:** When `trade_complete=1`, clear position and unrealized P&L but KEEP equity (no artificial cash reset). The model starts the next "episode" with its accumulated gains/losses.
+
+```cuda
+if (trade_complete) {
+    // Soft reset: clear position state, keep equity
+    ps[0] = 0.0f;   // position = 0 (already flat)
+    // ps[9] = equity stays (no artificial reset)
+    // Cash stays at current value
+    // Entry price = 0 (no active trade)
+    entry_price = 0.0f;
+}
+```
+
+#### 2. Remove Rank Normalization for Trade-Level Reward
+
+**What:** Disable `reward_rank_normalize` when using trade-level reward.
+
+**Why:** With trade-level reward, 95% of bars have reward ≈ 0 (holding cost) or exactly 0 (flat). Rank normalizing a distribution that's 95% zeros produces uniform noise — the $500 winner and the $5 winner get similar ranks. The model can't learn that bigger trades are better.
+
+**How (gpu_experience_collector.rs):**
+```rust
+// OLD: always rank normalize
+self.shape_rewards(&mut rewards_dst, n, reward_std, reward_mean)?;
+
+// NEW: skip rank normalization — pass raw trade P&L directly
+// The C51 distributional head handles the scale naturally.
+// No normalization needed when rewards are sparse trade-level P&L.
+```
+
+The C51 atoms with adaptive positions already model the return distribution. Rank normalization was needed for dense per-bar noise — with sparse trade-level P&L, it destroys the signal.
+
+#### 3. Align Sharpe Computation
+
+**What:** Both training and validation compute Sharpe the SAME way — per-trade, same annualization.
+
+**Why:** Currently training uses per-trade Sharpe with `sqrt(trades_per_year)` annualization = 121×. Validation uses per-bar Sharpe with `sqrt(bars_per_year)` annualization = 313×. These are incomparable. val_Sharpe=25 and training_Sharpe=0.57 might be SIMILAR performance measured differently.
+
+**How (two options, implement BOTH for comparison):**
+
+Option A — Change validation to per-trade Sharpe (match training):
+```rust
+// In backtest_metrics_kernel.cu or the Rust metrics aggregation:
+// Compute Sharpe from per-TRADE returns instead of per-BAR returns.
+// This requires tracking trade entries/exits in the backtest step loop.
+```
+
+Option B — Report un-annualized Sharpe alongside annualized:
+```rust
+// In training_loop.rs epoch logging:
+info!("Epoch {}: Sharpe_annualized={:.2} Sharpe_raw={:.4} (per-trade, un-annualized)",
+      epoch, financials.sharpe, financials.sharpe / trade_annualization);
+
+// In validation logging:
+info!("val_Sharpe_annualized={:.2} val_Sharpe_raw={:.4} (per-bar, un-annualized)",
+      val_sharpe, val_sharpe / bar_annualization);
+```
+
+**Primary metric going forward:** Un-annualized per-trade Sharpe (mean_trade_return / std_trade_return). No annualization factor. Directly comparable between training and validation.
+
+#### 4. Pass Raw Trade P&L to Replay Buffer
+
+**What:** The reward written to the replay buffer should be the RAW trade P&L, not the shaped reward.
+
+**Why:** The shaped reward (after DSR, commitment penalty, entropy bonus, label smoothing, drawdown penalty, capital floor) is a heavily transformed version of the actual trade outcome. The model optimizes for shaped reward but is EVALUATED on raw P&L. This creates a systematic disconnect.
+
+**How:**
+```cuda
+// In experience_env_step, at the reward write:
+// OLD: out_rewards[out_off] = reward;  // heavily shaped
+// NEW: out_rewards[out_off] = trade_level_pnl;  // raw trade P&L
+
+// The shaping components (DSR, drawdown penalty, etc.) should
+// modify the BELLMAN TARGET, not the reward. Or be removed entirely
+// and replaced by the self-improving enrichment loop which adapts
+// from actual eval performance.
+```
+
+This is the most aggressive change. Alternative: keep shaping but ALSO write raw P&L to a separate buffer for accurate metrics.
+
+---
+
+## What This Removes
+
+The following components become UNNECESSARY with this alignment:
+
+| Component | Why unnecessary |
+|-----------|----------------|
+| Adaptive DSR reward shaping (G2) | Raw P&L is the reward — no shaping needed |
+| Rank normalization (G13) | Sparse trade-level P&L shouldn't be ranked |
+| Commitment penalty (G15) | Natural trade lifecycle replaces artificial penalty |
+| Cost curriculum sigmoid | Costs are in the raw P&L from the portfolio sim |
+| Q-mean drift penalty | Q-values should align with raw P&L naturally |
+
+These components were FIGHTING the training environment's broken signal. With a correct signal, they become noise.
+
+**Keep:**
+- AdamW weight decay (G1) — regularization is always needed
+- L1 sparse input (G8) — feature selection is always needed
+- Ensemble heads — epistemic uncertainty is always useful
+- Mamba2 temporal context — temporal features are always useful
+- Risk branch — learned risk management is always useful
+- Self-improving enrichment loop — eval-as-training is always useful
+
+---
+
+## Implementation Order
+
+**PHASE 1: Episode alignment (the 45× fix)**
+1. Change `config/gpu/h100.toml`: `gpu_timesteps_per_episode = 5000`, `gpu_n_episodes = 1024`
+2. Modify done flag in `experience_env_step`: position-gated instead of timer
+3. Modify portfolio reset at done: soft reset (keep equity, clear position)
+
+**PHASE 2: Reward alignment**
+4. Disable rank normalization for trade-level reward
+5. Write raw trade P&L to replay buffer (remove or bypass reward shaping)
+
+**PHASE 3: Metrics alignment**
+6. Add un-annualized per-trade Sharpe logging to both training and validation
+7. Report both side-by-side for direct comparison
+
+---
+
+## Expected Outcome
+
+If the ONLY difference between training and validation is 1% exploration:
+
+```
+Current:
+  val_Sharpe (annualized per-bar):      25.67
+  training_Sharpe (annualized per-trade): 0.57
+  Gap: 45×
+
+After alignment:
+  val_Sharpe (un-annualized per-trade): ~0.21
+  training_Sharpe (un-annualized per-trade): ~0.19 (99% of greedy)
+  Gap: ~1.1× (explained by 1% exploration)
+
+  OR if episode length was the binding constraint:
+  training_Sharpe (un-annualized per-trade): ~0.15-0.21
+  Gap: < 2× (close to greedy)
+```
+
+**Success criteria:** training un-annualized per-trade Sharpe within 2× of validation un-annualized per-trade Sharpe with the same model weights.
+
+---
+
+## Risks
+
+| Risk | Mitigation |
+|------|-----------|
+| 5000-bar episodes exhaust VRAM | Reduce n_episodes from 4096 to 1024. Total VRAM: 1.6 GB. |
+| Position-gated done produces very long episodes (model never goes flat) | Fallback: hard cap at 5000 bars. Also: the model's risk branch should learn to exit. |
+| Removing rank normalization causes reward scale explosion | C51 v_min/v_max auto-scales (existing adaptive atoms). Trade P&L is bounded by position size × price range. |
+| Removing reward shaping removes useful training signal | The self-improving enrichment loop provides ALL the adaptive signals (epsilon, gamma, LR, curriculum). Shaping is redundant. |
+| Raw P&L has high variance → slow learning | C51 distributional head is DESIGNED for high-variance returns. 51 atoms model the distribution. |
+| 12× more experience per epoch → longer epoch time | Experience collection is ~0.3s currently. At 12× more steps: ~3.6s. Still small vs 88s training. |