# Training Environment Alignment — Design Spec

**Goal:** Align the training experience collector with the validation backtest evaluator so that training Sharpe reflects REAL model performance. Currently val_Sharpe=25 and training_Sharpe=0.19 with the SAME model weights — a 45× gap caused by structural differences between the two environments, not model quality.

**Root Cause Analysis (this session):**

Three bugs found and their status:

| Bug | Impact | Status |
|-----|--------|--------|
| `raw_next` (future price) in reward path | Actions misaligned with outcomes by 1 bar | **FIXED** (commit ae5743ef4) |
| 100-bar episode length on H100 | Forces exits every 100 bars, kills long trades | **NOT FIXED** |
| Different Sharpe computation (per-trade vs per-bar, different annualization) | 45× gap is partly measurement artifact | **NOT FIXED** |

Additional issues found:

| Issue | Impact |
|-------|--------|
| Rank normalization on sparse trade-level reward | 95% of rewards are 0 (holding), ranking zeros is meaningless |
| Forced flat at episode boundary | Model can't hold positions > 100 bars |
| Portfolio state reset every 100 bars | Equity/position history lost, Mamba2 context lost |
| Metrics use different units | Training: per-trade annualized. Validation: per-bar annualized. Incomparable. |

**Architecture:** ONE unified fix that makes the training environment match the validation backtest. Not 6 separate fixes — one coherent change with clear execution order.

---

## The Single Path

### Principle: Training = Validation + Exploration

The training environment should be IDENTICAL to the validation backtest except for epsilon-greedy exploration. If a greedy policy produces Sharpe X in validation, the same policy with 1% exploration should produce ~0.99X in training. Currently it produces 0.007X (45× gap).

### Changes (in execution order):

#### 1. Episode Length: 100 → Position-Gated (no timer)

**What:** Remove the fixed 100-bar episode timer. Episodes end ONLY when the model transitions to Flat after being positioned (natural trade completion).

**Why:** The validation backtest runs continuously over 800K bars with no forced exits. The training experience collector forces exits every 100 bars. A trade that needs 200 bars gets killed at bar 100. The model learns to trade within 100-bar windows instead of learning the optimal holding period.

**How (config/gpu/h100.toml):**
```toml
# OLD: gpu_timesteps_per_episode = 100
# NEW: large enough to never be the binding constraint
gpu_timesteps_per_episode = 5000
```

**How (experience_kernels.cu env_step done flag):**
```cuda
// OLD: done at fixed timer
int time_done = (step + 1) >= episode_length;

// NEW: done ONLY when model naturally completes a trade cycle
// (was positioned → chose Flat, completing a trade)
int trade_complete = (fabsf(pre_trade_position) > 0.001f && fabsf(position) < 0.001f);
int data_done = (bar_idx >= total_bars - 1);
int done = trade_complete || data_done;
```

**V(flat) = 0 is correct:** When the model has no position, its expected unrealized P&L is zero. This gives the Bellman backup a mathematically correct terminal anchor without artificial resets.

**VRAM:** `1024 episodes × 5000 timesteps × 80 dim × 4 bytes = 1.6 GB`. Fits easily in H100 80GB. Reduce n_episodes from 4096 to 1024 to compensate for longer episodes. Total experience per epoch: `1024 × 5000 = 5.12M steps` (was `4096 × 100 = 409.6K`). 12× more experience per epoch.

**Portfolio state at done:** When `trade_complete=1`, clear position and unrealized P&L but KEEP equity (no artificial cash reset). The model starts the next "episode" with its accumulated gains/losses.

```cuda
if (trade_complete) {
    // Soft reset: clear position state, keep equity
    ps[0] = 0.0f;   // position = 0 (already flat)
    // ps[9] = equity stays (no artificial reset)
    // Cash stays at current value
    // Entry price = 0 (no active trade)
    entry_price = 0.0f;
}
```

#### 2. Remove Rank Normalization for Trade-Level Reward

**What:** Disable `reward_rank_normalize` when using trade-level reward.

**Why:** With trade-level reward, 95% of bars have reward ≈ 0 (holding cost) or exactly 0 (flat). Rank normalizing a distribution that's 95% zeros produces uniform noise — the $500 winner and the $5 winner get similar ranks. The model can't learn that bigger trades are better.

**How (gpu_experience_collector.rs):**
```rust
// OLD: always rank normalize
self.shape_rewards(&mut rewards_dst, n, reward_std, reward_mean)?;

// NEW: skip rank normalization — pass raw trade P&L directly
// The C51 distributional head handles the scale naturally.
// No normalization needed when rewards are sparse trade-level P&L.
```

The C51 atoms with adaptive positions already model the return distribution. Rank normalization was needed for dense per-bar noise — with sparse trade-level P&L, it destroys the signal.

#### 3. Align Sharpe Computation

**What:** Both training and validation compute Sharpe the SAME way — per-trade, same annualization.

**Why:** Currently training uses per-trade Sharpe with `sqrt(trades_per_year)` annualization = 121×. Validation uses per-bar Sharpe with `sqrt(bars_per_year)` annualization = 313×. These are incomparable. val_Sharpe=25 and training_Sharpe=0.57 might be SIMILAR performance measured differently.

**How (two options, implement BOTH for comparison):**

Option A — Change validation to per-trade Sharpe (match training):
```rust
// In backtest_metrics_kernel.cu or the Rust metrics aggregation:
// Compute Sharpe from per-TRADE returns instead of per-BAR returns.
// This requires tracking trade entries/exits in the backtest step loop.
```

Option B — Report un-annualized Sharpe alongside annualized:
```rust
// In training_loop.rs epoch logging:
info!("Epoch {}: Sharpe_annualized={:.2} Sharpe_raw={:.4} (per-trade, un-annualized)",
      epoch, financials.sharpe, financials.sharpe / trade_annualization);

// In validation logging:
info!("val_Sharpe_annualized={:.2} val_Sharpe_raw={:.4} (per-bar, un-annualized)",
      val_sharpe, val_sharpe / bar_annualization);
```

**Primary metric going forward:** Un-annualized per-trade Sharpe (mean_trade_return / std_trade_return). No annualization factor. Directly comparable between training and validation.

#### 4. Pass Raw Trade P&L to Replay Buffer

**What:** The reward written to the replay buffer should be the RAW trade P&L, not the shaped reward.

**Why:** The shaped reward (after DSR, commitment penalty, entropy bonus, label smoothing, drawdown penalty, capital floor) is a heavily transformed version of the actual trade outcome. The model optimizes for shaped reward but is EVALUATED on raw P&L. This creates a systematic disconnect.

**How:**
```cuda
// In experience_env_step, at the reward write:
// OLD: out_rewards[out_off] = reward;  // heavily shaped
// NEW: out_rewards[out_off] = trade_level_pnl;  // raw trade P&L

// The shaping components (DSR, drawdown penalty, etc.) should
// modify the BELLMAN TARGET, not the reward. Or be removed entirely
// and replaced by the self-improving enrichment loop which adapts
// from actual eval performance.
```

This is the most aggressive change. Alternative: keep shaping but ALSO write raw P&L to a separate buffer for accurate metrics.

---

## What This Adapts

The shaping components aren't broken — they were fighting a broken signal. With a correct signal, they ADAPT:

| Component | Adaptation |
|-----------|-----------|
| DSR reward shaping | Operate on trade-level P&L instead of per-bar returns. Drawdown penalty on TRADE P&L is useful. |
| Rank normalization | Rank ONLY non-zero rewards (actual trades). Skip the 95% zeros. Preserves relative ordering without zero-dilution. |
| Commitment penalty | Soft signal — position-gated episodes handle the heavy lifting. Keep as gentle discouragement of sub-1-bar flips. |
| Cost curriculum | Already in portfolio sim. Sigmoid ramp still useful for early exploration. No change needed. |
| Q-mean drift penalty | Remove hardcoded CUDA penalty. E1 enrichment (Q-value reality check) handles drift adaptively from eval data. |

**Keep unchanged:**
- AdamW weight decay (G1) — regularization is always needed
- L1 sparse input (G8) — feature selection is always needed
- Ensemble heads — epistemic uncertainty is always useful
- Mamba2 temporal context — temporal features are always useful
- Risk branch — learned risk management is always useful
- Self-improving enrichment loop — eval-as-training is always useful
- Homeostatic regularizer — readiness-driven adaptive penalties still valuable

---

## Implementation Order

**PHASE 1: Episode alignment (the 45× fix)**
1. Change `config/gpu/h100.toml`: `gpu_timesteps_per_episode = 5000`, `gpu_n_episodes = 1024`
2. Modify done flag in `experience_env_step`: position-gated instead of timer
3. Modify portfolio reset at done: soft reset (keep equity, clear position)

**PHASE 2: Reward alignment**
4. Adapt rank normalization: rank only non-zero rewards (skip 95% zeros from holding bars)
5. Adapt DSR: operate on trade-level P&L not per-bar returns
6. Remove hardcoded Q-mean drift penalty from c51_grad_kernel (E1 enrichment handles it adaptively)

**PHASE 3: Metrics alignment**
7. Add un-annualized per-trade Sharpe logging to both training and validation
8. Report both side-by-side for direct comparison

---

## Expected Outcome

If the ONLY difference between training and validation is 1% exploration:

```
Current:
  val_Sharpe (annualized per-bar):      25.67
  training_Sharpe (annualized per-trade): 0.57
  Gap: 45×

After alignment:
  val_Sharpe (un-annualized per-trade): ~0.21
  training_Sharpe (un-annualized per-trade): ~0.19 (99% of greedy)
  Gap: ~1.1× (explained by 1% exploration)

  OR if episode length was the binding constraint:
  training_Sharpe (un-annualized per-trade): ~0.15-0.21
  Gap: < 2× (close to greedy)
```

**Success criteria:** training un-annualized per-trade Sharpe within 2× of validation un-annualized per-trade Sharpe with the same model weights.

---

## Risks

| Risk | Mitigation |
|------|-----------|
| 5000-bar episodes exhaust VRAM | Reduce n_episodes from 4096 to 1024. Total VRAM: 1.6 GB. |
| Position-gated done produces very long episodes (model never goes flat) | Fallback: hard cap at 5000 bars. Also: the model's risk branch should learn to exit. |
| Removing rank normalization causes reward scale explosion | C51 v_min/v_max auto-scales (existing adaptive atoms). Trade P&L is bounded by position size × price range. |
| Removing reward shaping removes useful training signal | The self-improving enrichment loop provides ALL the adaptive signals (epsilon, gamma, LR, curriculum). Shaping is redundant. |
| Raw P&L has high variance → slow learning | C51 distributional head is DESIGNED for high-variance returns. 51 atoms model the distribution. |
| 12× more experience per epoch → longer epoch time | Experience collection is ~0.3s currently. At 12× more steps: ~3.6s. Still small vs 88s training. |
