foxhunt/docs/superpowers/specs/2026-04-16-training-environment-alignment-design.md

Training Environment Alignment — Design Spec

Goal: Align the training experience collector with the validation backtest evaluator so that training Sharpe reflects REAL model performance. Currently val_Sharpe=25 and training_Sharpe=0.19 with the SAME model weights — a 45× gap caused by structural differences between the two environments, not model quality.

Root Cause Analysis (this session):

Three bugs found and their status:

Bug	Impact	Status
raw_next (future price) in reward path	Actions misaligned with outcomes by 1 bar	FIXED (commit ae5743ef4)
100-bar episode length on H100	Forces exits every 100 bars, kills long trades	NOT FIXED
Different Sharpe computation (per-trade vs per-bar, different annualization)	45× gap is partly measurement artifact	NOT FIXED

Additional issues found:

Issue	Impact
Rank normalization on sparse trade-level reward	95% of rewards are 0 (holding), ranking zeros is meaningless
Forced flat at episode boundary	Model can't hold positions > 100 bars
Portfolio state reset every 100 bars	Equity/position history lost, Mamba2 context lost
Metrics use different units	Training: per-trade annualized. Validation: per-bar annualized. Incomparable.

Architecture: ONE unified fix that makes the training environment match the validation backtest. Not 6 separate fixes — one coherent change with clear execution order.

---

The Single Path

Principle: Training = Validation + Exploration

The training environment should be IDENTICAL to the validation backtest except for epsilon-greedy exploration. If a greedy policy produces Sharpe X in validation, the same policy with 1% exploration should produce ~0.99X in training. Currently it produces 0.007X (45× gap).

Changes (in execution order):

1. Episode Length: 100 → Position-Gated (no timer)

What: Remove the fixed 100-bar episode timer. Episodes end ONLY when the model transitions to Flat after being positioned (natural trade completion).

Why: The validation backtest runs continuously over 800K bars with no forced exits. The training experience collector forces exits every 100 bars. A trade that needs 200 bars gets killed at bar 100. The model learns to trade within 100-bar windows instead of learning the optimal holding period.

How (config/gpu/h100.toml):

# OLD: gpu_timesteps_per_episode = 100
# NEW: large enough to never be the binding constraint
gpu_timesteps_per_episode = 5000

How (experience_kernels.cu env_step done flag):

// OLD: done at fixed timer
int time_done = (step + 1) >= episode_length;

// NEW: done ONLY when model naturally completes a trade cycle
// (was positioned → chose Flat, completing a trade)
int trade_complete = (fabsf(pre_trade_position) > 0.001f && fabsf(position) < 0.001f);
int data_done = (bar_idx >= total_bars - 1);
int done = trade_complete || data_done;

V(flat) = 0 is correct: When the model has no position, its expected unrealized P&L is zero. This gives the Bellman backup a mathematically correct terminal anchor without artificial resets.

VRAM: 1024 episodes × 5000 timesteps × 80 dim × 4 bytes = 1.6 GB. Fits easily in H100 80GB. Reduce n_episodes from 4096 to 1024 to compensate for longer episodes. Total experience per epoch: 1024 × 5000 = 5.12M steps (was 4096 × 100 = 409.6K). 12× more experience per epoch.

Portfolio state at done: When trade_complete=1, clear position and unrealized P&L but KEEP equity (no artificial cash reset). The model starts the next "episode" with its accumulated gains/losses.

if (trade_complete) {
    // Soft reset: clear position state, keep equity
    ps[0] = 0.0f;   // position = 0 (already flat)
    // ps[9] = equity stays (no artificial reset)
    // Cash stays at current value
    // Entry price = 0 (no active trade)
    entry_price = 0.0f;
}

2. Remove Rank Normalization for Trade-Level Reward

What: Disable reward_rank_normalize when using trade-level reward.

Why: With trade-level reward, 95% of bars have reward ≈ 0 (holding cost) or exactly 0 (flat). Rank normalizing a distribution that's 95% zeros produces uniform noise — the $500 winner and the $5 winner get similar ranks. The model can't learn that bigger trades are better.

How (gpu_experience_collector.rs):

// OLD: always rank normalize
self.shape_rewards(&mut rewards_dst, n, reward_std, reward_mean)?;

// NEW: skip rank normalization — pass raw trade P&L directly
// The C51 distributional head handles the scale naturally.
// No normalization needed when rewards are sparse trade-level P&L.

The C51 atoms with adaptive positions already model the return distribution. Rank normalization was needed for dense per-bar noise — with sparse trade-level P&L, it destroys the signal.

3. Align Sharpe Computation

What: Both training and validation compute Sharpe the SAME way — per-trade, same annualization.

Why: Currently training uses per-trade Sharpe with sqrt(trades_per_year) annualization = 121×. Validation uses per-bar Sharpe with sqrt(bars_per_year) annualization = 313×. These are incomparable. val_Sharpe=25 and training_Sharpe=0.57 might be SIMILAR performance measured differently.

How (two options, implement BOTH for comparison):

Option A — Change validation to per-trade Sharpe (match training):

// In backtest_metrics_kernel.cu or the Rust metrics aggregation:
// Compute Sharpe from per-TRADE returns instead of per-BAR returns.
// This requires tracking trade entries/exits in the backtest step loop.

Option B — Report un-annualized Sharpe alongside annualized:

// In training_loop.rs epoch logging:
info!("Epoch {}: Sharpe_annualized={:.2} Sharpe_raw={:.4} (per-trade, un-annualized)",
      epoch, financials.sharpe, financials.sharpe / trade_annualization);

// In validation logging:
info!("val_Sharpe_annualized={:.2} val_Sharpe_raw={:.4} (per-bar, un-annualized)",
      val_sharpe, val_sharpe / bar_annualization);

Primary metric going forward: Un-annualized per-trade Sharpe (mean_trade_return / std_trade_return). No annualization factor. Directly comparable between training and validation.

4. Pass Raw Trade P&L to Replay Buffer

What: The reward written to the replay buffer should be the RAW trade P&L, not the shaped reward.

Why: The shaped reward (after DSR, commitment penalty, entropy bonus, label smoothing, drawdown penalty, capital floor) is a heavily transformed version of the actual trade outcome. The model optimizes for shaped reward but is EVALUATED on raw P&L. This creates a systematic disconnect.

How:

// In experience_env_step, at the reward write:
// OLD: out_rewards[out_off] = reward;  // heavily shaped
// NEW: out_rewards[out_off] = trade_level_pnl;  // raw trade P&L

// The shaping components (DSR, drawdown penalty, etc.) should
// modify the BELLMAN TARGET, not the reward. Or be removed entirely
// and replaced by the self-improving enrichment loop which adapts
// from actual eval performance.

This is the most aggressive change. Alternative: keep shaping but ALSO write raw P&L to a separate buffer for accurate metrics.

---

What This Adapts

The shaping components aren't broken — they were fighting a broken signal. With a correct signal, they ADAPT:

Component	Adaptation
DSR reward shaping	Operate on trade-level P&L instead of per-bar returns. Drawdown penalty on TRADE P&L is useful.
Rank normalization	Rank ONLY non-zero rewards (actual trades). Skip the 95% zeros. Preserves relative ordering without zero-dilution.
Commitment penalty	Soft signal — position-gated episodes handle the heavy lifting. Keep as gentle discouragement of sub-1-bar flips.
Cost curriculum	Already in portfolio sim. Sigmoid ramp still useful for early exploration. No change needed.
Q-mean drift penalty	Remove hardcoded CUDA penalty. E1 enrichment (Q-value reality check) handles drift adaptively from eval data.

Keep unchanged:

• AdamW weight decay (G1) — regularization is always needed
• L1 sparse input (G8) — feature selection is always needed
• Ensemble heads — epistemic uncertainty is always useful
• Mamba2 temporal context — temporal features are always useful
• Risk branch — learned risk management is always useful
• Self-improving enrichment loop — eval-as-training is always useful
• Homeostatic regularizer — readiness-driven adaptive penalties still valuable

---

Implementation Order

PHASE 1: Episode alignment (the 45× fix)

1. Change config/gpu/h100.toml: gpu_timesteps_per_episode = 5000, gpu_n_episodes = 1024
2. Modify done flag in experience_env_step: position-gated instead of timer
3. Modify portfolio reset at done: soft reset (keep equity, clear position)

PHASE 2: Reward alignment 4. Adapt rank normalization: rank only non-zero rewards (skip 95% zeros from holding bars) 5. Adapt DSR: operate on trade-level P&L not per-bar returns 6. Remove hardcoded Q-mean drift penalty from c51_grad_kernel (E1 enrichment handles it adaptively)

PHASE 3: Metrics alignment 7. Add un-annualized per-trade Sharpe logging to both training and validation 8. Report both side-by-side for direct comparison

---

Expected Outcome

If the ONLY difference between training and validation is 1% exploration:

Current:
  val_Sharpe (annualized per-bar):      25.67
  training_Sharpe (annualized per-trade): 0.57
  Gap: 45×

After alignment:
  val_Sharpe (un-annualized per-trade): ~0.21
  training_Sharpe (un-annualized per-trade): ~0.19 (99% of greedy)
  Gap: ~1.1× (explained by 1% exploration)

  OR if episode length was the binding constraint:
  training_Sharpe (un-annualized per-trade): ~0.15-0.21
  Gap: < 2× (close to greedy)

Success criteria: training un-annualized per-trade Sharpe within 2× of validation un-annualized per-trade Sharpe with the same model weights.

---

Risks

Risk	Mitigation
5000-bar episodes exhaust VRAM	Reduce n_episodes from 4096 to 1024. Total VRAM: 1.6 GB.
Position-gated done produces very long episodes (model never goes flat)	Fallback: hard cap at 5000 bars. Also: the model's risk branch should learn to exit.
Removing rank normalization causes reward scale explosion	C51 v_min/v_max auto-scales (existing adaptive atoms). Trade P&L is bounded by position size × price range.
Removing reward shaping removes useful training signal	The self-improving enrichment loop provides ALL the adaptive signals (epsilon, gamma, LR, curriculum). Shaping is redundant.
Raw P&L has high variance → slow learning	C51 distributional head is DESIGNED for high-variance returns. 51 atoms model the distribution.
12× more experience per epoch → longer epoch time	Experience collection is ~0.3s currently. At 12× more steps: ~3.6s. Still small vs 88s training.