# Unified Training and Validation Environments — Design

## Problem

The DQN training loop produces a model whose training Sharpe and validation
Sharpe diverge catastrophically:

| Metric | Training | Validation |
|---|---|---|
| `Sharpe_raw` (per-bar) | +0.03 to +0.07 | -0.31 to -0.39 |
| `WinRate` | 15% – 28% (oscillating) | 26% – 32% (stable) |
| `MaxDD` | 0.002% – 0.006% | 10% – 15% |
| Sharpe (annualized) | -13 to +20 (noisy) | -97 to -150 (flat) |

Measured over a 50-epoch E1 smoke test run on 2026-04-20 after the
distillation-collapse fixes landed (commit `24f96ab78`).

**Same network, same weights, same data — drastically different outcomes.**
Validation metrics are *stable* across 50 epochs (the network converges to a
deterministic losing policy) while training metrics oscillate (shaped reward
varies with exploration noise). This is not a bug in any single component;
it is the product of two environments that have drifted apart over many
incremental design decisions.

### Evidence that it's architectural, not a single bug

A reward-shaping ablation (2026-04-20, E1 smoke) closed ~30% of the gap:

- Dropping `order_credit_weight`, `urgency_credit_weight`,
  `risk_efficiency_weight`, `exit_timing_weight`, `ofi_reward_weight`,
  `kelly_sizing_weight`, `reward_noise_scale`, `cea_weight`
- Result: val `Sharpe_raw` -0.39 → -0.31, val WinRate 30% → 41%, val MaxDD
  11% → 8%

The remaining ~70% gap is architectural: the environments execute the same
weights against different trade physics.

### Concrete divergences between the two environments

| Aspect | `experience_kernels.cu` (training) | `backtest_env_kernel.cu` (validation) |
|---|---|---|
| Spread scaling | `0.5× – 2.0×` via CUSUM regime proxy (line 1395) | Fixed (no scaling; `-1.0f` override) |
| Fill model | Saboteur: per-episode `(spread_mult, fill_prob, slippage_mult)` adversarial noise | Always fill at close price |
| Transaction cost | `tx_cost_multiplier` scaled by saboteur + curriculum | Fixed `tx_cost_bps` |
| Reward composition | 18 shaping terms (DSR, DD, churn, micro-reward, credits, entropy, HER, counterfactuals, plan params, OFI, Kelly…) | Pure P&L + inventory penalty + churn penalty |
| Action selection | ε-greedy with `ε ≈ 0.02` + noisy-net parameter noise + plan-constrained | Deterministic argmax (no noise) |
| Position dynamics | `plan_params` influence target position + counterfactual flips (N4) | Pure action → target position mapping |
| Trailing stop | Fixed parameters (aligned per comment: "removed regime scaling to eliminate mismatch") | Same fixed parameters |
| Capital floor | 25% drawdown liquidation | 25% drawdown liquidation (matched) |

The trailing-stop comment at `experience_kernels.cu:1418` tells the story:
*"Regime-adaptive scaling removed to eliminate train/eval mismatch."*
Someone was here before. They aligned *some* things. Many more remain
unaligned.

## Design goals

1. **Training and validation must produce comparable scalar P&L**, up to the
   noise floor introduced by intentional randomization (exploration).
2. **Validation must be an honest measure of deployed-model behavior** —
   adding shaping to the validation env to close the gap is the wrong
   direction.
3. **Training must still be stable** — the shaping terms were added to
   address real pathologies (churn, over-trading, drawdown); stripping them
   without replacement risks regressing training dynamics.
4. **The fix must be testable within the existing smoke-test budget** (~30s
   per run on RTX 3050 Ti; 20 epochs). Otherwise iteration becomes too slow
   to converge on the right answer.

## Options considered

### Option A — Backtest matches training

Make the backtest kernel execute the full shaping stack: DSR reward, churn,
credits, saboteur noise, counterfactuals, plan params.

- **Pro**: closes the gap immediately — val Sharpe would track training
  Sharpe by construction.
- **Con**: val is no longer an honest measure of deployed P&L. Training
  metric "looks good" but model still bleeds money in the real world. This
  is strictly worse than the current state — the gap was informing us of a
  real issue, and we'd be hiding it.
- **Verdict**: **Rejected**. Measurement degradation.

### Option B — Training matches backtest (strip shaping)

Remove all shaping terms. Training reward = trading P&L + minimal safety
(drawdown penalty only).

- **Pro**: pure alignment. One source of truth for "good model."
- **Con**: the shaping terms exist because naive P&L training exhibits
  pathologies — churning, over-trading, gambling for variance, collapse in
  sparse-reward regimes. Stripping without replacement regresses to those
  failure modes.
- **Verdict**: **Rejected**. Throws the baby out with the bathwater.

### Option C — Unified environment with conditional shaping layer (RECOMMENDED)

One environment kernel that implements trade physics identically for
training and validation. Reward is a *layered* computation: a core P&L term
(identical in both modes) plus a tunable *shaping contribution* that is
**additive, P&L-aligned, and documented in units of "dollars per bar."**

Key design principles:

1. **Core reward = per-step change in portfolio equity.** Same in training
   and validation. This is the ground truth.
2. **Shaping terms must be P&L-units additive.** Each shaping term's
   contribution is measured in the same units as the core reward (dollars,
   or normalized portfolio fraction). The sum of `core + sum(shaping)` IS
   the training signal; the validation signal is `core` alone. The gap
   between training and validation Sharpe becomes a *computable* quantity:
   `E[shaping] / σ(core)` — and can be bounded by caps on shaping
   magnitudes.
3. **Non-P&L-aligned shaping is forbidden.** Terms like
   `order_credit_weight` (reward for picking a certain order-type branch
   regardless of outcome) violate this and are deleted, not disabled —
   per `feedback_no_feature_flags.md`.
4. **Adversarial/exploration noise (saboteur, ε-greedy, plan nudges) only
   modulates the core state transition, not the reward.** The reward is
   whatever actually happened — fills, slippage, costs all reflected.
   Exploration doesn't get "credit" for trying; it gets credit only for
   what its trials actually produced.
5. **Validation = training with exploration = 0 AND shaping = 0.**
   Single flag at the env entry point (a scalar mode multiplier, not a
   boolean: `exploration_scale: f32`, `shaping_scale: f32`). In training,
   both are 1.0; in validation, both are 0.0. The env kernel itself is
   *identical code* — no branches on mode.

This is the same architectural move that fixed the collapse-prevention
distillation: compute the varying signal at the source (GPU, per-step)
from a scalar mode factor baked via pinned device-mapped memory, rather
than maintaining two parallel code paths.

### Option D — Keep both environments; use both signals meaningfully

Accept the two environments as-is. Use training Sharpe for hyperparameter
search and validation Sharpe for deployment gating. Report the gap as a
model-quality metric.

- **Pro**: no code changes. Documents reality.
- **Con**: doesn't *solve* anything. Leaves the divergent environment
  structure as permanent technical debt. Future changes to one env will
  continue to drift from the other.
- **Verdict**: **Fallback only**. Choose this only if C turns out to be too
  expensive to implement.

## Recommended path: Option C

### Phase 1 — Inventory shaping terms (≤1 day)

Document every term currently summed into training reward. For each:

- What was it added for? (find the commit)
- Is its contribution P&L-aligned? (does it reward outcomes or behaviors?)
- Can it be re-expressed as an additive dollar-cost term?

Expected output: two lists.
- **Keepers**: drawdown penalty, transaction-cost penalty, inventory
  penalty, capital-floor liquidation cost — all in P&L units.
- **Removers**: credit weights, entropy bonuses, DSR, micro-reward,
  counterfactual reward scaling — all behavioral.

Store the inventory as a commit-tracked doc. This is the one-time
reckoning.

### Phase 2 — Unified __device__ core (COMPLETED 2026-04-21)

**Executed differently from the original plan — with better results.**

Original plan was: write a brand-new `unified_env_kernel.cu` that replaces
both existing kernels. Problem: the two kernels have fundamentally
different threading models (episodes vs. windows) and output buffer
shapes — a full rewrite is a 3-day project and adds risk.

What was implemented instead: a `__device__ __forceinline__` helper
`unified_env_step_core(...)` in `trade_physics.cuh` that captures the
CANONICAL STEP LOGIC shared by both kernels:

  1. Action decode (4-branch: dir, mag, order, urgency)
  2. Hold action passthrough
  3. Target position via `compute_target_position_4branch`
  4. Margin cap (`apply_margin_cap`)
  5. Kelly cap with health-coupled safety (`apply_kelly_cap`)
  6. Trailing stop check (fixed 0.005/1.0/1.0 — regime-adaptive
     is a deferred gem, see task #25)
  7. `execute_trade` with sqrt-impact (spread_scale = -1.0)
  8. `record_kelly_trade_outcome`
  9. entry_price update (new entry / reversal / flat)
 10. hold_time tick via `update_hold_time`
 11. step_return = (new_value − prev_equity) / prev_equity (PURE P&L)
 12. Post-enforcement actual_dir + actual_mag (for actions_history)

Both kernels — `experience_env_step` (training) and `backtest_env_step{,_batch}`
(validation) — call this helper for their core step. Net −116 lines
across both files; drift becomes structurally impossible because any
change to the helper applies to both atomically at compile time.

Training-specific state (saboteur, plan_params, counterfactuals, reward
shaping) wraps the call — those stay in the training kernel body, gated
by `exploration_scale` and `shaping_scale`.

### Phase 3 — Old kernel deletion (deferred)

The two kernels still exist as separate `extern "C" __global__` entry
points with different threading models + output buffer shapes. Deleting
one of them requires unifying the data layout (training uses episodes
indexed into a global targets buffer; validation uses windows with
local per-window price buffers). This is now a pure data-pipeline
refactor, not a trade-physics problem — the physics is already unified.

Deferred to a follow-up because today's work achieves the original
motivation (closing the catastrophic train/val Sharpe gap) without
needing the kernel deletion. See commits 1ffdf38dd (pure-P&L val),
aadb6c13d (WinRate by direction), a3b6bc2f3 (unified core extraction).

### Phase 4 — Re-tune adaptive-learning mechanisms (≤2 days)

The distillation, collapse-prevention, and health-EMA machinery was tuned
against the old training reward magnitudes. With shaping removed or
re-scaled, the health-signal thresholds (`SNAPSHOT_HEALTH_THRESHOLD`,
`DISTILL_HEALTH_THRESHOLD`, etc.) may need recalibration. The E1 smoke
test is the canary.

### Phase 5 — Verify on L40S full-scale run (≤1 day)

Production 50-epoch run. Expected signatures of success:

- Training `Sharpe_raw` ≥ 0 by epoch 10; ≥ +0.03 by epoch 30
- Validation `Sharpe_raw` ≥ training `Sharpe_raw` − 0.05
- Validation `MaxDD` ≤ 3× training `MaxDD`
- Validation `WinRate` tracks training `WinRate` ± 5 percentage points

If any of these fail, the analysis is wrong and we revisit Phase 1.

## Risks

1. **Collapse-prevention regresses.** The distillation mechanism was tuned
   with the existing shaping weights in the loop. Pure-P&L training may
   cause the network to find attractor states the current mechanisms don't
   cover. Mitigation: Phase 4 recalibration, E1 smoke test as the gate.
2. **Training becomes too noisy.** Sparse P&L reward may have insufficient
   gradient signal for the Q-network to learn. Mitigation: the counterfactual
   curriculum (N4) should provide dense gradient signal even with pure P&L
   reward — verify this empirically.
3. **Historical reward-shaping decisions are load-bearing in ways we don't
   see.** Some terms may have been added to fix subtle numerical instability
   (e.g., `reward_noise_scale = 0.05` — label smoothing proxy). Mitigation:
   Phase 1's per-term inventory must include the provenance commit, not just
   the semantic purpose.

## Out of scope

- Changing the underlying state representation (96-dim layout, OFI
  integration, MTF).
- Changing the network architecture (4-branch factored DQN, ensemble heads,
  IQN/C51).
- Re-tuning transaction cost parameters (`tx_cost_bps`, `spread_cost`).
  These are calibrated to actual broker execution and should remain as-is.

## Success criteria

A pull request that:

1. Deletes `experience_env_step` and `backtest_env_step` kernels in favor
   of a single unified env step.
2. Deletes the non-P&L-aligned shaping terms (from Phase 1 inventory).
3. Passes the E1 collapse-prevention smoke test with final
   `epoch_q_gap > 0.05`.
4. Shows validation `Sharpe_raw` within 0.05 of training `Sharpe_raw` by
   epoch 30 on the L40S 50-epoch production run.
5. Documents the reward semantics in a single comment block at the
   unified kernel site.

## References

- Current distillation root-cause analysis: commit `24f96ab78`
- Smoke test run showing divergence: 2026-04-20 E1 trajectory
- Reward shaping ablation result: 30% gap closure from disabling 8 credit
  weights
- Existing hint at the problem: `experience_kernels.cu:1418` comment
  *"Regime-adaptive scaling removed to eliminate train/eval mismatch"*
