docs(design): unify training and validation environments
Design document for the follow-up to the adaptive-learning-rootcause session. Lays out the evidence, root cause, options considered, and recommended path for closing the 70% structural gap between training and validation Sharpe that remained after the distillation collapse fix landed. Key findings documented: - Reward-shaping ablation (2026-04-20) closed ~30% of the gap; ~70% remains architectural - Two env kernels (experience_env_step vs backtest_env_step) have drifted: spread scaling, fill model, saboteur noise, reward terms, action selection, position dynamics all differ - Hint from history: experience_kernels.cu:1418 comment "Regime- adaptive scaling removed to eliminate train/eval mismatch" shows someone aligned *some* things previously Recommended path (Option C, "unified env with layered reward"): - Single unified_env_step kernel replaces both - Core reward = pure P&L; shaping is additive and P&L-units-aligned - Validation = training with exploration_scale=0 AND shaping_scale=0 - Scale factors are pinned device-mapped scalars (same pattern used by the distillation alpha fix) Phased implementation plan with ~8-day budget and concrete success criteria: validation Sharpe_raw within 0.05 of training Sharpe_raw by epoch 30 on L40S production run. Rejected alternatives: backtest-matches-training (hides real issue), training-matches-backtest (regresses stability), two-environment with divergence as metric (fallback only). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
@@ -0,0 +1,254 @@
|
|||||||
|
# Unified Training and Validation Environments — Design
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
The DQN training loop produces a model whose training Sharpe and validation
|
||||||
|
Sharpe diverge catastrophically:
|
||||||
|
|
||||||
|
| Metric | Training | Validation |
|
||||||
|
|---|---|---|
|
||||||
|
| `Sharpe_raw` (per-bar) | +0.03 to +0.07 | -0.31 to -0.39 |
|
||||||
|
| `WinRate` | 15% – 28% (oscillating) | 26% – 32% (stable) |
|
||||||
|
| `MaxDD` | 0.002% – 0.006% | 10% – 15% |
|
||||||
|
| Sharpe (annualized) | -13 to +20 (noisy) | -97 to -150 (flat) |
|
||||||
|
|
||||||
|
Measured over a 50-epoch E1 smoke test run on 2026-04-20 after the
|
||||||
|
distillation-collapse fixes landed (commit `24f96ab78`).
|
||||||
|
|
||||||
|
**Same network, same weights, same data — drastically different outcomes.**
|
||||||
|
Validation metrics are *stable* across 50 epochs (the network converges to a
|
||||||
|
deterministic losing policy) while training metrics oscillate (shaped reward
|
||||||
|
varies with exploration noise). This is not a bug in any single component;
|
||||||
|
it is the product of two environments that have drifted apart over many
|
||||||
|
incremental design decisions.
|
||||||
|
|
||||||
|
### Evidence that it's architectural, not a single bug
|
||||||
|
|
||||||
|
A reward-shaping ablation (2026-04-20, E1 smoke) closed ~30% of the gap:
|
||||||
|
|
||||||
|
- Dropping `order_credit_weight`, `urgency_credit_weight`,
|
||||||
|
`risk_efficiency_weight`, `exit_timing_weight`, `ofi_reward_weight`,
|
||||||
|
`kelly_sizing_weight`, `reward_noise_scale`, `cea_weight`
|
||||||
|
- Result: val `Sharpe_raw` -0.39 → -0.31, val WinRate 30% → 41%, val MaxDD
|
||||||
|
11% → 8%
|
||||||
|
|
||||||
|
The remaining ~70% gap is architectural: the environments execute the same
|
||||||
|
weights against different trade physics.
|
||||||
|
|
||||||
|
### Concrete divergences between the two environments
|
||||||
|
|
||||||
|
| Aspect | `experience_kernels.cu` (training) | `backtest_env_kernel.cu` (validation) |
|
||||||
|
|---|---|---|
|
||||||
|
| Spread scaling | `0.5× – 2.0×` via CUSUM regime proxy (line 1395) | Fixed (no scaling; `-1.0f` override) |
|
||||||
|
| Fill model | Saboteur: per-episode `(spread_mult, fill_prob, slippage_mult)` adversarial noise | Always fill at close price |
|
||||||
|
| Transaction cost | `tx_cost_multiplier` scaled by saboteur + curriculum | Fixed `tx_cost_bps` |
|
||||||
|
| Reward composition | 18 shaping terms (DSR, DD, churn, micro-reward, credits, entropy, HER, counterfactuals, plan params, OFI, Kelly…) | Pure P&L + inventory penalty + churn penalty |
|
||||||
|
| Action selection | ε-greedy with `ε ≈ 0.02` + noisy-net parameter noise + plan-constrained | Deterministic argmax (no noise) |
|
||||||
|
| Position dynamics | `plan_params` influence target position + counterfactual flips (N4) | Pure action → target position mapping |
|
||||||
|
| Trailing stop | Fixed parameters (aligned per comment: "removed regime scaling to eliminate mismatch") | Same fixed parameters |
|
||||||
|
| Capital floor | 25% drawdown liquidation | 25% drawdown liquidation (matched) |
|
||||||
|
|
||||||
|
The trailing-stop comment at `experience_kernels.cu:1418` tells the story:
|
||||||
|
*"Regime-adaptive scaling removed to eliminate train/eval mismatch."*
|
||||||
|
Someone was here before. They aligned *some* things. Many more remain
|
||||||
|
unaligned.
|
||||||
|
|
||||||
|
## Design goals
|
||||||
|
|
||||||
|
1. **Training and validation must produce comparable scalar P&L**, up to the
|
||||||
|
noise floor introduced by intentional randomization (exploration).
|
||||||
|
2. **Validation must be an honest measure of deployed-model behavior** —
|
||||||
|
adding shaping to the validation env to close the gap is the wrong
|
||||||
|
direction.
|
||||||
|
3. **Training must still be stable** — the shaping terms were added to
|
||||||
|
address real pathologies (churn, over-trading, drawdown); stripping them
|
||||||
|
without replacement risks regressing training dynamics.
|
||||||
|
4. **The fix must be testable within the existing smoke-test budget** (~30s
|
||||||
|
per run on RTX 3050 Ti; 20 epochs). Otherwise iteration becomes too slow
|
||||||
|
to converge on the right answer.
|
||||||
|
|
||||||
|
## Options considered
|
||||||
|
|
||||||
|
### Option A — Backtest matches training
|
||||||
|
|
||||||
|
Make the backtest kernel execute the full shaping stack: DSR reward, churn,
|
||||||
|
credits, saboteur noise, counterfactuals, plan params.
|
||||||
|
|
||||||
|
- **Pro**: closes the gap immediately — val Sharpe would track training
|
||||||
|
Sharpe by construction.
|
||||||
|
- **Con**: val is no longer an honest measure of deployed P&L. Training
|
||||||
|
metric "looks good" but model still bleeds money in the real world. This
|
||||||
|
is strictly worse than the current state — the gap was informing us of a
|
||||||
|
real issue, and we'd be hiding it.
|
||||||
|
- **Verdict**: **Rejected**. Measurement degradation.
|
||||||
|
|
||||||
|
### Option B — Training matches backtest (strip shaping)
|
||||||
|
|
||||||
|
Remove all shaping terms. Training reward = trading P&L + minimal safety
|
||||||
|
(drawdown penalty only).
|
||||||
|
|
||||||
|
- **Pro**: pure alignment. One source of truth for "good model."
|
||||||
|
- **Con**: the shaping terms exist because naive P&L training exhibits
|
||||||
|
pathologies — churning, over-trading, gambling for variance, collapse in
|
||||||
|
sparse-reward regimes. Stripping without replacement regresses to those
|
||||||
|
failure modes.
|
||||||
|
- **Verdict**: **Rejected**. Throws the baby out with the bathwater.
|
||||||
|
|
||||||
|
### Option C — Unified environment with conditional shaping layer (RECOMMENDED)
|
||||||
|
|
||||||
|
One environment kernel that implements trade physics identically for
|
||||||
|
training and validation. Reward is a *layered* computation: a core P&L term
|
||||||
|
(identical in both modes) plus a tunable *shaping contribution* that is
|
||||||
|
**additive, P&L-aligned, and documented in units of "dollars per bar."**
|
||||||
|
|
||||||
|
Key design principles:
|
||||||
|
|
||||||
|
1. **Core reward = per-step change in portfolio equity.** Same in training
|
||||||
|
and validation. This is the ground truth.
|
||||||
|
2. **Shaping terms must be P&L-units additive.** Each shaping term's
|
||||||
|
contribution is measured in the same units as the core reward (dollars,
|
||||||
|
or normalized portfolio fraction). The sum of `core + sum(shaping)` IS
|
||||||
|
the training signal; the validation signal is `core` alone. The gap
|
||||||
|
between training and validation Sharpe becomes a *computable* quantity:
|
||||||
|
`E[shaping] / σ(core)` — and can be bounded by caps on shaping
|
||||||
|
magnitudes.
|
||||||
|
3. **Non-P&L-aligned shaping is forbidden.** Terms like
|
||||||
|
`order_credit_weight` (reward for picking a certain order-type branch
|
||||||
|
regardless of outcome) violate this and are deleted, not disabled —
|
||||||
|
per `feedback_no_feature_flags.md`.
|
||||||
|
4. **Adversarial/exploration noise (saboteur, ε-greedy, plan nudges) only
|
||||||
|
modulates the core state transition, not the reward.** The reward is
|
||||||
|
whatever actually happened — fills, slippage, costs all reflected.
|
||||||
|
Exploration doesn't get "credit" for trying; it gets credit only for
|
||||||
|
what its trials actually produced.
|
||||||
|
5. **Validation = training with exploration = 0 AND shaping = 0.**
|
||||||
|
Single flag at the env entry point (a scalar mode multiplier, not a
|
||||||
|
boolean: `exploration_scale: f32`, `shaping_scale: f32`). In training,
|
||||||
|
both are 1.0; in validation, both are 0.0. The env kernel itself is
|
||||||
|
*identical code* — no branches on mode.
|
||||||
|
|
||||||
|
This is the same architectural move that fixed the collapse-prevention
|
||||||
|
distillation: compute the varying signal at the source (GPU, per-step)
|
||||||
|
from a scalar mode factor baked via pinned device-mapped memory, rather
|
||||||
|
than maintaining two parallel code paths.
|
||||||
|
|
||||||
|
### Option D — Keep both environments; use both signals meaningfully
|
||||||
|
|
||||||
|
Accept the two environments as-is. Use training Sharpe for hyperparameter
|
||||||
|
search and validation Sharpe for deployment gating. Report the gap as a
|
||||||
|
model-quality metric.
|
||||||
|
|
||||||
|
- **Pro**: no code changes. Documents reality.
|
||||||
|
- **Con**: doesn't *solve* anything. Leaves the divergent environment
|
||||||
|
structure as permanent technical debt. Future changes to one env will
|
||||||
|
continue to drift from the other.
|
||||||
|
- **Verdict**: **Fallback only**. Choose this only if C turns out to be too
|
||||||
|
expensive to implement.
|
||||||
|
|
||||||
|
## Recommended path: Option C
|
||||||
|
|
||||||
|
### Phase 1 — Inventory shaping terms (≤1 day)
|
||||||
|
|
||||||
|
Document every term currently summed into training reward. For each:
|
||||||
|
|
||||||
|
- What was it added for? (find the commit)
|
||||||
|
- Is its contribution P&L-aligned? (does it reward outcomes or behaviors?)
|
||||||
|
- Can it be re-expressed as an additive dollar-cost term?
|
||||||
|
|
||||||
|
Expected output: two lists.
|
||||||
|
- **Keepers**: drawdown penalty, transaction-cost penalty, inventory
|
||||||
|
penalty, capital-floor liquidation cost — all in P&L units.
|
||||||
|
- **Removers**: credit weights, entropy bonuses, DSR, micro-reward,
|
||||||
|
counterfactual reward scaling — all behavioral.
|
||||||
|
|
||||||
|
Store the inventory as a commit-tracked doc. This is the one-time
|
||||||
|
reckoning.
|
||||||
|
|
||||||
|
### Phase 2 — Rewrite env as a single kernel (≤3 days)
|
||||||
|
|
||||||
|
New file: `crates/ml/src/cuda_pipeline/unified_env_kernel.cu`. One
|
||||||
|
`__global__ void unified_env_step(...)` that:
|
||||||
|
|
||||||
|
- Takes the exploration/shaping scale scalars as `const float*` device
|
||||||
|
pointers (pinned device-mapped, same pattern as distillation alpha).
|
||||||
|
- Computes trade physics once.
|
||||||
|
- Computes reward = `pnl_per_step + shaping_scale × shaping_bundle`.
|
||||||
|
- Writes `step_return = pnl_per_step` to a separate output buffer — this
|
||||||
|
is the **authentic** P&L signal used by validation regardless of
|
||||||
|
training/validation mode.
|
||||||
|
|
||||||
|
Both training and validation call the same kernel. The only difference is
|
||||||
|
the scalar values at the pinned pointers.
|
||||||
|
|
||||||
|
### Phase 3 — Deprecate the two old kernels (≤1 day)
|
||||||
|
|
||||||
|
Delete `experience_env_step_batch` and `backtest_env_step_batch`. Rewire
|
||||||
|
all callers to the unified kernel. Remove any dead utility code that was
|
||||||
|
specific to one path.
|
||||||
|
|
||||||
|
### Phase 4 — Re-tune adaptive-learning mechanisms (≤2 days)
|
||||||
|
|
||||||
|
The distillation, collapse-prevention, and health-EMA machinery was tuned
|
||||||
|
against the old training reward magnitudes. With shaping removed or
|
||||||
|
re-scaled, the health-signal thresholds (`SNAPSHOT_HEALTH_THRESHOLD`,
|
||||||
|
`DISTILL_HEALTH_THRESHOLD`, etc.) may need recalibration. The E1 smoke
|
||||||
|
test is the canary.
|
||||||
|
|
||||||
|
### Phase 5 — Verify on L40S full-scale run (≤1 day)
|
||||||
|
|
||||||
|
Production 50-epoch run. Expected signatures of success:
|
||||||
|
|
||||||
|
- Training `Sharpe_raw` ≥ 0 by epoch 10; ≥ +0.03 by epoch 30
|
||||||
|
- Validation `Sharpe_raw` ≥ training `Sharpe_raw` − 0.05
|
||||||
|
- Validation `MaxDD` ≤ 3× training `MaxDD`
|
||||||
|
- Validation `WinRate` tracks training `WinRate` ± 5 percentage points
|
||||||
|
|
||||||
|
If any of these fail, the analysis is wrong and we revisit Phase 1.
|
||||||
|
|
||||||
|
## Risks
|
||||||
|
|
||||||
|
1. **Collapse-prevention regresses.** The distillation mechanism was tuned
|
||||||
|
with the existing shaping weights in the loop. Pure-P&L training may
|
||||||
|
cause the network to find attractor states the current mechanisms don't
|
||||||
|
cover. Mitigation: Phase 4 recalibration, E1 smoke test as the gate.
|
||||||
|
2. **Training becomes too noisy.** Sparse P&L reward may have insufficient
|
||||||
|
gradient signal for the Q-network to learn. Mitigation: the counterfactual
|
||||||
|
curriculum (N4) should provide dense gradient signal even with pure P&L
|
||||||
|
reward — verify this empirically.
|
||||||
|
3. **Historical reward-shaping decisions are load-bearing in ways we don't
|
||||||
|
see.** Some terms may have been added to fix subtle numerical instability
|
||||||
|
(e.g., `reward_noise_scale = 0.05` — label smoothing proxy). Mitigation:
|
||||||
|
Phase 1's per-term inventory must include the provenance commit, not just
|
||||||
|
the semantic purpose.
|
||||||
|
|
||||||
|
## Out of scope
|
||||||
|
|
||||||
|
- Changing the underlying state representation (96-dim layout, OFI
|
||||||
|
integration, MTF).
|
||||||
|
- Changing the network architecture (4-branch factored DQN, ensemble heads,
|
||||||
|
IQN/C51).
|
||||||
|
- Re-tuning transaction cost parameters (`tx_cost_bps`, `spread_cost`).
|
||||||
|
These are calibrated to actual broker execution and should remain as-is.
|
||||||
|
|
||||||
|
## Success criteria
|
||||||
|
|
||||||
|
A pull request that:
|
||||||
|
|
||||||
|
1. Deletes `experience_env_step` and `backtest_env_step` kernels in favor
|
||||||
|
of a single unified env step.
|
||||||
|
2. Deletes the non-P&L-aligned shaping terms (from Phase 1 inventory).
|
||||||
|
3. Passes the E1 collapse-prevention smoke test with final
|
||||||
|
`epoch_q_gap > 0.05`.
|
||||||
|
4. Shows validation `Sharpe_raw` within 0.05 of training `Sharpe_raw` by
|
||||||
|
epoch 30 on the L40S 50-epoch production run.
|
||||||
|
5. Documents the reward semantics in a single comment block at the
|
||||||
|
unified kernel site.
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- Current distillation root-cause analysis: commit `24f96ab78`
|
||||||
|
- Smoke test run showing divergence: 2026-04-20 E1 trajectory
|
||||||
|
- Reward shaping ablation result: 30% gap closure from disabling 8 credit
|
||||||
|
weights
|
||||||
|
- Existing hint at the problem: `experience_kernels.cu:1418` comment
|
||||||
|
*"Regime-adaptive scaling removed to eliminate train/eval mismatch"*
|
||||||
Reference in New Issue
Block a user