docs(design): unify training and validation environments
Design document for the follow-up to the adaptive-learning-rootcause session. Lays out the evidence, root cause, options considered, and recommended path for closing the 70% structural gap between training and validation Sharpe that remained after the distillation collapse fix landed. Key findings documented: - Reward-shaping ablation (2026-04-20) closed ~30% of the gap; ~70% remains architectural - Two env kernels (experience_env_step vs backtest_env_step) have drifted: spread scaling, fill model, saboteur noise, reward terms, action selection, position dynamics all differ - Hint from history: experience_kernels.cu:1418 comment "Regime- adaptive scaling removed to eliminate train/eval mismatch" shows someone aligned *some* things previously Recommended path (Option C, "unified env with layered reward"): - Single unified_env_step kernel replaces both - Core reward = pure P&L; shaping is additive and P&L-units-aligned - Validation = training with exploration_scale=0 AND shaping_scale=0 - Scale factors are pinned device-mapped scalars (same pattern used by the distillation alpha fix) Phased implementation plan with ~8-day budget and concrete success criteria: validation Sharpe_raw within 0.05 of training Sharpe_raw by epoch 30 on L40S production run. Rejected alternatives: backtest-matches-training (hides real issue), training-matches-backtest (regresses stability), two-environment with divergence as metric (fallback only). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
@@ -0,0 +1,254 @@
|
||||
# Unified Training and Validation Environments — Design
|
||||
|
||||
## Problem
|
||||
|
||||
The DQN training loop produces a model whose training Sharpe and validation
|
||||
Sharpe diverge catastrophically:
|
||||
|
||||
| Metric | Training | Validation |
|
||||
|---|---|---|
|
||||
| `Sharpe_raw` (per-bar) | +0.03 to +0.07 | -0.31 to -0.39 |
|
||||
| `WinRate` | 15% – 28% (oscillating) | 26% – 32% (stable) |
|
||||
| `MaxDD` | 0.002% – 0.006% | 10% – 15% |
|
||||
| Sharpe (annualized) | -13 to +20 (noisy) | -97 to -150 (flat) |
|
||||
|
||||
Measured over a 50-epoch E1 smoke test run on 2026-04-20 after the
|
||||
distillation-collapse fixes landed (commit `24f96ab78`).
|
||||
|
||||
**Same network, same weights, same data — drastically different outcomes.**
|
||||
Validation metrics are *stable* across 50 epochs (the network converges to a
|
||||
deterministic losing policy) while training metrics oscillate (shaped reward
|
||||
varies with exploration noise). This is not a bug in any single component;
|
||||
it is the product of two environments that have drifted apart over many
|
||||
incremental design decisions.
|
||||
|
||||
### Evidence that it's architectural, not a single bug
|
||||
|
||||
A reward-shaping ablation (2026-04-20, E1 smoke) closed ~30% of the gap:
|
||||
|
||||
- Dropping `order_credit_weight`, `urgency_credit_weight`,
|
||||
`risk_efficiency_weight`, `exit_timing_weight`, `ofi_reward_weight`,
|
||||
`kelly_sizing_weight`, `reward_noise_scale`, `cea_weight`
|
||||
- Result: val `Sharpe_raw` -0.39 → -0.31, val WinRate 30% → 41%, val MaxDD
|
||||
11% → 8%
|
||||
|
||||
The remaining ~70% gap is architectural: the environments execute the same
|
||||
weights against different trade physics.
|
||||
|
||||
### Concrete divergences between the two environments
|
||||
|
||||
| Aspect | `experience_kernels.cu` (training) | `backtest_env_kernel.cu` (validation) |
|
||||
|---|---|---|
|
||||
| Spread scaling | `0.5× – 2.0×` via CUSUM regime proxy (line 1395) | Fixed (no scaling; `-1.0f` override) |
|
||||
| Fill model | Saboteur: per-episode `(spread_mult, fill_prob, slippage_mult)` adversarial noise | Always fill at close price |
|
||||
| Transaction cost | `tx_cost_multiplier` scaled by saboteur + curriculum | Fixed `tx_cost_bps` |
|
||||
| Reward composition | 18 shaping terms (DSR, DD, churn, micro-reward, credits, entropy, HER, counterfactuals, plan params, OFI, Kelly…) | Pure P&L + inventory penalty + churn penalty |
|
||||
| Action selection | ε-greedy with `ε ≈ 0.02` + noisy-net parameter noise + plan-constrained | Deterministic argmax (no noise) |
|
||||
| Position dynamics | `plan_params` influence target position + counterfactual flips (N4) | Pure action → target position mapping |
|
||||
| Trailing stop | Fixed parameters (aligned per comment: "removed regime scaling to eliminate mismatch") | Same fixed parameters |
|
||||
| Capital floor | 25% drawdown liquidation | 25% drawdown liquidation (matched) |
|
||||
|
||||
The trailing-stop comment at `experience_kernels.cu:1418` tells the story:
|
||||
*"Regime-adaptive scaling removed to eliminate train/eval mismatch."*
|
||||
Someone was here before. They aligned *some* things. Many more remain
|
||||
unaligned.
|
||||
|
||||
## Design goals
|
||||
|
||||
1. **Training and validation must produce comparable scalar P&L**, up to the
|
||||
noise floor introduced by intentional randomization (exploration).
|
||||
2. **Validation must be an honest measure of deployed-model behavior** —
|
||||
adding shaping to the validation env to close the gap is the wrong
|
||||
direction.
|
||||
3. **Training must still be stable** — the shaping terms were added to
|
||||
address real pathologies (churn, over-trading, drawdown); stripping them
|
||||
without replacement risks regressing training dynamics.
|
||||
4. **The fix must be testable within the existing smoke-test budget** (~30s
|
||||
per run on RTX 3050 Ti; 20 epochs). Otherwise iteration becomes too slow
|
||||
to converge on the right answer.
|
||||
|
||||
## Options considered
|
||||
|
||||
### Option A — Backtest matches training
|
||||
|
||||
Make the backtest kernel execute the full shaping stack: DSR reward, churn,
|
||||
credits, saboteur noise, counterfactuals, plan params.
|
||||
|
||||
- **Pro**: closes the gap immediately — val Sharpe would track training
|
||||
Sharpe by construction.
|
||||
- **Con**: val is no longer an honest measure of deployed P&L. Training
|
||||
metric "looks good" but model still bleeds money in the real world. This
|
||||
is strictly worse than the current state — the gap was informing us of a
|
||||
real issue, and we'd be hiding it.
|
||||
- **Verdict**: **Rejected**. Measurement degradation.
|
||||
|
||||
### Option B — Training matches backtest (strip shaping)
|
||||
|
||||
Remove all shaping terms. Training reward = trading P&L + minimal safety
|
||||
(drawdown penalty only).
|
||||
|
||||
- **Pro**: pure alignment. One source of truth for "good model."
|
||||
- **Con**: the shaping terms exist because naive P&L training exhibits
|
||||
pathologies — churning, over-trading, gambling for variance, collapse in
|
||||
sparse-reward regimes. Stripping without replacement regresses to those
|
||||
failure modes.
|
||||
- **Verdict**: **Rejected**. Throws the baby out with the bathwater.
|
||||
|
||||
### Option C — Unified environment with conditional shaping layer (RECOMMENDED)
|
||||
|
||||
One environment kernel that implements trade physics identically for
|
||||
training and validation. Reward is a *layered* computation: a core P&L term
|
||||
(identical in both modes) plus a tunable *shaping contribution* that is
|
||||
**additive, P&L-aligned, and documented in units of "dollars per bar."**
|
||||
|
||||
Key design principles:
|
||||
|
||||
1. **Core reward = per-step change in portfolio equity.** Same in training
|
||||
and validation. This is the ground truth.
|
||||
2. **Shaping terms must be P&L-units additive.** Each shaping term's
|
||||
contribution is measured in the same units as the core reward (dollars,
|
||||
or normalized portfolio fraction). The sum of `core + sum(shaping)` IS
|
||||
the training signal; the validation signal is `core` alone. The gap
|
||||
between training and validation Sharpe becomes a *computable* quantity:
|
||||
`E[shaping] / σ(core)` — and can be bounded by caps on shaping
|
||||
magnitudes.
|
||||
3. **Non-P&L-aligned shaping is forbidden.** Terms like
|
||||
`order_credit_weight` (reward for picking a certain order-type branch
|
||||
regardless of outcome) violate this and are deleted, not disabled —
|
||||
per `feedback_no_feature_flags.md`.
|
||||
4. **Adversarial/exploration noise (saboteur, ε-greedy, plan nudges) only
|
||||
modulates the core state transition, not the reward.** The reward is
|
||||
whatever actually happened — fills, slippage, costs all reflected.
|
||||
Exploration doesn't get "credit" for trying; it gets credit only for
|
||||
what its trials actually produced.
|
||||
5. **Validation = training with exploration = 0 AND shaping = 0.**
|
||||
Single flag at the env entry point (a scalar mode multiplier, not a
|
||||
boolean: `exploration_scale: f32`, `shaping_scale: f32`). In training,
|
||||
both are 1.0; in validation, both are 0.0. The env kernel itself is
|
||||
*identical code* — no branches on mode.
|
||||
|
||||
This is the same architectural move that fixed the collapse-prevention
|
||||
distillation: compute the varying signal at the source (GPU, per-step)
|
||||
from a scalar mode factor baked via pinned device-mapped memory, rather
|
||||
than maintaining two parallel code paths.
|
||||
|
||||
### Option D — Keep both environments; use both signals meaningfully
|
||||
|
||||
Accept the two environments as-is. Use training Sharpe for hyperparameter
|
||||
search and validation Sharpe for deployment gating. Report the gap as a
|
||||
model-quality metric.
|
||||
|
||||
- **Pro**: no code changes. Documents reality.
|
||||
- **Con**: doesn't *solve* anything. Leaves the divergent environment
|
||||
structure as permanent technical debt. Future changes to one env will
|
||||
continue to drift from the other.
|
||||
- **Verdict**: **Fallback only**. Choose this only if C turns out to be too
|
||||
expensive to implement.
|
||||
|
||||
## Recommended path: Option C
|
||||
|
||||
### Phase 1 — Inventory shaping terms (≤1 day)
|
||||
|
||||
Document every term currently summed into training reward. For each:
|
||||
|
||||
- What was it added for? (find the commit)
|
||||
- Is its contribution P&L-aligned? (does it reward outcomes or behaviors?)
|
||||
- Can it be re-expressed as an additive dollar-cost term?
|
||||
|
||||
Expected output: two lists.
|
||||
- **Keepers**: drawdown penalty, transaction-cost penalty, inventory
|
||||
penalty, capital-floor liquidation cost — all in P&L units.
|
||||
- **Removers**: credit weights, entropy bonuses, DSR, micro-reward,
|
||||
counterfactual reward scaling — all behavioral.
|
||||
|
||||
Store the inventory as a commit-tracked doc. This is the one-time
|
||||
reckoning.
|
||||
|
||||
### Phase 2 — Rewrite env as a single kernel (≤3 days)
|
||||
|
||||
New file: `crates/ml/src/cuda_pipeline/unified_env_kernel.cu`. One
|
||||
`__global__ void unified_env_step(...)` that:
|
||||
|
||||
- Takes the exploration/shaping scale scalars as `const float*` device
|
||||
pointers (pinned device-mapped, same pattern as distillation alpha).
|
||||
- Computes trade physics once.
|
||||
- Computes reward = `pnl_per_step + shaping_scale × shaping_bundle`.
|
||||
- Writes `step_return = pnl_per_step` to a separate output buffer — this
|
||||
is the **authentic** P&L signal used by validation regardless of
|
||||
training/validation mode.
|
||||
|
||||
Both training and validation call the same kernel. The only difference is
|
||||
the scalar values at the pinned pointers.
|
||||
|
||||
### Phase 3 — Deprecate the two old kernels (≤1 day)
|
||||
|
||||
Delete `experience_env_step_batch` and `backtest_env_step_batch`. Rewire
|
||||
all callers to the unified kernel. Remove any dead utility code that was
|
||||
specific to one path.
|
||||
|
||||
### Phase 4 — Re-tune adaptive-learning mechanisms (≤2 days)
|
||||
|
||||
The distillation, collapse-prevention, and health-EMA machinery was tuned
|
||||
against the old training reward magnitudes. With shaping removed or
|
||||
re-scaled, the health-signal thresholds (`SNAPSHOT_HEALTH_THRESHOLD`,
|
||||
`DISTILL_HEALTH_THRESHOLD`, etc.) may need recalibration. The E1 smoke
|
||||
test is the canary.
|
||||
|
||||
### Phase 5 — Verify on L40S full-scale run (≤1 day)
|
||||
|
||||
Production 50-epoch run. Expected signatures of success:
|
||||
|
||||
- Training `Sharpe_raw` ≥ 0 by epoch 10; ≥ +0.03 by epoch 30
|
||||
- Validation `Sharpe_raw` ≥ training `Sharpe_raw` − 0.05
|
||||
- Validation `MaxDD` ≤ 3× training `MaxDD`
|
||||
- Validation `WinRate` tracks training `WinRate` ± 5 percentage points
|
||||
|
||||
If any of these fail, the analysis is wrong and we revisit Phase 1.
|
||||
|
||||
## Risks
|
||||
|
||||
1. **Collapse-prevention regresses.** The distillation mechanism was tuned
|
||||
with the existing shaping weights in the loop. Pure-P&L training may
|
||||
cause the network to find attractor states the current mechanisms don't
|
||||
cover. Mitigation: Phase 4 recalibration, E1 smoke test as the gate.
|
||||
2. **Training becomes too noisy.** Sparse P&L reward may have insufficient
|
||||
gradient signal for the Q-network to learn. Mitigation: the counterfactual
|
||||
curriculum (N4) should provide dense gradient signal even with pure P&L
|
||||
reward — verify this empirically.
|
||||
3. **Historical reward-shaping decisions are load-bearing in ways we don't
|
||||
see.** Some terms may have been added to fix subtle numerical instability
|
||||
(e.g., `reward_noise_scale = 0.05` — label smoothing proxy). Mitigation:
|
||||
Phase 1's per-term inventory must include the provenance commit, not just
|
||||
the semantic purpose.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Changing the underlying state representation (96-dim layout, OFI
|
||||
integration, MTF).
|
||||
- Changing the network architecture (4-branch factored DQN, ensemble heads,
|
||||
IQN/C51).
|
||||
- Re-tuning transaction cost parameters (`tx_cost_bps`, `spread_cost`).
|
||||
These are calibrated to actual broker execution and should remain as-is.
|
||||
|
||||
## Success criteria
|
||||
|
||||
A pull request that:
|
||||
|
||||
1. Deletes `experience_env_step` and `backtest_env_step` kernels in favor
|
||||
of a single unified env step.
|
||||
2. Deletes the non-P&L-aligned shaping terms (from Phase 1 inventory).
|
||||
3. Passes the E1 collapse-prevention smoke test with final
|
||||
`epoch_q_gap > 0.05`.
|
||||
4. Shows validation `Sharpe_raw` within 0.05 of training `Sharpe_raw` by
|
||||
epoch 30 on the L40S 50-epoch production run.
|
||||
5. Documents the reward semantics in a single comment block at the
|
||||
unified kernel site.
|
||||
|
||||
## References
|
||||
|
||||
- Current distillation root-cause analysis: commit `24f96ab78`
|
||||
- Smoke test run showing divergence: 2026-04-20 E1 trajectory
|
||||
- Reward shaping ablation result: 30% gap closure from disabling 8 credit
|
||||
weights
|
||||
- Existing hint at the problem: `experience_kernels.cu:1418` comment
|
||||
*"Regime-adaptive scaling removed to eliminate train/eval mismatch"*
|
||||
Reference in New Issue
Block a user