docs(design): phase-1 inventory — gems/pearls/novels for env unification
Extends the reward inventory with a review pass that preserves the *ideas* behind behavioral terms while deleting the wrong-level mechanics. Gems (relocate, don't delete): - Probabilistic fill model (real physics, not shaping) → apply in both modes via shared Philox RNG + exploration_scale - Kelly-as-physics-constraint → hard position-size cap in trade_physics, not a soft reward - Path quality via per-step drawdown integration → no separate term, just run drawdown penalty every bar - Saboteur relocation → both modes, scaled by exploration_scale (training 1.0, validation 0.5, diagnostic 0.0) — avoids creating a new clean-vs- noisy mismatch in the opposite direction Pearls: - Sparse reward is fine if TD propagation works — diagnose before deleting micro_reward_scale; measure cov(Q(s_entry,a), trade_return) - Every behavioral term maps to one of four failure modes: double-count, misplaced physics, wrong-level regularization, or compensating for a downstream bug. None are semantically "neutral" shaping. Novel architectural moves: - Diagnostic-only entropy tracking (no reward, just HEALTH_DIAG logging) - Single exploration_scale scalar replacing all mode toggles (feedback_no_feature_flags compliant) - Layered reward with per-term budget caps — makes the train/val Sharpe gap computable instead of uncomputable magic - Q-target smoothing replacing reward_noise_scale (regularization at gradient level, not reward level) Updated disposition table: 4 DELETE, 2 MOVE, 2 RELOCATE-to-physics, 1 DIAGNOSE-then-DELETE, 4 delete-dead-plumbing. Three relocations (Kelly cap, saboteur scaling, Q-target smoothing) can land as independent PRs before the unified_env_kernel rewrite, shrinking Phase 2's change surface. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
@@ -100,6 +100,164 @@ core signal on any positioned bar) and `risk_efficiency_weight` (asymmetric
|
||||
amplifier of winners). These are prime candidates for the first pass of
|
||||
deletion because their magnitudes can dominate the core signal.
|
||||
|
||||
## Gems, pearls, and novel moves
|
||||
|
||||
Reviewing the remover list for ideas worth preserving *in a different form*.
|
||||
Each "behavioral" term was added to address a real phenomenon; deletion
|
||||
without replacement regresses to the pathology the term was patching.
|
||||
|
||||
### Gems (preserve the underlying idea, relocate the mechanism)
|
||||
|
||||
#### 🔹 Probabilistic fill model
|
||||
|
||||
Training has saboteur-driven `fill_ioc_fill_prob ∈ [0.65, 0.95]` — 15%+ of
|
||||
limit orders don't fill. Validation always fills deterministically. This
|
||||
is **real-world execution physics**, not a shaping term. The asymmetry
|
||||
explains part of the train/val Sharpe gap: training policies are robust to
|
||||
missed fills; validation policies are brittle.
|
||||
|
||||
**Relocation**: In the unified env, probabilistic fills apply in BOTH modes
|
||||
via a shared Philox RNG stream. Training uses the full adversarial range
|
||||
(0.65–0.95); validation uses a single calibrated mean (e.g. 0.80). Neither
|
||||
uses `1.0` — that would create a new mismatch in the opposite direction.
|
||||
|
||||
#### 🔹 Kelly as physics constraint, not reward
|
||||
|
||||
`kelly_sizing_weight` penalizes deviation from Kelly-optimal fraction.
|
||||
Wrong level: rewards matching a formula instead of rewarding outcomes. But
|
||||
Kelly IS a mathematically-correct cap on growth-optimal sizing.
|
||||
|
||||
**Relocation**: Delete the reward term. Add a hard position-size constraint
|
||||
in `trade_physics.cuh`:
|
||||
|
||||
```c
|
||||
float kelly_cap = kelly_fraction(win_stats) * max_position * safety_multiplier;
|
||||
target_position = clampf(target_position, -kelly_cap, +kelly_cap);
|
||||
```
|
||||
|
||||
The network can't over-lever because the environment physically won't let
|
||||
it — not because the reward penalizes it. Same destination, different
|
||||
layer of the stack.
|
||||
|
||||
#### 🔹 Path quality via per-step drawdown integration
|
||||
|
||||
`risk_efficiency_weight` rewards trades with low intra-trade drawdown —
|
||||
i.e., path quality beyond final P&L. It double-counts asymmetrically
|
||||
(only on winners), but the underlying signal is real.
|
||||
|
||||
**Relocation**: The time-integral of `compute_drawdown_penalty` IS path
|
||||
quality, applied uniformly (not winners-only). Make sure the drawdown
|
||||
penalty runs every bar, not just at trade completion. No separate reward
|
||||
term needed — integration does the work.
|
||||
|
||||
#### 🔹 Saboteur noise relocated, not deleted
|
||||
|
||||
Saboteur (`spread_mult`, `fill_prob`, `slippage_mult` per-episode) made
|
||||
the training policy robust. Validation had none of it, so training
|
||||
learned strategies that rely on clean execution.
|
||||
|
||||
**Relocation**: Saboteur becomes a per-bar Philox-driven noise source
|
||||
scaled by `exploration_scale ∈ [0, 1]` (pinned device-mapped scalar).
|
||||
Training: `exploration_scale = 1.0` (full adversarial). Validation:
|
||||
`exploration_scale = 0.5` (calibrated to realistic execution noise, NOT
|
||||
zero). Sets a deliberate train/val gap that mirrors deployment-vs-backtest
|
||||
conditions rather than hiding them.
|
||||
|
||||
### Pearls (subtle insights)
|
||||
|
||||
#### 💎 Sparse reward is fine if TD propagation works
|
||||
|
||||
The reason `micro_reward_scale` was added: trade-completion reward is
|
||||
sparse (one reward per completed trade), so gradient signal is weak on
|
||||
holding bars. But the Q-learning framework handles this via TD bootstrap
|
||||
— IQN, C51, n-step exist specifically to propagate sparse signal across
|
||||
time.
|
||||
|
||||
**Diagnostic before deleting**: verify whether Q-values for entry bars
|
||||
respond to trade-completion rewards. If Q(entry) tracks the average of
|
||||
subsequent trade outcomes, TD works — delete `micro_reward_scale`
|
||||
unconditionally. If Q(entry) is decoupled from outcomes, the bug is in
|
||||
n-step propagation (possibly `td_lambda = 0.9` being too low, or reward
|
||||
clipping truncating the signal before TD sees it). Fix that, then delete.
|
||||
|
||||
#### 💎 Every behavioral term is one of four failure modes
|
||||
|
||||
| Category | Terms | Fix |
|
||||
|---|---|---|
|
||||
| Double-count of P&L-aligned terms | `order_credit`, `urgency_credit`, `risk_efficiency` | Just delete |
|
||||
| Misplaced physics | `commitment_lambda`, `kelly_sizing`, `saboteur` (relocation) | Move to trade_physics / env constraints |
|
||||
| Wrong-level regularization | `reward_noise_scale`, `position_entropy_weight` | Move to Q-target smoothing / diagnostic logging |
|
||||
| Compensating for downstream bug | `micro_reward_scale` | Fix the downstream bug (TD propagation) |
|
||||
|
||||
No term is semantically "neutral." Each was a patch for a specific
|
||||
pathology better solved elsewhere.
|
||||
|
||||
### Novel architectural moves
|
||||
|
||||
#### ⭐ Diagnostic-only entropy tracking
|
||||
|
||||
Don't reward entropy. But TRACK it. `H(dir_hist)` and `H(mag_hist)` per
|
||||
epoch, logged alongside HEALTH_DIAG. When entropy drops below a
|
||||
threshold, existing `LOW EXPOSURE DIVERSITY` warnings fire — elevate
|
||||
those from warnings to principled collapse signals. Exploration bonuses
|
||||
belong in ε-greedy / noisy-nets, not in the reward.
|
||||
|
||||
#### ⭐ Single `exploration_scale` scalar replaces all mode toggles
|
||||
|
||||
Unified env takes `exploration_scale: f32` via pinned device-mapped
|
||||
memory (same pattern as distillation alpha — see
|
||||
`2026-04-21-unified-train-val-env-design.md` §Option C):
|
||||
|
||||
- `1.0` → full training: saboteur on, ε-greedy at current ε,
|
||||
counterfactual flips at `cf_ratio`
|
||||
- `0.5` → semi-realistic validation: saboteur half-strength, ε=0, no
|
||||
counterfactuals
|
||||
- `0.0` → deterministic diagnostic: all adversarial off, pure argmax
|
||||
|
||||
One scalar, three behaviors. No boolean flags
|
||||
(`feedback_no_feature_flags.md` compliant). Kernel reads it once per step
|
||||
from the pinned pointer.
|
||||
|
||||
#### ⭐ Layered reward with per-term budget caps
|
||||
|
||||
```
|
||||
reward = pnl_per_step // core, always present
|
||||
+ shaping_scale × clamp(0, DRAWDOWN_BUDGET, drawdown_penalty)
|
||||
+ shaping_scale × clamp(0, CHURN_BUDGET, churn_penalty)
|
||||
+ shaping_scale × clamp(0, HOLDING_BUDGET, holding_cost)
|
||||
```
|
||||
|
||||
Each shaping term has an absolute dollar-equivalent budget. Total shaping
|
||||
contribution is bounded by `Σ budgets × shaping_scale`. The train/val
|
||||
Sharpe gap upper bound becomes **computable**:
|
||||
`gap_upper_bound = Σ budgets × shaping_scale × num_bars / σ(pnl_per_step)`.
|
||||
Today that gap is uncomputable magic; tomorrow it's a scalar we can
|
||||
budget against.
|
||||
|
||||
#### ⭐ Q-target smoothing replaces reward noise
|
||||
|
||||
`reward_noise_scale` adds uniform noise to the reward, changing the
|
||||
objective. Better: add N(0, σ) noise to the Q-target distribution in the
|
||||
C51 Bellman backward (`c51_loss_kernel.cu`). Same regularization effect
|
||||
(prevents overfitting to exact target values) without perturbing what
|
||||
the network believes "reward" means. Applied at gradient level, not
|
||||
reward level.
|
||||
|
||||
## Updated disposition table (Phase 2 scope)
|
||||
|
||||
| Term | Disposition | Replacement |
|
||||
|---|---|---|
|
||||
| `order_credit_weight` | DELETE | Core tx_cost already captures real fill savings |
|
||||
| `risk_efficiency_weight` | DELETE | Per-step drawdown integration |
|
||||
| `urgency_credit_weight` | DELETE | Redundant with vol-normalized core return |
|
||||
| `kelly_sizing_weight` | DELETE → physics | Kelly-fraction position cap in `trade_physics.cuh` |
|
||||
| `micro_reward_scale` | DIAGNOSE → DELETE | Verify TD propagation first; fix `td_lambda`/n-step if broken |
|
||||
| `commitment_lambda` | DELETE | Strengthen tx_cost slippage model instead |
|
||||
| `reward_noise_scale` | MOVE | N(0,σ) on Q-target in C51 Bellman |
|
||||
| `position_entropy_weight` | MOVE | Diagnostic log via HEALTH_DIAG |
|
||||
| `saboteur_params` (training-only today) | RELOCATE | Apply in both modes scaled by `exploration_scale` |
|
||||
| `w_dsr`, `exit_timing_weight`, `ofi_reward_weight`, `opp_cost_scale` | DELETE dead plumbing | N/A — already dormant |
|
||||
|
||||
## Next step (Phase 2)
|
||||
|
||||
Per the design doc:
|
||||
@@ -112,5 +270,27 @@ Per the design doc:
|
||||
> - Computes reward = `pnl_per_step + shaping_scale × shaping_bundle`.
|
||||
> - Writes `step_return = pnl_per_step` to a separate output buffer.
|
||||
|
||||
With Phase 1 done, Phase 2 has a concrete deletion list rather than a
|
||||
judgment call. Safe to proceed.
|
||||
With Phase 1 done, Phase 2 has a concrete deletion list AND a concrete
|
||||
relocation plan. The relocations (Kelly cap, saboteur scaling, Q-target
|
||||
smoothing) can land as independent PRs before the unified kernel rewrite,
|
||||
reducing Phase 2's change surface.
|
||||
|
||||
## Open questions for Phase 2
|
||||
|
||||
1. **TD propagation diagnostic**: before deleting `micro_reward_scale`, we
|
||||
need to verify Q-values at trade-entry bars respond to
|
||||
trade-completion rewards. Simplest test: train with sparse rewards
|
||||
only for 10 epochs; measure `cov(Q(s_entry, a), realized_trade_return)`.
|
||||
If > 0.3, TD is working. If < 0.1, fix TD first.
|
||||
|
||||
2. **Kelly priors**: current `kelly_sizing_weight` uses priors
|
||||
`win_rate ≈ 0.5, payoff_ratio ≈ 1.0` before any trades. For the
|
||||
position-cap version, need a conservative initial `kelly_cap` (e.g.
|
||||
0.25 × max_position) that loosens as real win/loss stats accumulate.
|
||||
Open: should the cap be per-episode or per-run? Per-run gives more
|
||||
data but crosses regime boundaries.
|
||||
|
||||
3. **Saboteur calibration for validation**: `exploration_scale = 0.5`
|
||||
picks "halfway adversarial" — but what's the RIGHT noise level for
|
||||
honest backtest? Calibrate against live broker execution stats if
|
||||
available; otherwise use a known-conservative midpoint.
|
||||
|
||||
Reference in New Issue
Block a user