feat(v8): wire 5 dead CUDA kernels + config fixes
Kernels now wired with Rust load + launch methods: - popart_normalize_kernel: running mean/var + in-place normalization - exposure_pretrain_step: supervised direction pre-training with backward GEMM - td_lambda_kernel: exponentially-weighted lambda-returns - compute_difficulty_scores: per-bar curriculum difficulty - hindsight_relabel_kernel: optimal exit relabeling Config fixes: - batch_size hyperopt range: [64, 512] → [512, 8192] (H100 capacity) - micro_reward_scale: 0.001 → 0.01 (10x stronger bootstrap) - micro_reward hyperopt range: [0, 0.005] → [0, 0.05] 900 tests pass, 0 regressions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
@@ -5,7 +5,7 @@
|
||||
[search_space]
|
||||
# Base parameters
|
||||
learning_rate = [0.00001, 0.0003] # log scale in adapter
|
||||
batch_size = [64, 512]
|
||||
batch_size = [512, 8192]
|
||||
gamma = [0.90, 0.99] # wider range — v_range computed dynamically from gamma
|
||||
buffer_size = [50000, 100000] # log scale in adapter
|
||||
max_leverage = [2.0, 10.0] # leverage ratio; position computed from capital/price
|
||||
@@ -95,7 +95,7 @@ reward_noise_scale = [0.01, 0.1]
|
||||
exposure_aux_weight = [0.1, 1.0]
|
||||
|
||||
# v8 search ranges
|
||||
micro_reward_scale = [0.0, 0.005]
|
||||
micro_reward_scale = [0.0, 0.05]
|
||||
td_lambda = [0.5, 0.99]
|
||||
hindsight_fraction = [0.0, 0.3]
|
||||
hindsight_lookahead = [5, 20]
|
||||
@@ -118,7 +118,7 @@ q_clip_max = 200.0
|
||||
|
||||
[reward]
|
||||
# v8 comprehensive training overhaul
|
||||
micro_reward_scale = 0.001
|
||||
micro_reward_scale = 0.01
|
||||
td_lambda = 0.9
|
||||
max_trace_length = 7
|
||||
hindsight_fraction = 0.1
|
||||
@@ -162,7 +162,7 @@ branch_hidden_dim = 64
|
||||
dueling_hidden_dim = 128
|
||||
v_range = 1.0
|
||||
# v8 phase_fast overrides
|
||||
micro_reward_scale = 0.001
|
||||
micro_reward_scale = 0.01
|
||||
td_lambda = 0.9
|
||||
hindsight_fraction = 0.1
|
||||
hindsight_lookahead = 10
|
||||
|
||||
@@ -114,7 +114,7 @@ causal_intensity = 1.0
|
||||
|
||||
[reward]
|
||||
# v8 comprehensive training overhaul
|
||||
micro_reward_scale = 0.001
|
||||
micro_reward_scale = 0.01
|
||||
td_lambda = 0.9
|
||||
max_trace_length = 7
|
||||
hindsight_fraction = 0.1
|
||||
|
||||
@@ -125,7 +125,7 @@ causal_intensity = 1.0
|
||||
|
||||
[reward]
|
||||
# v8 comprehensive training overhaul
|
||||
micro_reward_scale = 0.001
|
||||
micro_reward_scale = 0.01
|
||||
td_lambda = 0.9
|
||||
max_trace_length = 7
|
||||
hindsight_fraction = 0.1
|
||||
|
||||
@@ -121,7 +121,7 @@ causal_intensity = 1.0
|
||||
|
||||
[reward]
|
||||
# v8 comprehensive training overhaul
|
||||
micro_reward_scale = 0.001
|
||||
micro_reward_scale = 0.01
|
||||
td_lambda = 0.9
|
||||
max_trace_length = 7
|
||||
hindsight_fraction = 0.1
|
||||
|
||||
1359
docs/superpowers/plans/2026-04-07-reward-v8-master.md
Normal file
1359
docs/superpowers/plans/2026-04-07-reward-v8-master.md
Normal file
File diff suppressed because it is too large
Load Diff
297
docs/superpowers/specs/2026-04-07-reward-v7-design.md
Normal file
297
docs/superpowers/specs/2026-04-07-reward-v7-design.md
Normal file
@@ -0,0 +1,297 @@
|
||||
# Reward v7: Counterfactual Branch Attribution Design
|
||||
|
||||
**Goal:** Fix the 95% breakeven win rate caused by v6 penalty stacking and introduce novel per-branch credit assignment for the branching DQN's factored 81-action space.
|
||||
|
||||
**Architecture:** Sparse trade-completion reward with asymmetric soft-clamp, counterfactual exposure advantage (CEA), order-type microstructure credit, and intra-trade risk efficiency. Removes 5 penalty layers, adds 3 novel reward components.
|
||||
|
||||
**Tech Stack:** CUDA kernel (`experience_kernels.cu`), f32 arithmetic, branching DQN (9 exposure × 3 order × 3 urgency = 81 actions), C51 distributional RL.
|
||||
|
||||
---
|
||||
|
||||
## 1. Problem Statement
|
||||
|
||||
Reward v6 applies 6+ penalty layers that compound to require a **94.9% win rate** for positive expected reward — mathematically impossible on ES futures. The model rationally converges to a "don't trade" (Flat) policy across ALL hyperopt trials. The exposure branch shows degenerate `i%3` Q-value patterns, confirming it receives no meaningful gradient signal.
|
||||
|
||||
### Root Causes
|
||||
|
||||
| Issue | Impact |
|
||||
|-------|--------|
|
||||
| Loss aversion (1.5×) applied BEFORE tanh | 2× asymmetry: wins +2.62, losses -3.80 |
|
||||
| Counterfactual regret always ≤ 0 | Systematically drags expected reward negative |
|
||||
| Hold scale redundant with min_hold_bars | Double-penalizes short trades |
|
||||
| Trade clustering penalty | Adds noise without meaningful signal |
|
||||
| Beta penalty | Already disabled (0.0), dead code |
|
||||
| Single scalar reward for 81 factored actions | Exposure branch can't learn action semantics |
|
||||
|
||||
### Evidence
|
||||
|
||||
- 50-trial hyperopt: ALL trials collapse to 100% Flat within 2-3 epochs
|
||||
- Win rate stuck at exactly ~30% (random epsilon baseline) across every configuration
|
||||
- Q-values: S100≈S25≈L50, S75≈Flat≈L75, S50≈L25≈L100 (i%3 degeneracy)
|
||||
- Research: stacking >3 reward penalties causes signal collapse (per RL literature review)
|
||||
|
||||
## 2. Design
|
||||
|
||||
### 2.1 Reward Pipeline Overview
|
||||
|
||||
```
|
||||
AT TRADE EXIT:
|
||||
1. segment_return = segment_pnl / prev_equity
|
||||
2. vol_normalize(segment_return, atr, hold_time)
|
||||
3. scale: base_reward = 10.0 * vol_normalized_return
|
||||
4. squash: asymmetric_soft_clamp(base_reward) ← replaces loss_aversion + tanh
|
||||
5. + cea_weight * exposure_advantage ← THE GEM (novel)
|
||||
6. + order_weight * order_credit ← microstructure (novel)
|
||||
7. + risk_eff_weight * risk_efficiency ← path quality (novel)
|
||||
8. + drawdown_penalty (dense, every bar) ← kept from v6
|
||||
9. capital_floor → reward = -10 ← kept from v6
|
||||
|
||||
DURING TRADE / WHEN FLAT:
|
||||
reward = 0.0 + drawdown_penalty (if DD > threshold)
|
||||
```
|
||||
|
||||
### 2.2 Layer 1: Sparse Trade-Completion Base (kept from v6)
|
||||
|
||||
No changes to the core sparse reward. Validated by ETDQN (Takara et al., 2023) — outperforms dense rewards by 1.46-7.13× on intraday trading.
|
||||
|
||||
```cuda
|
||||
float segment_return = segment_pnl / (prev_equity > 1.0f ? prev_equity : 1.0f);
|
||||
float vol_normalized_return = segment_return / vol_norm;
|
||||
float base_reward = 10.0f * vol_normalized_return;
|
||||
```
|
||||
|
||||
Vol normalization uses ATR(14) proxy: `vol_norm = atr_pct * sqrt(hold_time)`. Makes rewards comparable across trending and ranging regimes.
|
||||
|
||||
### 2.3 Layer 2: Asymmetric Soft-Clamp (replaces loss_aversion + tanh)
|
||||
|
||||
**Removes:** `loss_aversion` parameter, separate `tanh` squash.
|
||||
|
||||
The key insight: loss aversion applied BEFORE a nonlinear squash creates compound asymmetry that makes expected reward negative even for a fair game. Instead, use a single function that IS the risk aversion:
|
||||
|
||||
```cuda
|
||||
__device__ float asymmetric_soft_clamp(float x) {
|
||||
// Gains: linear with hard cap at +10 (full gradient preserved)
|
||||
// Losses: exponential compression (smooth, natural risk aversion)
|
||||
if (x >= 0.0f) return fminf(x, 10.0f);
|
||||
return -10.0f * (1.0f - expf(x / 10.0f));
|
||||
}
|
||||
```
|
||||
|
||||
**Properties:**
|
||||
- `f(+5.0) = +5.0` — full value, no compression
|
||||
- `f(-5.0) = -3.93` — compressed 21%
|
||||
- `f(-15.0) = -7.77` — compressed 48%
|
||||
- `f(+15.0) = +10.0` — capped (C51 bound)
|
||||
- Continuous, differentiable everywhere
|
||||
- No separate parameter needed — the function shape IS the risk preference
|
||||
- Gradient for gains: 1.0 (full). Gradient for losses at x=-5: `exp(-0.5) = 0.61`
|
||||
|
||||
**Breakeven impact:** Expected reward for 50/50 win/loss at ±5 magnitude:
|
||||
- v6: `0.5 × 4.62 + 0.5 × (-9.05) = -2.22` (need 66% wins)
|
||||
- v7: `0.5 × 5.0 + 0.5 × (-3.93) = +0.54` (breakeven at ~44%)
|
||||
|
||||
### 2.4 Layer 3: Counterfactual Exposure Advantage — CEA (novel)
|
||||
|
||||
**Replaces:** `regret_blend` (always ≤ 0, pure punishment).
|
||||
|
||||
**Core insight:** The branching DQN decomposes `Q(s,a) = V(s) + A_exposure(i) + A_order(j) + A_urgency(k)`. With one scalar reward, the network must implicitly discover which branch caused the outcome. CEA provides **explicit advantage signal** for the exposure branch.
|
||||
|
||||
**Method:** At trade exit, compute the squashed reward for all 9 exposure levels using the same bar's price movement:
|
||||
|
||||
```cuda
|
||||
float sum_alt_rewards = 0.0f;
|
||||
float price_delta = raw_next - raw_close;
|
||||
for (int k = 0; k < b0_size; k++) {
|
||||
float alt_pos = compute_target_position(k, b0_size, max_position);
|
||||
float alt_pnl = alt_pos * price_delta;
|
||||
float alt_return = alt_pnl / (prev_equity > 1.0f ? prev_equity : 1.0f);
|
||||
float alt_vol_norm = alt_return / vol_norm;
|
||||
float alt_reward = asymmetric_soft_clamp(10.0f * alt_vol_norm);
|
||||
sum_alt_rewards += alt_reward;
|
||||
}
|
||||
float mean_alt_reward = sum_alt_rewards / (float)b0_size;
|
||||
float exposure_advantage = squashed_reward - mean_alt_reward;
|
||||
|
||||
reward += cea_weight * exposure_advantage;
|
||||
```
|
||||
|
||||
**Properties:**
|
||||
- Can be **positive** (model outperformed average exposure) or **negative** (underperformed)
|
||||
- **Zero-mean by construction** — no systematic drag on expected reward
|
||||
- Zero additional compute cost — we already have the counterfactual loop from v6 regret
|
||||
- Directly breaks i%3 Q-value degeneracy: S100 has different advantage than L50 because the position sizes differ
|
||||
- For Flat action: exposure_advantage is always ≤ 0 when any directional exposure would have profited, naturally encouraging the model to explore directional actions
|
||||
|
||||
**Why this is novel:** Counterfactual advantage decomposition has not been applied to branching DQN in the RL trading literature. It gives the exposure branch an **exact, per-step attribution signal** rather than forcing implicit credit assignment through shared scalar rewards.
|
||||
|
||||
**Hyperopt parameter:** `cea_weight` in [0.1, 1.0], default 0.3.
|
||||
|
||||
### 2.5 Layer 4: Order Type Microstructure Credit (novel)
|
||||
|
||||
**New component:** Rewards cost-efficient execution by comparing the taken order type's transaction cost against the worst-case (MarketOrder).
|
||||
|
||||
```cuda
|
||||
if (segment_complete && fabsf(position) > 0.001f) {
|
||||
float market_cost = compute_tx_cost(position, raw_close, tx_cost_multiplier,
|
||||
spread_cost, max_position, 0/*Market*/, -1.0f);
|
||||
float taken_cost = compute_tx_cost(position, raw_close, tx_cost_multiplier,
|
||||
spread_cost, max_position, order_type_idx, spread_scale);
|
||||
float order_credit = (market_cost - taken_cost) / (prev_equity > 1.0f ? prev_equity : 1.0f);
|
||||
reward += order_weight * fminf(order_credit * 100.0f, 2.0f); // cap at +2.0
|
||||
}
|
||||
```
|
||||
|
||||
**Properties:**
|
||||
- Always ≥ 0 (LimitMaker saves spread, Market is baseline)
|
||||
- On ES: LimitMaker saves ~$3.12/trade → `order_credit ≈ 0.009%` → scaled to ~0.09
|
||||
- Gives the ORDER branch its own learning signal
|
||||
- Capped at +2.0 to prevent dominating the sparse exit reward
|
||||
|
||||
**Hyperopt parameter:** `order_credit_weight` in [0.0, 0.5], default 0.1.
|
||||
|
||||
### 2.6 Layer 5: Intra-Trade Risk Efficiency (novel)
|
||||
|
||||
**New component:** Rewards clean winning trades (minimal intra-trade drawdown) over volatile ones.
|
||||
|
||||
**Requires:** New portfolio state field `ps[22] = intra_trade_max_dd` (bf16), tracking the worst unrealized drawdown during the current trade. Reset to 0 at trade entry.
|
||||
|
||||
**Per-bar update (during trade):**
|
||||
```cuda
|
||||
if (fabsf(position) > 0.001f) {
|
||||
float unrealized = (position > 0) ? (raw_close - entry_price) / entry_price
|
||||
: (entry_price - raw_close) / entry_price;
|
||||
float current_dd = fminf(unrealized, 0.0f); // negative when underwater
|
||||
intra_trade_max_dd = fminf(intra_trade_max_dd, current_dd);
|
||||
}
|
||||
```
|
||||
|
||||
**At trade exit (winners only):**
|
||||
```cuda
|
||||
if (segment_return > 0.0f && intra_trade_max_dd < -0.001f) {
|
||||
float risk_eff = segment_return / fabsf(intra_trade_max_dd);
|
||||
reward += risk_eff_weight * fminf(risk_eff, 5.0f);
|
||||
}
|
||||
```
|
||||
|
||||
**Properties:**
|
||||
- Only fires for winning trades with intra-trade drawdown > 0.1%
|
||||
- A +2% trade that never went negative: no bonus (perfect entry doesn't need it)
|
||||
- A +2% trade that dipped -1% first: `risk_eff = 2.0` → bonus = +0.2 (at default weight)
|
||||
- A +2% trade that dipped -5% first: `risk_eff = 0.4` → bonus = +0.04 (less reward for volatility)
|
||||
- Captures **path quality** — novel in RL trading
|
||||
- Capped at 5.0 to prevent outlier domination
|
||||
|
||||
**Hyperopt parameter:** `risk_efficiency_weight` in [0.0, 0.5], default 0.1.
|
||||
|
||||
### 2.7 Layer 6: Drawdown Penalty (kept from v6)
|
||||
|
||||
No changes. The only dense reward component. Fires every bar when drawdown exceeds `dd_threshold`.
|
||||
|
||||
```cuda
|
||||
reward += compute_drawdown_penalty(equity, peak_equity, dd_threshold, w_dd);
|
||||
```
|
||||
|
||||
Essential for risk management. Validated by research as the only dense component worth keeping.
|
||||
|
||||
### 2.8 Layer 7: Capital Floor (kept from v6)
|
||||
|
||||
No changes. Regulatory requirement (PDT $25K rule). Terminal penalty.
|
||||
|
||||
```cuda
|
||||
if (check_capital_floor(portfolio_value, peak_equity)) {
|
||||
reward = -10.0f;
|
||||
}
|
||||
```
|
||||
|
||||
## 3. Removed Components
|
||||
|
||||
| Component | Reason for Removal |
|
||||
|-----------|-------------------|
|
||||
| `loss_aversion` multiplier | Replaced by asymmetric_soft_clamp function shape |
|
||||
| `regret_blend` | Replaced by CEA (symmetric, can be positive) |
|
||||
| `hold_scale` / hold-time reward scaling | Redundant with min_hold_bars enforcement in trade physics |
|
||||
| `trade_clustering_penalty` | Weak impact (0.05 × CV), adds noise, redundant with HER temporal diversity |
|
||||
| `beta_penalty` | Already disabled (0.0), dead code removed |
|
||||
|
||||
## 4. New Portfolio State Field
|
||||
|
||||
Add `ps[22] = intra_trade_max_dd` (bf16):
|
||||
- Reset to 0.0 at trade entry (`entering_trade` or `reversing_trade` new segment)
|
||||
- Updated every bar during active trade: `min(current_unrealized_dd, intra_trade_max_dd)`
|
||||
- Read at trade exit for risk efficiency computation
|
||||
|
||||
This increases portfolio state from 22 to 23 bf16 fields. No layout changes elsewhere — the portfolio state buffer is dynamically sized from `PORTFOLIO_STATE_DIM`.
|
||||
|
||||
## 5. Configuration Changes
|
||||
|
||||
### New Parameters (experience kernel args)
|
||||
|
||||
| Parameter | Type | Default | Hyperopt Range | Purpose |
|
||||
|-----------|------|---------|---------------|---------|
|
||||
| `cea_weight` | f32 | 0.3 | [0.1, 1.0] | Counterfactual exposure advantage blend |
|
||||
| `order_credit_weight` | f32 | 0.1 | [0.0, 0.5] | Order type microstructure bonus |
|
||||
| `risk_efficiency_weight` | f32 | 0.1 | [0.0, 0.5] | Intra-trade path quality bonus |
|
||||
|
||||
### Removed Parameters
|
||||
|
||||
| Parameter | Previously | Status |
|
||||
|-----------|-----------|--------|
|
||||
| `loss_aversion` | 1.5 | REMOVED — function shape provides risk aversion |
|
||||
| `regret_blend` | 0.3 | REMOVED — replaced by CEA |
|
||||
| `trade_clustering_penalty` | 0.05 | REMOVED |
|
||||
| `beta_penalty` | 0.0 | REMOVED (was disabled) |
|
||||
| `position_entropy_weight` | 0.0 | KEPT (already disabled, may enable later) |
|
||||
|
||||
### TOML Config Updates
|
||||
|
||||
All 4 config files need updates:
|
||||
- `config/training/dqn-production.toml`
|
||||
- `config/training/dqn-hyperopt.toml`
|
||||
- `config/training/dqn-smoketest.toml`
|
||||
- `config/training/dqn-localdev.toml`
|
||||
|
||||
## 6. Expected Breakeven Analysis
|
||||
|
||||
| Configuration | Win Rate for E[R] = 0 |
|
||||
|---------------|----------------------|
|
||||
| v6 (current, all penalties) | 94.9% |
|
||||
| v7 base (soft-clamp only) | ~44% |
|
||||
| v7 + CEA (advantage boosts good picks) | ~42% |
|
||||
| v7 + CEA + order credit (saves ~$3/trade) | ~40% |
|
||||
| Realistic ES target with tx costs | ~52-55% |
|
||||
|
||||
The ~12% reduction in breakeven win rate (from impossible 95% to achievable 52-55%) is the critical fix.
|
||||
|
||||
## 7. Files Modified
|
||||
|
||||
| File | Changes |
|
||||
|------|---------|
|
||||
| `crates/ml/src/cuda_pipeline/experience_kernels.cu` | Reward v7 logic, new kernel args, portfolio state field |
|
||||
| `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs` | Wire new kernel args, update PORTFOLIO_STATE_DIM |
|
||||
| `config/training/dqn-production.toml` | New params, remove old ones |
|
||||
| `config/training/dqn-hyperopt.toml` | New hyperopt ranges, remove old params |
|
||||
| `config/training/dqn-smoketest.toml` | New params |
|
||||
| `config/training/dqn-localdev.toml` | New params |
|
||||
| `crates/ml/src/trainers/dqn/config.rs` | Add new config fields, remove old ones |
|
||||
| `crates/ml/src/hyperopt/adapters/dqn.rs` | Wire new hyperopt params |
|
||||
|
||||
## 8. Testing
|
||||
|
||||
### Smoke Test Validation
|
||||
- Run `cargo test -p ml --lib -- smoke_tests --ignored` with v7 reward
|
||||
- Verify: reward distribution is NOT all-zero (model trades)
|
||||
- Verify: exposure_advantage has both positive and negative values
|
||||
- Verify: win rate > 30% by epoch 3 (exceeds random baseline)
|
||||
- Verify: Q-values do NOT show i%3 degeneracy pattern
|
||||
|
||||
### Numerical Sanity Checks
|
||||
- Breakeven trade (0 PnL): reward ≈ 0 + small CEA adjustment
|
||||
- +1% trade, 5-bar hold, 0.5% ATR: reward ≈ +4.5 (before CEA)
|
||||
- -1% trade, 5-bar hold, 0.5% ATR: reward ≈ -3.5 (after soft-clamp compression)
|
||||
- Capital floor breach: reward = -10.0 exactly
|
||||
|
||||
### Regression Tests
|
||||
- Kelly statistics accumulation unchanged (wins/losses/returns tracking)
|
||||
- Drawdown penalty unchanged
|
||||
- Capital floor behavior unchanged
|
||||
- Episode termination unchanged
|
||||
Reference in New Issue
Block a user