feat: v9 Differential Sharpe Ratio reward — optimize for consistency, not PnL
Fundamental shift: reward optimizes Sharpe ratio contribution per trade instead of directional PnL magnitude. Targets Sharpe 5+ via many small consistent gains (spread capture, execution quality) instead of few large directional bets. DSR implementation (Moody & Saffell, 2001): - Per-episode EMA statistics in portfolio state slots [3]-[5] - DSR = (B*R - 0.5*A*R²) / (B - A²)^1.5 at trade completion - 10-trade warmup, clamped [-5, +5] - w_dsr=5.0 (primary reward signal) Reward hierarchy restructured: - DSR: 5.0 (NEW — primary, rewards Sharpe consistency) - Directional PnL: 2.0 (was 10.0 — demoted to secondary) - Order credit: 1.0 (was 0.1 — spread capture amplified 10×) - Urgency credit: 0.5 (was 0.1 — fill quality amplified 5×) - Inventory penalty: -0.005×|pos|/max_pos per bar (NEW) - Dense micro-reward: 0.0 (removed — noisy direction signal) Expected: WinRate increases (many small spread captures), per-trade variance decreases, Sharpe rises from consistency not prediction. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
293
docs/superpowers/plans/2026-04-12-dsr-reward-restructure.md
Normal file
293
docs/superpowers/plans/2026-04-12-dsr-reward-restructure.md
Normal file
@@ -0,0 +1,293 @@
|
||||
# Differential Sharpe Ratio Reward Restructure
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Replace directional PnL as the dominant reward with Differential Sharpe Ratio (DSR) — the model optimizes for Sharpe consistency instead of PnL magnitude, targeting Sharpe 5+.
|
||||
|
||||
**Architecture:** DSR is computed at trade completion in the `experience_env_step` CUDA kernel using per-episode EMA statistics stored in portfolio state slots [3] (dsr_A) and [4] (dsr_B). The reward hierarchy flips: DSR (w=5.0) becomes primary, directional PnL demoted (10→2), order credit amplified (0.1→1.0), inventory penalty added, dense micro-reward removed. Config fields `w_dsr` and `dsr_eta` already exist — just need kernel implementation + TOML weight changes.
|
||||
|
||||
**Tech Stack:** CUDA 12.4 (experience_kernels.cu), Rust config wiring, TOML profiles
|
||||
|
||||
---
|
||||
|
||||
## File Map
|
||||
|
||||
| File | Action | Changes |
|
||||
|------|--------|---------|
|
||||
| `crates/ml/src/cuda_pipeline/experience_kernels.cu` | Modify | Add DSR computation + inventory penalty in reward kernel, add `w_dsr`/`dsr_eta` kernel params |
|
||||
| `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs` | Modify | Pass `w_dsr`/`dsr_eta` to kernel launch |
|
||||
| `config/training/dqn-production.toml` | Modify | New reward weights |
|
||||
| `config/training/dqn-smoketest.toml` | Modify | Same |
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Add DSR computation to reward kernel + inventory penalty
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu`
|
||||
|
||||
- [ ] **Step 1: Add `w_dsr`, `dsr_eta` kernel parameters**
|
||||
|
||||
In the `experience_env_step` kernel signature (after `float micro_reward_scale` at line 1080), add two new parameters:
|
||||
|
||||
```c
|
||||
float micro_reward_scale, /* v8: dense directional micro-reward scale */
|
||||
float w_dsr, /* v9: Differential Sharpe Ratio weight */
|
||||
float dsr_eta, /* v9: DSR EMA decay rate (0.01 = ~100-trade window) */
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Define DSR portfolio state slot constants**
|
||||
|
||||
Near the top of the file (after `#define PORTFOLIO_STRIDE 23` at line 58), add slot documentation:
|
||||
|
||||
```c
|
||||
/* v9: Differential Sharpe Ratio running statistics (Moody & Saffell, 2001).
|
||||
* Stored in per-episode portfolio state — persistent across bars within an episode.
|
||||
* dsr_A = EMA of realized trade returns (updated at trade completion).
|
||||
* dsr_B = EMA of squared realized trade returns.
|
||||
* DSR_t = (B*R - 0.5*A*R²) / (B - A²)^1.5 measures how much each trade improves Sharpe. */
|
||||
#define DSR_A_SLOT 3
|
||||
#define DSR_B_SLOT 4
|
||||
#define DSR_TRADE_COUNT_SLOT 5
|
||||
```
|
||||
|
||||
Update the portfolio state layout comment (lines 23-38) to reflect:
|
||||
```c
|
||||
* [3] dsr_A — EMA of realized trade returns (v9 DSR)
|
||||
* [4] dsr_B — EMA of squared trade returns (v9 DSR)
|
||||
* [5] dsr_trade_count — number of completed trades (for DSR warmup)
|
||||
* [6] (reserved)
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Implement DSR reward at trade completion**
|
||||
|
||||
Inside the `if (segment_complete && segment_hold_time > 0)` block (after the vol normalization at line 1522 and BEFORE the base_reward computation at line 1525), add DSR computation:
|
||||
|
||||
```c
|
||||
/* ── v9: Differential Sharpe Ratio ──
|
||||
* Moody & Saffell (2001): reward = how much this trade improves the running Sharpe.
|
||||
* Small consistent gains → high DSR. Large volatile wins → low DSR.
|
||||
* Directly optimizes for risk-adjusted returns, not raw PnL. */
|
||||
float R_t = vol_normalized_return; /* already computed above */
|
||||
float dsr_a = ps[DSR_A_SLOT];
|
||||
float dsr_b = ps[DSR_B_SLOT];
|
||||
float dsr_count = ps[DSR_TRADE_COUNT_SLOT];
|
||||
|
||||
/* Update running EMA statistics */
|
||||
float new_a = dsr_a + dsr_eta * (R_t - dsr_a);
|
||||
float new_b = dsr_b + dsr_eta * (R_t * R_t - dsr_b);
|
||||
ps[DSR_A_SLOT] = new_a;
|
||||
ps[DSR_B_SLOT] = new_b;
|
||||
ps[DSR_TRADE_COUNT_SLOT] = dsr_count + 1.0f;
|
||||
|
||||
/* Compute DSR after warmup (need >= 10 trades for stable statistics) */
|
||||
if (w_dsr > 0.0f && dsr_count >= 10.0f) {
|
||||
float variance = dsr_b - dsr_a * dsr_a;
|
||||
if (variance > 1e-8f) {
|
||||
float dsr = (dsr_b * R_t - 0.5f * dsr_a * R_t * R_t)
|
||||
/ powf(variance, 1.5f);
|
||||
/* Soft-clamp DSR to [-5, +5] to prevent single-trade domination */
|
||||
dsr = fmaxf(fminf(dsr, 5.0f), -5.0f);
|
||||
reward += w_dsr * dsr;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Add inventory penalty (dense, every bar)**
|
||||
|
||||
After the existing micro-reward block (after line 1625), add an inventory penalty that fires EVERY bar (not just trade completion):
|
||||
|
||||
```c
|
||||
/* ── v9: Inventory penalty — penalize holding positions (reduces variance) ──
|
||||
* Market makers minimize inventory. Holding = directional risk = Sharpe killer.
|
||||
* Scales with position size: Full(1.0) penalized 4× more than Quarter(0.25). */
|
||||
if (fabsf(position) > 0.001f) {
|
||||
float inventory_cost = -0.005f * fabsf(position) / fmaxf(max_position, 0.001f);
|
||||
reward += inventory_cost;
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Demote directional PnL base reward**
|
||||
|
||||
Change line 1525 from:
|
||||
```c
|
||||
float base_reward = 10.0f * vol_normalized_return;
|
||||
```
|
||||
to:
|
||||
```c
|
||||
float base_reward = 2.0f * vol_normalized_return;
|
||||
```
|
||||
|
||||
This demotes directional PnL from 10× to 2× (DSR at 5× is now the primary signal).
|
||||
|
||||
- [ ] **Step 6: Verify kernel compiles**
|
||||
|
||||
```bash
|
||||
touch crates/ml/build.rs && SQLX_OFFLINE=true cargo build -p ml --lib 2>&1 | grep -E "experience_kernels|error"
|
||||
```
|
||||
|
||||
Expected: compilation fails (Rust launch site doesn't pass new params yet).
|
||||
|
||||
- [ ] **Step 7: Commit kernel changes**
|
||||
|
||||
```bash
|
||||
git add crates/ml/src/cuda_pipeline/experience_kernels.cu
|
||||
git commit -m "feat: DSR reward + inventory penalty in experience kernel (v9 reward restructure)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Wire DSR parameters from Rust to kernel
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs`
|
||||
|
||||
- [ ] **Step 1: Add `w_dsr` to ExperienceConfig**
|
||||
|
||||
In the `ExperienceConfig` struct (around line 240), add after `dsr_eta`:
|
||||
|
||||
```rust
|
||||
pub w_dsr: f32,
|
||||
```
|
||||
|
||||
Add default in the Default impl:
|
||||
```rust
|
||||
w_dsr: 5.0,
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Pass `w_dsr` and `dsr_eta` to kernel launch**
|
||||
|
||||
In the `env_step_kernel` launch (around line 2114), after `.arg(&config.micro_reward_scale)`, add:
|
||||
|
||||
```rust
|
||||
.arg(&config.w_dsr)
|
||||
.arg(&config.dsr_eta)
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Wire from hyperparams to ExperienceConfig**
|
||||
|
||||
In `training_loop.rs` where `ExperienceConfig` is constructed (search for `micro_reward_scale:`), add:
|
||||
|
||||
```rust
|
||||
w_dsr: self.hyperparams.w_dsr as f32,
|
||||
dsr_eta: self.hyperparams.dsr_eta as f32,
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Verify compilation and run tests**
|
||||
|
||||
```bash
|
||||
touch crates/ml/build.rs && SQLX_OFFLINE=true cargo build -p ml --lib
|
||||
SQLX_OFFLINE=true cargo test -p ml-dqn --lib
|
||||
SQLX_OFFLINE=true cargo test -p ml --lib -- gradient_budget config monitoring
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add crates/ml/src/cuda_pipeline/gpu_experience_collector.rs crates/ml/src/trainers/dqn/trainer/training_loop.rs
|
||||
git commit -m "feat: wire w_dsr + dsr_eta from config to experience kernel"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Update TOML profiles with new reward weights
|
||||
|
||||
**Files:**
|
||||
- Modify: `config/training/dqn-production.toml`
|
||||
- Modify: `config/training/dqn-smoketest.toml`
|
||||
|
||||
- [ ] **Step 1: Update production TOML**
|
||||
|
||||
In the `[reward]` section of `config/training/dqn-production.toml`, update:
|
||||
|
||||
```toml
|
||||
[reward]
|
||||
# v9: DSR-primary reward restructure — optimize for Sharpe, not PnL
|
||||
popart_enabled = true
|
||||
micro_reward_scale = 0.0 # REMOVED: noisy directional micro-reward
|
||||
td_lambda = 0.9
|
||||
max_trace_length = 7
|
||||
hindsight_fraction = 0.1
|
||||
hindsight_lookahead = 10
|
||||
loss_aversion = 1.0
|
||||
q_gap_threshold = 0.1
|
||||
w_dd = 1.0
|
||||
dd_threshold = 0.02
|
||||
cea_weight = 0.3
|
||||
order_credit_weight = 1.0 # AMPLIFIED: spread capture is primary alpha (was 0.1)
|
||||
risk_efficiency_weight = 0.1
|
||||
urgency_credit_weight = 0.5 # AMPLIFIED: fill quality matters for consistency (was 0.1)
|
||||
exit_timing_weight = 0.1 # Slightly increased (was 0.05)
|
||||
ofi_reward_weight = 0.2
|
||||
kelly_sizing_weight = 0.1
|
||||
reward_noise_scale = 0.05
|
||||
w_dsr = 5.0 # NEW: Differential Sharpe Ratio (primary reward)
|
||||
b3_size = 3
|
||||
|
||||
[advanced]
|
||||
dsr_eta = 0.01 # DSR EMA decay (~100-trade window)
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Update smoketest TOML**
|
||||
|
||||
Same changes in `config/training/dqn-smoketest.toml`:
|
||||
|
||||
```toml
|
||||
micro_reward_scale = 0.0
|
||||
order_credit_weight = 1.0
|
||||
urgency_credit_weight = 0.5
|
||||
exit_timing_weight = 0.1
|
||||
w_dsr = 5.0
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Run all tests**
|
||||
|
||||
```bash
|
||||
SQLX_OFFLINE=true cargo test -p ml --lib -- training_profile
|
||||
SQLX_OFFLINE=true cargo test -p ml-dqn --lib
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add config/training/dqn-production.toml config/training/dqn-smoketest.toml
|
||||
git commit -m "feat: v9 reward weights — DSR primary, directional PnL demoted, spread capture amplified"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Smoke test + H100 baseline
|
||||
|
||||
**Files:** (no changes — verification only)
|
||||
|
||||
- [ ] **Step 1: Run smoke test**
|
||||
|
||||
```bash
|
||||
FOXHUNT_TEST_DATA=test_data/futures-baseline SQLX_OFFLINE=true cargo test -p ml --lib -- smoke_tests::feature_coverage --ignored --test-threads=1 --nocapture 2>&1 | grep -E "Sharpe|WinRate|Action div|Q-value"
|
||||
```
|
||||
|
||||
Expected: WinRate should start shifting (maybe not immediately in 10-epoch smoketest, but the reward structure change should be visible in the Sharpe metrics).
|
||||
|
||||
- [ ] **Step 2: Run full smoke test suite**
|
||||
|
||||
```bash
|
||||
FOXHUNT_TEST_DATA=test_data/futures-baseline SQLX_OFFLINE=true cargo test -p ml --lib -- smoke_tests --ignored --test-threads=1
|
||||
```
|
||||
|
||||
Expected: 19 passed, 0 failed.
|
||||
|
||||
- [ ] **Step 3: Commit and push**
|
||||
|
||||
```bash
|
||||
git add -A
|
||||
git commit -m "feat: v9 DSR reward restructure — Sharpe-optimal training for market-making alpha"
|
||||
git push origin main
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Launch H100 baseline**
|
||||
|
||||
```bash
|
||||
argo submit -n foxhunt --from=wftmpl/train -p model=dqn -p gpu-pool=ci-training-h100 -p hyperopt-trials=0 -p train-epochs=200
|
||||
```
|
||||
|
||||
Monitor: watch for WinRate climbing above 50% and Sharpe trajectory.
|
||||
Reference in New Issue
Block a user