# GPU Composite Reward Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Replace the raw PnL reward in the CUDA experience kernel with an 8-component composite reward (DSR, normalized PnL, drawdown penalty, idle penalty, regime scaling, asymmetric loss, position-time decay, transaction costs) — all GPU-native, zero CPU involvement. **Architecture:** Extend `experience_env_step` kernel with composite reward computation. Portfolio state buffer grows from 3→12 floats per episode. 7 new reward weights flow through TOML config → `DQNHyperparameters` → `ExperienceCollectorConfig` → kernel args. Hyperopt search space grows from 31D→38D. **Tech Stack:** CUDA C (experience kernel), Rust (config pipeline, hyperopt adapter), TOML (training profiles) **Spec:** `docs/superpowers/specs/2026-03-22-gpu-composite-reward-design.md` --- ## File Structure | File | Responsibility | Action | |------|---------------|--------| | `crates/ml/src/cuda_pipeline/experience_kernels.cu` | GPU reward computation | Modify: new kernel params, composite reward formula, stride 3→12 | | `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs` | Buffer allocation, kernel launch | Modify: PORTFOLIO_STRIDE, allocation, init, launch args | | `crates/ml/src/trainers/dqn/config.rs` | DQNHyperparameters struct | Modify: add 7 reward fields | | `crates/ml/src/training_profile.rs` | TOML deserialization | Modify: add RewardSection, apply_to mapping | | `crates/ml/src/hyperopt/adapters/dqn.rs` | Hyperopt parameter space | Modify: DQNParams 31D→38D, bounds, from/to_continuous | | `config/training/dqn-production.toml` | Production config | Modify: add [reward] section | | `config/training/dqn-smoketest.toml` | Smoketest config | Modify: add [reward] section | | `config/training/dqn-hyperopt.toml` | Hyperopt search bounds | Modify: add reward search bounds + phase_fast | | `crates/ml-dqn/src/reward.rs` | CPU reward (to be gutted) | Modify: remove RewardNormalizer, CPU DSR | --- ### Task 1: CUDA Kernel — Composite Reward **Files:** - Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu` - [ ] **Step 1: Add PORTFOLIO_STRIDE constant and update stride references** At top of file, add: ```cuda #define PORTFOLIO_STRIDE 12 ``` Replace all `* 3` portfolio stride references: - Line 169: `portfolio_states + (long long)i * 3` → `* PORTFOLIO_STRIDE` - Line 420: `portfolio_states + (long long)i * 3` → `* PORTFOLIO_STRIDE` Do NOT change `portfolio_sim_kernel` (uses stride 8, separate path). - [ ] **Step 2: Add reward parameters to `experience_env_step` kernel signature** Add after existing `float hold_reward` parameter (which will be removed): ```cuda float w_dsr, float w_pnl, float w_dd, float w_idle, float dd_threshold, float loss_aversion, float time_decay_rate, float eta, const float* __restrict__ features, int market_dim ``` Remove `float hold_reward` parameter. - [ ] **Step 3: Replace reward computation (lines 454-465) with composite formula** Replace the current: ```cuda float reward = pnl - tx_cost; if (fabsf(position) < 0.001f) { reward += hold_reward; } ``` With the full 8-component composite. Read portfolio state indices 3-11 at kernel start, write updated values at kernel end. Include: - DSR with pre-update A/B (Moody & Saffell) - Normalized PnL with running EMA - Drawdown penalty with peak_equity guard - Idle penalty (replaces hold_reward) - Regime scaling from ADX/CUSUM features - Asymmetric loss scaling - Position-time decay - All division-by-zero guards from spec - [ ] **Step 4: Update portfolio state initialization in `experience_state_gather`** Ensure indices 3-11 are preserved across steps (not overwritten by gather kernel). The gather kernel currently reads indices 0-2 for state assembly — it must NOT touch indices 3-11. - [ ] **Step 5: Compile check** ```bash SQLX_OFFLINE=true cargo check -p ml 2>&1 | tail -5 ``` Expected: compilation errors from Rust side (kernel params changed but Rust not updated yet). CUDA `.cu` file compiles via NVRTC at runtime. - [ ] **Step 6: Commit** ```bash git add crates/ml/src/cuda_pipeline/experience_kernels.cu git commit -m "feat: 8-component composite reward CUDA kernel" ``` --- ### Task 2: Rust Config Pipeline — DQNHyperparameters + TOML Profile **Files:** - Modify: `crates/ml/src/trainers/dqn/config.rs` - Modify: `crates/ml/src/training_profile.rs` - Modify: `config/training/dqn-production.toml` - Modify: `config/training/dqn-smoketest.toml` - Modify: `config/training/dqn-hyperopt.toml` - [ ] **Step 1: Add 7 reward fields to DQNHyperparameters** In `config.rs`, add to the struct (near existing `hold_penalty` / `transaction_cost_multiplier`): ```rust pub w_dsr: f64, pub w_pnl: f64, pub w_dd: f64, pub w_idle: f64, pub dd_threshold: f64, pub loss_aversion: f64, pub time_decay_rate: f64, ``` Set defaults in `conservative()`: ```rust w_dsr: 1.0, w_pnl: 0.3, w_dd: 1.0, w_idle: 0.01, dd_threshold: 0.02, loss_aversion: 1.5, time_decay_rate: 0.0005, ``` Remove `hold_penalty` field (replaced by `w_idle`). - [ ] **Step 2: Add RewardSection to training_profile.rs** ```rust #[derive(Debug, Clone, Deserialize, Default)] pub struct RewardSection { pub w_dsr: Option, pub w_pnl: Option, pub w_dd: Option, pub w_idle: Option, pub dd_threshold: Option, pub loss_aversion: Option, pub time_decay_rate: Option, } ``` Add `pub reward: Option` to `DqnTrainingProfile`. Add `apply_to` mapping in the `[reward]` section handler. - [ ] **Step 3: Add [reward] section to all 3 TOML profiles** All profiles get the same defaults: ```toml [reward] w_dsr = 1.0 w_pnl = 0.3 w_dd = 1.0 w_idle = 0.01 dd_threshold = 0.02 loss_aversion = 1.5 time_decay_rate = 0.0005 ``` Remove `hold_reward` from `[experience]` sections. - [ ] **Step 4: Compile and run profile tests** ```bash SQLX_OFFLINE=true cargo test -p ml --lib training_profile -- --test-threads=1 ``` Expected: all profile tests pass. - [ ] **Step 5: Commit** ```bash git add crates/ml/src/trainers/dqn/config.rs crates/ml/src/training_profile.rs config/training/ git commit -m "feat: reward config pipeline — DQNHyperparameters + TOML profiles" ``` --- ### Task 3: Experience Collector — Buffer Allocation + Kernel Launch **Files:** - Modify: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs` - [ ] **Step 1: Add PORTFOLIO_STRIDE constant and update allocation** ```rust const PORTFOLIO_STRIDE: usize = 12; ``` Replace all `* 3` with `* PORTFOLIO_STRIDE` at lines 575, 579, 581, 1166, 1168. Update portfolio_init to set indices 7 and 9 to `initial_capital`: ```rust portfolio_init[off + 7] = initial_capital; // peak_equity portfolio_init[off + 9] = initial_capital; // prev_equity ``` - [ ] **Step 2: Add reward fields to ExperienceCollectorConfig** ```rust pub w_dsr: f32, pub w_pnl: f32, pub w_dd: f32, pub w_idle: f32, pub dd_threshold: f32, pub loss_aversion: f32, pub time_decay_rate: f32, ``` Remove `hold_reward` field. Remove `use_dsr` field — DSR is always enabled, no opt-out path. - [ ] **Step 3: Update kernel launch args** In the `launch_timestep_loop` method, replace `hold_rw` kernel arg with the 7 reward weights + feature buffer pointer + market_dim. The features buffer pointer is already available as `self.features_buf` (the market data uploaded for state_gather). Pass it to `experience_env_step` as well. - [ ] **Step 4: Update all callers that construct ExperienceCollectorConfig** Search for all places that create `ExperienceCollectorConfig` and add the new fields. The training loop constructs it in `collect_gpu_experiences()`. - [ ] **Step 5: Compile check** ```bash SQLX_OFFLINE=true cargo check -p ml 2>&1 | tail -5 ``` Expected: clean compilation. - [ ] **Step 6: Commit** ```bash git add crates/ml/src/cuda_pipeline/gpu_experience_collector.rs git commit -m "feat: experience collector — PORTFOLIO_STRIDE=12, reward kernel args" ``` --- ### Task 4: Hyperopt Adapter — 31D → 38D Parameter Space **Files:** - Modify: `crates/ml/src/hyperopt/adapters/dqn.rs` - Modify: `config/training/dqn-hyperopt.toml` - [ ] **Step 1: Add 7 reward fields to DQNParams struct** Add after existing fields: ```rust pub w_dsr: f64, pub w_pnl: f64, pub w_dd: f64, pub w_idle: f64, pub dd_threshold: f64, pub loss_aversion: f64, pub time_decay_rate: f64, ``` Remove `hold_penalty_weight` field. - [ ] **Step 2: Update continuous_bounds() — add 7 dimensions** Add bounds loaded from TOML `[search_space]`: ```rust let w_dsr_b = b("w_dsr", (0.1, 2.0)); let w_pnl_b = b("w_pnl", (0.0, 1.0)); let w_dd_b = b("w_dd", (0.0, 5.0)); let w_idle_b = b("w_idle", (0.0, 0.1)); let dd_thresh_b = b("dd_threshold", (0.01, 0.10)); let loss_av_b = b("loss_aversion", (1.0, 3.0)); let time_dec_b = b("time_decay_rate", (0.0001, 0.005)); ``` Append to bounds vec (indices 31-37). Remove `hold_penalty_weight` bound. - [ ] **Step 3: Update from_continuous() and to_continuous()** `from_continuous`: expect 38 elements, map indices 31-37 to reward fields. `to_continuous`: output 38 elements, append reward fields. `param_names`: add 7 names, remove `hold_penalty_weight`. - [ ] **Step 4: Update [search_space] and [phase_fast] in dqn-hyperopt.toml** ```toml [search_space] # ... existing ... w_dsr = [0.1, 2.0] w_pnl = [0.0, 1.0] w_dd = [0.0, 5.0] w_idle = [0.0, 0.1] dd_threshold = [0.01, 0.10] loss_aversion = [1.0, 3.0] time_decay_rate = [0.0001, 0.005] [phase_fast] # ... existing ... w_dsr = 1.0 w_pnl = 0.3 w_dd = 1.0 w_idle = 0.01 dd_threshold = 0.02 loss_aversion = 1.5 time_decay_rate = 0.0005 ``` - [ ] **Step 5: Update SearchSpaceSection in training_profile.rs** Add 7 `Option<[f64; 2]>` fields to `SearchSpaceSection` and the `bound()` match. - [ ] **Step 6: Fix all tests that reference 31D or hold_penalty_weight** Update `test_dqn_params_bounds`, `test_qr_dqn_activation_threshold`, and any other tests that assert dimension count or specific param indices. - [ ] **Step 7: Run hyperopt tests** ```bash SQLX_OFFLINE=true cargo test -p ml --lib hyperopt -- --test-threads=1 ``` Expected: all pass with 38D search space. - [ ] **Step 8: Commit** ```bash git add crates/ml/src/hyperopt/adapters/dqn.rs crates/ml/src/training_profile.rs config/training/dqn-hyperopt.toml git commit -m "feat: hyperopt 31D→38D — 7 reward weight dimensions" ``` --- ### Task 5: Remove Dead CPU Reward Code **Files:** - Modify: `crates/ml-dqn/src/reward.rs` - Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs` - [ ] **Step 1: Remove RewardNormalizer usage from training loop** The training loop may call `reward_fn.normalize()` or similar. Remove those calls — normalization is now in the GPU kernel. - [ ] **Step 2: Remove CPU DSR code from reward.rs** Keep the file but remove the `RewardNormalizer` struct and the CPU DSR computation. Keep any types or traits that are still referenced. - [ ] **Step 3: Remove `use_dsr` from ExperienceCollectorConfig callers** Search for all `use_dsr` references and remove them. `w_dsr > 0` implies DSR enabled. - [ ] **Step 4: Compile check** ```bash SQLX_OFFLINE=true cargo check --workspace 2>&1 | grep "error\[" | head -5 ``` Expected: clean or only unrelated warnings. - [ ] **Step 5: Commit** ```bash git add crates/ml-dqn/src/reward.rs crates/ml/src/trainers/dqn/trainer/training_loop.rs git commit -m "refactor: remove dead CPU reward code (DSR + normalizer now in GPU kernel)" ``` --- ### Task 6: Integration Test — Composite Reward on GPU **Files:** - Test with existing: `crates/ml/tests/dqn_training_smoke_test.rs` - [ ] **Step 1: Run smoke test locally** ```bash SQLX_OFFLINE=true cargo test -p ml --test dqn_training_smoke_test -- --nocapture --test-threads=1 ``` Expected: passes with composite reward (TOML profile provides defaults). Q-values should be non-zero. Loss should decrease. - [ ] **Step 2: Run full hyperopt test suite** ```bash SQLX_OFFLINE=true cargo test -p ml --lib hyperopt -- --test-threads=1 ``` Expected: all pass with 38D space. - [ ] **Step 3: Run local hyperopt smoketest (2 trials, 2 epochs)** ```bash SQLX_OFFLINE=true timeout 60 ./target/release/examples/hyperopt_baseline_rl \ --model dqn --phase fast --trials 6 --n-initial 5 --epochs 2 \ --data-dir test_data/futures-baseline --symbol ES.FUT \ --output /tmp/reward_test.json --seed 42 ``` Expected: 6 trials complete, Q-values non-zero, Sharpe improves from trial 1. - [ ] **Step 4: Run all 17 GPU tests** ```bash SQLX_OFFLINE=true cargo test -p ml -p ml-dqn -p ml-core -- --test-threads=1 ``` Expected: all pass. - [ ] **Step 5: Commit and push** ```bash git push origin main ``` --- ### Task 7: H100 Validation - [ ] **Step 1: Submit GPU test pipeline** Verify 17/17 pass on H100 with composite reward. - [ ] **Step 2: Submit hyperopt smoketest (6 trials × 10 epochs)** Verify non-zero Q-values, improving Sharpe, all trials complete. - [ ] **Step 3: Compare metrics** Compare best Sharpe, win rate, max drawdown with the old PnL-only results (best was Sharpe 0.06, win rate 3%). The composite reward should show improvement.