From 3f3d32d5ba14fd61394a7c48fd481cf0b9b12260 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Thu, 16 Apr 2026 01:24:03 +0200 Subject: [PATCH] spec: trade-level reward attribution + exploration risk budget + homeostatic regularization MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two fundamental fixes for training Sharpe breakthrough: 1. Trade-level rewards: replace per-bar noise (SNR=0.01) with trade P&L attribution (SNR=0.1-0.5). C51 atoms model trade outcome distributions, not random walk noise. 2. Exploration risk budget: protection stack (CVaR, epistemic gate, commitment, DSR) scaled by iqn_readiness². Loose during exploration, tight when converged. Model can discover edges before being punished. Also: homeostatic regularization spec (unified adaptive penalties). Co-Authored-By: Claude Opus 4.6 (1M context) --- ...04-16-homeostatic-regularization-design.md | 155 +++++++++++++ ...6-trade-level-reward-exploration-design.md | 205 ++++++++++++++++++ 2 files changed, 360 insertions(+) create mode 100644 docs/superpowers/specs/2026-04-16-homeostatic-regularization-design.md create mode 100644 docs/superpowers/specs/2026-04-16-trade-level-reward-exploration-design.md diff --git a/docs/superpowers/specs/2026-04-16-homeostatic-regularization-design.md b/docs/superpowers/specs/2026-04-16-homeostatic-regularization-design.md new file mode 100644 index 000000000..36d4c3219 --- /dev/null +++ b/docs/superpowers/specs/2026-04-16-homeostatic-regularization-design.md @@ -0,0 +1,155 @@ +# Homeostatic Regularization — Design Spec + +**Goal:** Unified self-regulating framework that maintains healthy training dynamics by adapting penalty strengths based on deviation from homeostatic targets. Replaces ad-hoc fixed lambdas (Q-mean drift=0.01, branch independence=0.01, temporal consistency=0.005) with a single principled mechanism borrowed from biological neural homeostasis. + +**Problem:** Multiple training observables can drift or collapse (Q-mean, atom utilization, branch correlation, trade frequency, Q-gap). Each has its own fixed lambda that was hand-tuned. When one observable drifts faster than expected (Q-mean reached +4.89 by epoch 10 with lambda=0.01), the fixed penalty is too weak. When it's stable, the penalty wastes gradient budget. No coordinated response — each penalty operates independently. + +**Architecture:** One CUDA kernel `homeostatic_regularizer` reads N observable scalars from pinned device-mapped memory, compares to N adaptive targets, produces N penalty gradients scaled by deviation². Targets self-calibrate from the first 5 epochs' healthy range. Penalties coordinate — if multiple observables drift simultaneously, total penalty budget is capped to prevent gradient domination. + +--- + +## The Mechanism + +Biological neurons maintain homeostasis through intrinsic plasticity: fire too much → excitability decreases. Fire too little → excitability increases. The system converges to a target firing rate without external tuning. + +Applied to DQN training: + +``` +For each observable k (Q-mean, atom_util, branch_corr, trade_freq, Q-gap): + error_k = (observed_k - target_k) / max(|target_k|, eps) // normalized error + penalty_k = lambda_base * error_k^2 * sign(error_k) // quadratic, signed + + // Adaptive target: EMA of healthy range (epochs 1-5) + if epoch <= 5: + target_k = EMA(observed_k, alpha=0.3) // calibrate + // else: target frozen, penalties active + + // Budget cap: total penalty gradient <= 10% of C51 gradient + total_penalty = sum(|penalty_k|) + if total_penalty > budget: + scale = budget / total_penalty + penalty_k *= scale +``` + +## Observables and Targets + +| Observable | Target | Why | Source | +|-----------|--------|-----|--------| +| Q-mean | 0.0 | Prevents bootstrapping drift | q_mean_scratch_pinned | +| Atom utilization | 0.85 | Keeps C51 resolution healthy | q_stats_buf[6] | +| Max branch correlation | 0.1 | Forces orthogonal branch features | branch_indep_penalty_buf | +| Trade frequency | calibrated | Prevents churn (too high) or inactivity (too low) | GPU epoch summary | +| Q-gap | calibrated | Prevents collapse (too low) or explosion (too high) | per_branch_q_gaps | +| Q-variance | calibrated | Prevents overconfidence (too low) or chaos (too high) | q_stats_buf[4] | + +## Implementation + +### CUDA Kernel + +```cuda +extern "C" __global__ void homeostatic_regularizer( + const float* __restrict__ observables, /* [N_OBS] current values (pinned) */ + const float* __restrict__ targets, /* [N_OBS] homeostatic targets (pinned) */ + float* __restrict__ penalties, /* [N_OBS] output penalty gradients */ + float* __restrict__ total_penalty, /* [1] sum of |penalty_k| for budget cap */ + int n_obs, + float lambda_base, /* 0.01 */ + float budget_fraction /* 0.10 = max 10% of C51 gradient */ +) { + // Single-block kernel (n_obs <= 8) + int k = threadIdx.x; + if (k >= n_obs) return; + + float obs = observables[k]; + float tgt = targets[k]; + float denom = fmaxf(fabsf(tgt), 1e-6f); + float error = (obs - tgt) / denom; + + // Quadratic signed penalty + float pen = lambda_base * error * fabsf(error); + penalties[k] = pen; + + // Atomic sum for budget cap + atomicAdd(total_penalty, fabsf(pen)); +} +``` + +### Rust Integration + +```rust +struct HomeostaticState { + observables_pinned: *mut f32, // [N_OBS] pinned device-mapped + observables_dev_ptr: u64, + targets_pinned: *mut f32, // [N_OBS] pinned device-mapped + targets_dev_ptr: u64, + penalties_buf: CudaSlice, // [N_OBS] + total_penalty_buf: CudaSlice, // [1] + kernel: CudaFunction, + calibration_epoch: usize, // epoch at which targets freeze (5) + n_obs: usize, // number of observables (6) +} +``` + +### Observable Update (every training step) + +```rust +fn update_observables(&self) { + unsafe { + let obs = self.observables_pinned; + *obs.add(0) = *self.q_mean_scratch_pinned; // Q-mean + *obs.add(1) = self.utilization_ema; // atom util + *obs.add(2) = 0.0; // branch corr (read from penalty buf after compute) + *obs.add(3) = self.epoch_trade_count as f32; // trade freq + *obs.add(4) = self.last_q_gap; // Q-gap + *obs.add(5) = self.last_q_var; // Q-variance + } +} +``` + +### Target Calibration (epochs 1-5) + +```rust +fn calibrate_targets(&self, epoch: usize) { + if epoch > self.calibration_epoch { return; } + let alpha = 0.3_f32; + unsafe { + for k in 0..self.n_obs { + let obs = *self.observables_pinned.add(k); + let old = *self.targets_pinned.add(k); + *self.targets_pinned.add(k) = (1.0 - alpha) * old + alpha * obs; + } + } + // Override Q-mean target to always be 0.0 (no drift is always the goal) + unsafe { *self.targets_pinned.add(0) = 0.0; } +} +``` + +### Penalty Application + +The penalties feed into the C51 gradient (same path as the current Q-mean drift penalty). The `homeostatic_regularizer` kernel runs after Q-stats, and the penalty for observable 0 (Q-mean) replaces the current hardcoded drift penalty in `c51_grad_kernel.cu`. + +For branch correlation (observable 2) and Q-gap (observable 4), the penalties feed into the respective auxiliary loss accumulator buffers. + +## Interaction with Existing Components + +- **Replaces**: Q-mean drift penalty (G9d), fixed branch independence lambda (G6), fixed temporal consistency lambda (G10) +- **Coordinates with**: gamma annealing (G4 uses atom_util, homeostatic also targets atom_util — the two mechanisms reinforce each other) +- **Enhances**: cost curriculum (G2) — the sigmoid center (currently hardcoded epoch 10) should be driven by model readiness: `center = epoch where training_Sharpe_ema first exceeds 0.3`. If the model finds edges quickly, costs ramp sooner. If it struggles, costs stay low longer. This replaces the hardcoded epoch 10 with an adaptive trigger. + +## Success Criteria + +| Metric | Current (fixed lambda) | Target (homeostatic) | +|--------|----------------------|---------------------| +| Q-mean at epoch 10 | +4.89 (drifting) | < ±1.0 (contained) | +| Atom util at epoch 10 | 87% (healthy) | > 85% (maintained) | +| Training Sharpe oscillation | ±0.3 | < ±0.15 (damped) | +| Number of hand-tuned lambdas | 5+ | 1 (lambda_base) | + +## Risks + +| Risk | Mitigation | +|------|-----------| +| Calibration period too short | 5 epochs × 88s = 7 min. Sufficient for healthy range. Extend to 8 if needed. | +| Budget cap too aggressive | 10% of C51 gradient. Increase to 20% if penalties never fire. | +| Targets miscalibrated from bad early epochs | Q-mean target hardcoded to 0.0. Others have sensible fallbacks. | +| Penalty coordination masks individual issues | Log each penalty_k separately for diagnosis. | diff --git a/docs/superpowers/specs/2026-04-16-trade-level-reward-exploration-design.md b/docs/superpowers/specs/2026-04-16-trade-level-reward-exploration-design.md new file mode 100644 index 000000000..7dd9d6caf --- /dev/null +++ b/docs/superpowers/specs/2026-04-16-trade-level-reward-exploration-design.md @@ -0,0 +1,205 @@ +# Trade-Level Reward Attribution + Exploration Risk Budget — Design Spec + +**Goal:** Fix two fundamental reward problems: (1) per-bar reward is 99% noise — replace with trade-level P&L attribution for 10-50× higher SNR, (2) the risk protection stack prevents the model from discovering profitable strategies that require temporary risk-taking. + +**Problem 1 — Noisy Reward:** The reward is `next_close - close` (1-bar price change). For ES futures, single-bar returns are dominated by random walk noise. The model receives 100 bars of noise per trade, but the TRADE-LEVEL P&L (sum of those bars) is the actual signal. C51's 51 atoms model the distribution of per-bar noise instead of the distribution of trade outcomes. + +**Problem 2 — Over-Protection:** The risk protection stack (CVaR + epistemic gate + commitment penalty + cost curriculum + adaptive DSR + Q-anchoring) collectively pushes the model toward inactivity. During EXPLORATION, the model needs to take risky trades to discover which edges survive costs. The protection stack kills exploration before the model can evaluate trade outcomes. + +**Architecture:** Two complementary components: +1. **Trade-Level Reward Attribution**: accumulate P&L over trades, assign to entry decision. Per-bar holding reward = 0. +2. **Exploration Risk Budget**: during exploration (high epsilon / low iqn_readiness), relax the protection stack. As the model converges (low epsilon / high readiness), tighten protection. + +--- + +## Component 1: Trade-Level Reward Attribution + +### The Problem in Detail + +Current reward flow per bar: +``` +Bar 1: enter Long → reward = +0.02% (noise) +Bar 2: hold → reward = -0.08% (noise) +Bar 3: hold → reward = +0.15% (noise) +Bar 4: hold → reward = -0.03% (noise) +Bar 5: exit → reward = +0.01% (noise) + ───────────────── + Total: +0.07% (signal, but model never sees this as one number) +``` + +The model gets 5 separate rewards. The C51 distribution learns: "this state produces returns centered at +0.02% with std=0.08%" — that's the PER-BAR noise distribution, not the trade outcome distribution. + +### The Solution + +Track trade state in the env_step kernel. When a trade closes (position changes or episode ends): +1. Compute total trade P&L = (exit_price - entry_price) × direction × size - costs +2. Assign trade P&L as reward to the ENTRY bar +3. Holding bars get reward = 0 (no decision to evaluate) + +``` +Bar 1: enter Long → reward = 0 (pending — trade not closed yet) +Bar 2: hold → reward = 0 +Bar 3: hold → reward = 0 +Bar 4: hold → reward = 0 +Bar 5: exit → reward = +0.07% - costs (assigned to bar 1 retroactively via n-step) +``` + +### Implementation + +In `experience_kernels.cu` env_step kernel: + +```cuda +// Per-episode trade tracking state (in episode_states buffer): +// trade_entry_price: float — price at position entry +// trade_entry_bar: int — bar index of entry (for retroactive attribution) +// trade_direction: int — +1 Long, -1 Short, 0 Flat + +// When position changes: +if (new_position != old_position) { + if (old_position != 0) { + // Close existing trade + float trade_pnl = (current_price - trade_entry_price) + * trade_direction * fabsf(old_position) + * contract_multiplier; + trade_pnl -= spread_cost * 2.0f * cost_scale; // entry + exit costs + + // Trade-level reward replaces per-bar reward + reward = trade_pnl; + } + if (new_position != 0) { + // Open new trade + trade_entry_price = current_price; + trade_direction = (new_position > 0) ? 1 : -1; + } +} else { + // Holding — reward = small holding cost (opportunity cost of capital) + reward = -0.0001f * fabsf(old_position); // tiny negative to discourage idle holding +} +``` + +### State Buffers + +Need 3 new per-episode floats in the episode state buffer: +- `trade_entry_price: f32` — saved at position entry +- `trade_direction: f32` — +1/-1/0 +- `bars_held: i32` — counter for holding period statistics + +These are already tracked by the portfolio simulator (`portfolio_states` buffer). The implementation reuses the existing portfolio state rather than adding new buffers. + +### Interaction with Existing Reward Pipeline + +- **DSR shaping**: operates on trade-level P&L (amplifies drawdown penalty per trade, not per bar) +- **Rank normalization (G13)**: ranks trade P&Ls instead of per-bar noise — MUCH higher signal quality +- **Cost curriculum (G2)**: costs already subtracted from trade P&L +- **Commitment penalty (G15)**: fires at position change (entry/exit) — synergistic +- **Counterfactual flip (G7)**: flips the entire trade, not individual bars — cleaner symmetry + +### Backwards Compatibility + +The per-bar reward is not removed — it's replaced for bars where a trade closes. Holding bars get reward=0 instead of `next_close - close`. The n-step return computation accumulates these zeros naturally (gamma^k × 0 = 0 for holding bars). + +--- + +## Component 2: Exploration Risk Budget + +### The Problem in Detail + +The risk protection stack collectively applies: +``` +CVaR: alpha=0.3-0.5 → targets bottom 30-50% of returns +Epistemic gate: scales magnitude toward Small when uncertain +Commitment penalty: -0.01 per position change +Cost curriculum: 0-100% spread costs +Adaptive DSR: w_dsr up to 3× when Sharpe negative +Q-anchoring: anchors to Flat (doing nothing = zero penalty) +``` + +During exploration (epsilon=0.05, iqn_readiness=0.1), the model takes random actions. The protection stack punishes ALL random actions heavily. The model learns: "random actions are bad → stay Flat." But random actions are HOW the model discovers which state→action pairs are profitable. + +### The Solution + +Scale the protection stack by exploration readiness: + +``` +protection_scale = iqn_readiness^2 // 0 at start, 1 when converged + +CVaR alpha: 0.5 * (1 + protection_scale * 0.8) // 0.5→0.9 (balanced→risk-averse) +Epistemic gate: threshold *= (1 + 2 * protection_scale) // looser early, tighter late +Commitment penalty: *= protection_scale // zero early, full late +DSR multiplier: clamp(1 - sharpe_ema, 0.5, 0.5 + 2.5 * protection_scale) // narrower early +``` + +When iqn_readiness=0 (exploring): CVaR=0.5 (balanced), no commitment penalty, no epistemic gate, narrow DSR. The model freely explores. + +When iqn_readiness=1 (converged): CVaR=0.9 (risk-averse), full commitment penalty, tight epistemic gate, wide DSR. The model is protected. + +### Implementation + +Single pinned device-mapped scalar `protection_scale` written from CPU (from iqn_readiness, already computed). + +In the CUDA kernels that apply protection: + +```cuda +// CVaR (c51_loss_kernel.cu): +float alpha = 0.5f - 0.4f * protection_scale; // already uses iqn_readiness, just rename + +// Epistemic gate (experience_kernels.cu): +float effective_threshold = var_ema * (1.0f + 2.0f * protection_scale); + +// Commitment penalty (experience_kernels.cu): +float effective_commitment = 0.01f * protection_scale; + +// DSR (training_loop.rs): +let dsr_range = 0.5 + 2.5 * protection_scale as f64; +let adaptive_mult = (1.0 - sharpe_ema).clamp(0.5, 0.5 + dsr_range); +``` + +Most of these are 1-line changes to existing kernels/Rust code. + +--- + +## Combined Effect + +``` +Epoch 1-5 (exploring, readiness=0.0-0.2): + - Trade-level rewards: model sees clean trade P&L signal + - Protection: minimal — model freely discovers edges + - Learns: "Long in this regime, held 8 bars, made +$300 net of costs" + +Epoch 5-15 (learning, readiness=0.2-0.7): + - Trade-level rewards: refines trade timing and sizing + - Protection: ramping — starts penalizing bad trades + - Learns: "this edge survives costs, this one doesn't" + +Epoch 15+ (converged, readiness=0.7-1.0): + - Trade-level rewards: fine-tunes around profitable strategies + - Protection: full — conservative risk management + - Result: OOS-ready strategies that survived cost + risk pressure +``` + +--- + +## Success Criteria + +| Metric | Current (per-bar, over-protected) | Target (trade-level, exploration budget) | +|--------|----------------------------------|----------------------------------------| +| Training Sharpe by epoch 10 | oscillating ±0.3 | > 1.0 sustained | +| Reward SNR | ~0.01 (per-bar noise) | ~0.1-0.5 (trade-level signal) | +| Trade count | ~60K/epoch (churn) | ~5-15K/epoch (meaningful trades) | +| val/OOS gap | unknown (can't measure at Sharpe~0) | < 15% | +| Exploration trades tried | few (protection kills) | many early, few late | + +## Implementation Order + +1. Trade-level reward attribution (~30 lines CUDA in env_step) +2. Exploration risk budget (~10 lines across 4 files: 1 line each in CVaR, epistemic gate, commitment, DSR) +3. Trade state tracking (reuse existing portfolio_states buffer) + +## Risks + +| Risk | Mitigation | +|------|-----------| +| Trade-level reward = sparse signal (only at trade close) | Holding bars get small negative (opportunity cost), not exactly zero | +| Exploration budget too loose → model learns bad habits | protection_scale ramps with iqn_readiness² (quadratic), bites quickly | +| Long holding periods → delayed reward attribution | n-step return (n=config) handles multi-bar attribution naturally | +| Trade tracking adds state to env_step | Reuse portfolio_states buffer (already tracks position, entry price) |