gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: reward v5 — trade-aware hybrid with dynamic trailing stop

Browse Source 
Replace reward v4 (pure mark-to-market return) with reward v5, a two-component
trade-aware hybrid that separates dense per-bar signal from sparse trade-exit signal:

Dense (every bar, weight 0.1): raw_pnl / equity when in a trade, zero when flat.
Keeps gradients flowing without overwhelming the sparse trade completion signal.

Sparse (at trade exit, weight 2.0): trade_return * patience_multiplier where
patience = sqrt(hold_time / expected_hold). Regime-adaptive expected hold via
ADX: trending (ADX>30) = 20 bars, ranging (ADX<20) = 8 bars, default = 12 bars.

Dynamic trailing stop: regime-adaptive trail distance (0.5% base, widens with
volatility via CUSUM and trend via ADX). Activates when trade is profitable and
held > 2 bars. Locks in profits by forcing exit when unrealized P&L drops below
the trailing floor.

Also fixes kernel signature mismatch: removes 7 old reward v2 parameters
(w_dsr, w_pnl, w_dd, w_idle, dd_threshold, time_decay_rate, eta) that were
already removed from the Rust launcher in a prior commit.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-03-24 22:16:31 +01:00
parent be2672596d
commit 263997ad31
2 changed files with 162 additions and 315 deletions
Download Patch File Download Diff File Expand all files Collapse all files

475
crates/ml/src/cuda_pipeline/experience_kernels.cu
View File

@@ -23,18 +23,18 @@
* [market_dim .. market_dim+3) : portfolio features (value_norm, position, cash_norm)
* [market_dim+3 .. state_dim) : zero-pad for tensor-core alignment
*
* Portfolio state layout used by experience kernels ([N, PORTFOLIO_STRIDE=12]):
* Portfolio state layout used by experience kernels ([N, PORTFOLIO_STRIDE=20]):
* [0] position — current contract position (signed)
* [1] cash — cash balance
* [2] portfolio_value — mark-to-market total value (cash + position * price)
* [3] dsr_A — EMA of returns (Moody & Saffell DSR)
* [4] dsr_B — EMA of squared returns
* [5] pnl_ema — running mean PnL (for z-score normalization)
* [6] pnl_var — running variance PnL
* [3] (reserved) — was dsr_A, unused by reward v5
* [4] (reserved) — was dsr_B, unused by reward v5
* [5] (reserved) — was pnl_ema, unused by reward v5
* [6] (reserved) — was pnl_var, unused by reward v5
* [7] peak_equity — high-water mark (init to initial_capital)
* [8] flat_counter — consecutive flat steps (float for GPU simplicity)
* [9] prev_equity — equity at previous step (init to initial_capital)
* [10] hold_time — consecutive steps with position
* [10] hold_time — consecutive steps with position (total, not just losing)
* [11] realized_pnl — cumulative realized PnL
*
* Branching DQN (Tavakoli et al., 2018):
@@ -432,23 +432,22 @@ extern "C" __global__ void experience_action_select(
/* ================================================================== */
/**
* Portfolio simulation, 8-component composite reward, and done detection.
* Portfolio simulation, reward v5 (trade-aware hybrid), and done detection.
*
* For each episode i this kernel:
* 1. Reads the current-bar raw prices from targets[bar_idx].
* 2. Decodes the factored action to an exposure branch index.
* 3. Computes target position = exposure_fraction * max_position.
* 4. Applies position adjustment delta with a flat proportional tx cost.
* 5. Computes composite reward:
* reward = eff_w_dsr * dsr_t (Sharpe contribution)
* + eff_w_pnl * asym_pnl (asymmetric normalized PnL)
* - eff_w_dd * drawdown_penalty_t (drawdown penalty)
* - w_idle * idle_penalty_t (idle penalty)
* - hold_time_penalty (position staleness rent)
* - tx_cost (transaction friction)
* 6. Writes (batch_states, action, reward, done) to output replay buffer.
* 7. Updates portfolio_states[0..11] in place.
* 8. Increments current_timesteps[i].
* 4. Applies position adjustment delta with volatility-scaled tx cost.
* 5. Runs dynamic trailing stop (regime-adaptive).
* 6. Computes reward v5:
* dense = raw_pnl / equity (only when in trade, keeps gradients flowing)
* sparse = trade_return * patience_mult (at trade exit, primary signal)
* reward = 0.1 * dense + 2.0 * sparse
* if reward < 0: reward *= loss_aversion (prospect theory)
* 7. Writes (batch_states, action, reward, done) to output replay buffer.
* 8. Updates portfolio_states[0..19] in place.
* 9. Increments current_timesteps[i].
*
* Grid: ceil(N / 256), Block: 256. One thread per episode.
*
@@ -458,14 +457,14 @@ extern "C" __global__ void experience_action_select(
* [2] raw_close — raw close price (for position cost + tx)
* [3] raw_next — raw next-bar close (for mark-to-market PnL)
*
* portfolio_states layout: [N, PORTFOLIO_STRIDE=12] (read-write)
* portfolio_states layout: [N, PORTFOLIO_STRIDE=20] (read-write)
* See file header for field definitions.
*
* @param targets [total_bars, 4] price data
* @param episode_starts [N] global bar offset per episode
* @param current_timesteps [N] current episode step (incremented)
* @param actions [N] factored action from action_select
* @param portfolio_states [N, 12] full per-episode state (read-write)
* @param portfolio_states [N, 20] full per-episode state (read-write)
* @param out_states [N, L, state_dim] replay output: states
* @param out_actions [N, L] replay output: actions
* @param out_rewards [N, L] replay output: rewards
@@ -473,24 +472,19 @@ extern "C" __global__ void experience_action_select(
* @param batch_states [N, state_dim] current state batch from gather kernel
* @param max_position max allowed contract position (absolute)
* @param tx_cost_multiplier proportional transaction cost rate
* @param w_dsr DSR (Sharpe) weight
* @param w_pnl normalized PnL weight
* @param w_dd drawdown penalty weight
* @param w_idle idle penalty weight
* @param dd_threshold drawdown tolerance before penalty
* @param loss_aversion asymmetric loss scaling factor
* @param time_decay_rate position staleness rent per step
* @param eta DSR/normalization EMA decay rate
* @param loss_aversion asymmetric loss scaling factor (~1.5)
* @param features [total_bars, market_dim] market features for regime detection
* @param market_dim MARKET_DIM (42)
* @param L episode length (replay buffer columns)
* @param N number of episodes
* @param total_bars length of targets/market_features
* @param state_dim state vector length
* @param b0_size exposure branch size (default 5)
* @param b0_size exposure branch size (default 9)
* @param b1_size order branch size (default 3)
* @param b2_size urgency branch size (default 3)
* @param current_t timestep index (0-based) in this batch
* @param cvar_scales [N] or NULL CVaR position scaling
* @param q_gaps [N] or NULL Q-gap conviction scaling
*/
extern "C" __global__ void experience_env_step(
const float* __restrict__ targets,
@@ -505,14 +499,7 @@ extern "C" __global__ void experience_env_step(
const float* __restrict__ batch_states,
float max_position,
float tx_cost_multiplier,
float w_dsr,
float w_pnl,
float w_dd,
float w_idle,
float dd_threshold,
float loss_aversion,
float time_decay_rate,
float eta,
const float* __restrict__ features,
int market_dim,
int L,
@@ -562,15 +549,12 @@ extern "C" __global__ void experience_env_step(
if (raw_close <= 0.0f) raw_close = 1.0f;
if (raw_next <= 0.0f) raw_next = raw_close;
/* ---- Read full portfolio state (PORTFOLIO_STRIDE=12) ---- */
/* ---- Read full portfolio state (PORTFOLIO_STRIDE=20) ---- */
float* ps = portfolio_states + (long long)i * PORTFOLIO_STRIDE;
float position = ps[0];
float cash = ps[1];
/* ps[2] = portfolio_value (updated at end) */
float dsr_A = ps[3];
float dsr_B = ps[4];
float pnl_ema = ps[5];
float pnl_var = ps[6];
/* ps[3:6] reserved (unused by reward v5) */
float peak_equity = ps[7];
float flat_counter = ps[8];
float prev_equity = ps[9];
@@ -696,78 +680,9 @@ extern "C" __global__ void experience_env_step(
/* ---- Mark-to-market PnL for this timestep ---- */
float raw_pnl = position * (raw_next - raw_close);
/* prev_pos needed for profit-taking bonus (computed in reward v2 block) */
float prev_pos = ps[0]; /* still the pre-delta value (writeback at kernel end) */
/* ==================================================================== */
/* REWARD v2: Each component individually clamped to [-1, +1] BEFORE */
/* weighting. No global clamp. Components are additive with clear */
/* semantics: positive = good action, negative = bad action. */
/* */
/* Design principles (from root cause analysis): */
/* 1. DSR denominator floor = 0.01 (was 1e-12 → millions early) */
/* 2. Each component clamped individually (was: global [-1,+1]) */
/* 3. Positive reward achievable (was: 6/7 components ≤ 0) */
/* 4. No confidence scaling (was: positive feedback loop) */
/* 5. No profit_take_bonus (was: 100x reward hacking) */
/* ==================================================================== */
/* ==== Component 1: Differential Sharpe Ratio (Moody & Saffell 2001) ==== */
float equity_denom = (prev_equity > 1.0f) ? prev_equity : 1.0f;
float R_t = raw_pnl / equity_denom;
float delta_A = R_t - dsr_A;
float delta_B = R_t * R_t - dsr_B;
float variance = dsr_B - dsr_A * dsr_A;
float abs_var = (variance >= 0.0f) ? variance : -variance;
float var_32 = abs_var * sqrtf(abs_var + 1e-8f); /* |variance|^(3/2) */
/* FIX: floor = 0.01 not 1e-12. Old floor produced DSR in millions
* early in episodes when variance ≈ 0, drowning all other signal. */
float dsr_denom = (var_32 > 0.01f) ? var_32 : 0.01f;
float dsr_t = (dsr_B * delta_A - 0.5f * dsr_A * delta_B) / dsr_denom;
/* NaN guard: if any DSR input was NaN (from prior accumulation), zero out */
if (isnan(dsr_t) || isinf(dsr_t)) dsr_t = 0.0f;
/* DSR warm-up: skip for first 50 steps when EMA has insufficient history.
* Early DSR values are noise (numerator/denominator both near zero).
* The 0.01 floor prevents millions, but the signal is still unreliable. */
if (t < 50) dsr_t = 0.0f;
/* Clamp DSR individually to [-1, +1] BEFORE weighting */
dsr_t = fmaxf(-1.0f, fminf(1.0f, dsr_t));
dsr_A = dsr_A + eta * delta_A;
dsr_B = dsr_B + eta * delta_B;
/* ==== Component 2: Normalized PnL (z-scored, clamped) ==== */
pnl_ema = pnl_ema + eta * (raw_pnl - pnl_ema);
float pnl_diff = raw_pnl - pnl_ema;
pnl_var = pnl_var + eta * (pnl_diff * pnl_diff - pnl_var);
float pnl_std = sqrtf(pnl_var + 1e-8f);
float normalized_pnl = pnl_diff / pnl_std;
if (isnan(normalized_pnl) || isinf(normalized_pnl)) normalized_pnl = 0.0f;
/* Clamp z-score to [-3, +3] (99.7% of distribution). Old code
* allowed ±10+ outliers that dominated after asymmetric scaling. */
normalized_pnl = fmaxf(-3.0f, fminf(3.0f, normalized_pnl));
/* Asymmetric: losses hurt 1.5x more (prospect theory) */
float asym_pnl = (normalized_pnl >= 0.0f)
? normalized_pnl / 3.0f /* scale to [0, 1] */
: normalized_pnl * loss_aversion / 3.0f; /* scale to [-1.5, 0] then /3 → [-0.5, 0] */
/* Result: asym_pnl in [-0.5, +0.33] — bounded, asymmetric */
/* ==== Component 3: Drawdown penalty (clamped) ==== */
/* ---- Mark-to-market equity (needed for peak_equity and reward) ---- */
float equity = cash + position * raw_next;
peak_equity = (equity > peak_equity) ? equity : peak_equity;
float drawdown = 0.0f;
if (peak_equity > 0.0f) {
drawdown = (peak_equity - equity) / peak_equity;
}
float drawdown_penalty_t = (drawdown > dd_threshold)
? (drawdown - dd_threshold)
: 0.0f;
/* Clamp drawdown penalty to [0, 1]. Drawdown is already fractional [0,1]
* so this rarely triggers, but guards against numerical issues. */
drawdown_penalty_t = fminf(1.0f, drawdown_penalty_t);
/* ==== Trade lifecycle tracking (v3) ==== */
/* ps[12] = entry_price, ps[13] = trade_start_pnl — dedicated slots,
@@ -786,23 +701,114 @@ extern "C" __global__ void experience_env_step(
trade_start_pnl = ps[11]; /* cumulative realized_pnl at entry */
}
/* Trade completion: full trade return (entry → exit) */
float trade_completion_reward = 0.0f;
/* ════════════════════════════════════════════════════════════════════
* DYNAMIC TRAILING STOP — regime-adaptive, locks in profits.
*
* Only activates when in a trade, profitable, and held > 2 bars.
* Trail distance adapts to regime via ADX (trend) and CUSUM (vol).
* ════════════════════════════════════════════════════════════════════ */
float unrealized_trade_pnl = 0.0f;
if (!was_flat && hold_time > 0.0f) {
unrealized_trade_pnl = (ps[11] + raw_pnl - trade_start_pnl) / (prev_equity > 1.0f ? prev_equity : 1.0f);
}
/* Trail distance adapts to regime */
float trail_adx = 0.0f, trail_cusum = 0.0f;
if (bar_idx < total_bars && market_dim >= 42) {
trail_adx = features[(long long)bar_idx * market_dim + 40];
trail_cusum = features[(long long)bar_idx * market_dim + 41];
}
float vol_scale = 1.0f + (trail_cusum > 0.5f ? trail_cusum : 0.0f);
vol_scale = fminf(vol_scale, 2.5f);
float trend_scale = 1.0f + (trail_adx > 25.0f ? (trail_adx - 25.0f) / 50.0f : 0.0f);
trend_scale = fminf(trend_scale, 2.0f);
float trail_distance = 0.005f * vol_scale * trend_scale; /* 0.5% base, widens in vol/trend */
/* Trailing stop: only when profitable and held > 2 bars */
if (!was_flat && hold_time > 2.0f && peak_equity > 1.0f) {
float peak_return = (peak_equity - prev_equity) / (prev_equity > 1.0f ? prev_equity : 1.0f);
if (peak_return > trail_distance) {
float trail_floor = peak_return - trail_distance;
if (unrealized_trade_pnl < trail_floor) {
/* Trail triggered — force exit */
position = 0.0f;
float exit_cost = fabsf(ps[0]) * raw_close * tx_cost_multiplier * 0.0001f;
cash = ps[1] + ps[0] * raw_close - exit_cost;
is_flat = 1.0f;
exiting_trade = 1;
}
}
}
/* Early-exit override: model tries to exit before 5 bars — force hold.
* CRITICAL: fix the stored action to match what actually executed.
* Without this, the replay buffer stores (state, action=Flat, reward_from_Hold)
* which corrupts Q-values for Flat — classic action aliasing bug. */
if (!was_flat && is_flat > 0.5f && hold_time < 5.0f && hold_time > 0.0f && !exiting_trade) {
position = ps[0];
cash = ps[1];
is_flat = 0.0f;
exiting_trade = 0;
float prev_exposure_frac = ps[0] / (max_position > 0.0f ? max_position : 1.0f);
int held_exposure = (int)((prev_exposure_frac + 1.0f) / 0.25f + 0.5f);
if (held_exposure < 0) held_exposure = 0;
if (held_exposure >= b0_size) held_exposure = b0_size - 1;
int original_order = (action_idx / b2_size) % b1_size;
int original_urgency = action_idx % b2_size;
action_idx = held_exposure * b1_size * b2_size + original_order * b2_size + original_urgency;
out_actions[out_off] = action_idx;
}
/* Hold time tracking: counts TOTAL bars in position (not just losing).
* Reset to 0 when flat. This feeds the patience multiplier in reward v5. */
if (is_flat < 0.5f) {
hold_time += 1.0f;
} else {
hold_time = 0.0f;
}
/* Idle counter (flat bars) — used for state features, no penalty */
if (is_flat > 0.5f) {
flat_counter += 1.0f;
} else {
flat_counter = 0.0f;
}
/* ════════════════════════════════════════════════════════════════════
* REWARD v5: Trade-Aware Hybrid
*
* Two components:
* DENSE (every bar, small weight): per-bar return when in trade.
* Keeps gradients flowing — the network sees direction each bar.
* When flat: reward_dense = 0 (no trade, no signal).
*
* SPARSE (at trade exit, large weight): trade return × patience.
* Primary learning signal. A trade held longer than expected
* gets a patience bonus (sqrt scaling). Regime-adaptive expected
* hold time via ADX.
*
* Combined: reward = 0.1 * dense + 2.0 * sparse
* Loss aversion: negative rewards weighted 1.5x (prospect theory).
* ════════════════════════════════════════════════════════════════════ */
/* ---- Dense component: per-bar return when in a trade ---- */
float reward_dense;
if (fabsf(position) > 0.001f) {
reward_dense = raw_pnl / (prev_equity > 1.0f ? prev_equity : 1.0f);
} else {
reward_dense = 0.0f;
}
/* ---- Sparse component: trade return × patience multiplier ---- */
float reward_sparse = 0.0f;
if (exiting_trade && hold_time > 0.0f) {
float trade_pnl = ps[11] + raw_pnl - trade_start_pnl;
float trade_return = trade_pnl / (prev_equity > 1.0f ? prev_equity : 1.0f);
/* Scale ×200: 1 ES tick = 0.036% × 200 = 0.072 reward.
* 10 ticks (a meaningful move) = 0.72 reward.
* Clamp [-2, +2] so the model sees the MAGNITUDE of returns,
* not just the sign. Old ×1000 clamped to ±1 made all trades equal. */
trade_completion_reward = fmaxf(-2.0f, fminf(2.0f, trade_return * 200.0f));
/* Bonus for holding longer — reward scales with sqrt(hold_time). */
float hold_bonus = sqrtf(hold_time) * 0.02f;
trade_completion_reward += (trade_completion_reward > 0.0f) ? hold_bonus : -hold_bonus;
/* Track full Kelly statistics (GPU-native, no CPU).
* Accumulate: win/loss counts, sum of returns, sum of squared returns.
* These feed the enhanced Kelly: weighted blend of continuous + discrete. */
/* Kelly statistics (GPU-native, no CPU).
* Tracked here AFTER trailing stop may have set exiting_trade=1,
* so trail-triggered exits are counted in Kelly stats. */
if (trade_return > 0.0f) {
win_count += 1.0f;
sum_wins += trade_return;
@@ -810,197 +816,43 @@ extern "C" __global__ void experience_env_step(
loss_count += 1.0f;
sum_losses += fabsf(trade_return);
}
/* Running moments for continuous Kelly (μ/σ²) */
sum_returns += trade_return;
sum_sq_returns += trade_return * trade_return;
}
/* Dynamic trade management: once entered, the trade runs until a stop
* condition is met. The model learns ENTRY quality, not exit timing.
*
* Stop conditions (checked when model tries to exit):
* 1. Stop loss: unrealized P&L < -0.3% of equity → force exit (bad entry)
* 2. Take profit: unrealized P&L > +0.5% of equity → force exit (good entry)
* 3. Time stop: held > 100 bars → force exit (stale position)
* 4. Model exit: model chooses Flat AND held >= 5 bars (learned exit)
*
* If none met and model tries to exit early (<5 bars), override to hold. */
float unrealized_trade_pnl = 0.0f;
if (!was_flat && hold_time > 0.0f) {
unrealized_trade_pnl = (ps[11] + raw_pnl - trade_start_pnl) / (prev_equity > 1.0f ? prev_equity : 1.0f);
}
/* Regime-adaptive stop levels using ADX and CUSUM (already in feature vector).
* High volatility (CUSUM > 0.5) → wider stops (more room for noise).
* Strong trend (ADX > 25) → wider take-profit (let trends run).
* Read from features buffer (same as used for regime scaling earlier). */
float stop_adx = 0.0f, stop_cusum = 0.0f;
if (bar_idx < total_bars && market_dim >= 42) {
long long feat_off = (long long)bar_idx * market_dim;
stop_adx = features[feat_off + 40];
stop_cusum = features[feat_off + 41];
}
/* Vol scale: [1.0, 2.5] — wider stops in volatile markets */
float vol_mult = 1.0f + (stop_cusum > 0.5f ? stop_cusum : 0.0f);
vol_mult = (vol_mult > 2.5f) ? 2.5f : vol_mult;
/* Trend scale: [1.0, 2.0] — wider TP in strong trends */
float trend_mult = 1.0f + (stop_adx > 25.0f ? (stop_adx - 25.0f) / 50.0f : 0.0f);
trend_mult = (trend_mult > 2.0f) ? 2.0f : trend_mult;
/* Conviction-scaled stops: high Q-gap → wider stops (let it run).
* Low Q-gap → tighter stops (uncertain, exit quickly).
* conv_mult = clamp(q_gap, 0.5, 2.0) — range [0.5x, 2x] of base stops.
* This makes the reward distribution CONTINUOUS (not bimodal at ±fixed). */
float q_gap_val = (q_gaps != NULL) ? q_gaps[i] : 1.0f;
float conv_mult = (q_gap_val < 0.5f) ? 0.5f : ((q_gap_val > 2.0f) ? 2.0f : q_gap_val);
/* Stop levels: 1% SL / 2% TP = 2:1 reward-to-risk ratio.
* Old 0.3% SL = 8.3 ticks on ES — normal 1-min noise (4-6 ticks) triggers it.
* New 1% SL = ~28 ticks (7 ES points) — survives normal market noise.
* New 2% TP = ~56 ticks (14 ES points) — captures meaningful moves. */
float sl_level = -0.01f * vol_mult * conv_mult; /* 1% base SL */
float tp_level = 0.02f * vol_mult * trend_mult * conv_mult; /* 2% base TP, 2:1 R:R */
float time_stop_bars = 100.0f * vol_mult * conv_mult;
/* Trailing stop: once trade is profitable, stop TRAILS the peak.
* If peak unrealized = +0.4%, trailing stop = +0.4% - 0.3% = +0.1%.
* This locks in profits: the model can't give back more than sl_level
* from the best point of the trade. Uses peak_equity as the HWM. */
float trail_from_peak = 0.0f;
if (!was_flat && hold_time > 0.0f && peak_equity > 1.0f) {
float peak_return = (peak_equity - (prev_equity > 1.0f ? prev_equity : 1.0f)) / prev_equity;
if (peak_return > 0.001f) {
/* Trade has been profitable — trail the stop */
trail_from_peak = peak_return + sl_level; /* e.g., +0.4% + (-0.3%) = +0.1% floor */
/* Regime-adaptive expected hold time via ADX.
* Trending (ADX > 30): hold longer → expected_hold = 20 bars.
* Ranging (ADX < 20): shorter trades → expected_hold = 8 bars.
* Default: expected_hold = 12 bars. */
float adx_val = 0.0f;
if (bar_idx < total_bars && market_dim >= 42) {
adx_val = features[(long long)bar_idx * market_dim + 40];
}
}
/* Trailing stop: if unrealized drops below the trailing floor, exit */
float effective_sl = (trail_from_peak > sl_level) ? trail_from_peak : sl_level;
float expected_hold = 12.0f;
if (adx_val > 30.0f) expected_hold = 20.0f;
else if (adx_val < 20.0f) expected_hold = 8.0f;
/* Auto-exit triggers (force flat regardless of model's action) */
int auto_stop_loss = (!was_flat && unrealized_trade_pnl < effective_sl) ? 1 : 0;
int auto_take_profit = (!was_flat && unrealized_trade_pnl > tp_level) ? 1 : 0;
int auto_time_stop = (!was_flat && hold_time >= time_stop_bars) ? 1 : 0;
/* Patience multiplier: sqrt(hold_time / expected_hold).
* Held exactly expected: 1.0. Held 2x: 1.41. Held 0.25x: 0.5.
* Capped at 3.0 to prevent extreme outliers. */
float patience_mult = sqrtf(hold_time / expected_hold);
patience_mult = fminf(patience_mult, 3.0f);
if (auto_stop_loss || auto_take_profit || auto_time_stop) {
/* Force exit — override model's action to flat */
position = 0.0f;
float exit_cost = fabsf(ps[0]) * raw_close * tx_cost_multiplier * 0.0001f;
cash = ps[1] + ps[0] * raw_close - exit_cost;
is_flat = 1.0f;
exiting_trade = 1;
} else if (!was_flat && is_flat > 0.5f && hold_time < 5.0f && hold_time > 0.0f) {
/* Model tries to exit too early — override to hold.
* CRITICAL: also fix the stored action to match what actually executed.
* Without this, the replay buffer stores (state, action=Flat, reward_from_Hold)
* which corrupts Q-values for Flat — classic action aliasing bug. */
position = ps[0];
cash = ps[1];
is_flat = 0.0f;
exiting_trade = 0;
/* Overwrite the action output to the PREVIOUS action (hold = keep position).
* The previous action's exposure_idx maps back to a factored action. */
float prev_exposure_frac = ps[0] / (max_position > 0.0f ? max_position : 1.0f);
int held_exposure = (int)((prev_exposure_frac + 1.0f) / 0.25f + 0.5f);
if (held_exposure < 0) held_exposure = 0;
if (held_exposure >= b0_size) held_exposure = b0_size - 1;
/* Reconstruct factored action with held exposure + original order/urgency */
int original_order = (action_idx / b2_size) % b1_size;
int original_urgency = action_idx % b2_size;
action_idx = held_exposure * b1_size * b2_size + original_order * b2_size + original_urgency;
/* Overwrite the ALREADY-WRITTEN action in the output buffer */
out_actions[out_off] = action_idx;
reward_sparse = trade_return * patience_mult;
}
/* Idle counter (flat bars) — NO penalty. DSR already penalizes inaction
* (zero returns reduce Sharpe). Idle penalty was making Flat the worst
* Q-value, forcing the model to trade on every bar. */
if (is_flat > 0.5f) {
flat_counter += 1.0f;
} else {
flat_counter = 0.0f;
/* ---- Combined reward with loss aversion ---- */
float dense_weight = 0.1f; /* Small: keeps gradients flowing */
float sparse_weight = 2.0f; /* Large: trade completion is the primary signal */
float reward = dense_weight * reward_dense + sparse_weight * reward_sparse;
/* Loss aversion: negative rewards weighted 1.5x (prospect theory) */
if (reward < 0.0f) {
reward *= loss_aversion;
}
/* ==== Component 5: Position-time decay (losing positions only) ==== */
float hold_time_penalty = 0.0f;
if (is_flat < 0.5f && raw_pnl < 0.0f) {
hold_time += 1.0f;
hold_time_penalty = hold_time * time_decay_rate;
} else {
hold_time = 0.0f;
}
/* Clamp to [0, 0.3] — prevent dominating the reward */
hold_time_penalty = fminf(0.3f, hold_time_penalty);
/* ==== Profit-taking bonus (scaled to match trading signal) ==== */
/* FIX: scaled from 0.1 → 0.01. Old value was 100x larger than typical
* per-bar PnL signal (~0.001), causing reward hacking: agent entered
* on noise then immediately exited to farm the bonus. */
float profit_take_bonus_v2 = 0.0f;
if (fabsf(delta) > 0.001f && fabsf(position) < fabsf(prev_pos)) {
if (ps[11] > 0.0f) {
profit_take_bonus_v2 = 0.01f;
}
}
/* ==== Composite reward (v3: trade-completion + dense shaping) ==== */
/* The model thinks in TRADES, not in bars.
*
* Three states, three reward levels:
*
* 1. TRADE EXIT (sparse, strong): full trade return entry→exit.
* This is the PRIMARY learning signal. A trade that's underwater
* on bar 5 but profitable on bar 50 gets a POSITIVE reward here.
* 10x stronger than the dense shaping signal.
*
* 2. IN TRADE (dense, weak): 0.1x shaping signal per bar.
* Just enough gradient for the network to learn direction,
* but doesn't overwhelm the trade-completion signal.
* DSR provides risk-adjusted dense signal.
*
* 3. FLAT (near-zero): doing nothing is almost free.
* Small idle nudge prevents permanent inaction.
*
* This teaches the model to hold positions for meaningful periods
* and evaluate trades on their full lifecycle, not per-bar noise.
*/
/* ═══════════════════════════════════════════════════════════════════
* REWARD v4: Pure mark-to-market portfolio return per bar.
*
* reward_t = (equity_t - equity_{t-1}) / equity_{t-1}
*
* This is the simplest correct reward for trading RL:
* - Losing bar → equity drops → NEGATIVE reward (every bar, not just exit)
* - Winning bar → equity rises → POSITIVE reward
* - Flat bar → equity unchanged → ZERO reward
* - Trade entry → tx_cost hits cash → equity drops → NEGATIVE (cost felt immediately)
* - Trade exit → position closed at market → PnL realized → reward reflects outcome
*
* No dense shaping, no DSR, no trade-completion signal, no hold penalty.
* The portfolio return IS the reward. Simple, correct, aligned with the goal.
*
* Loss aversion: losses weighted 1.5x to teach risk management.
* ═══════════════════════════════════════════════════════════════════ */
/* ---- Portfolio value for floor check ---- */
float new_portfolio_value_pre_floor = cash + position * raw_next;
float prev_equity = ps[9]; /* previous bar's portfolio value */
float step_return = (prev_equity > 1.0f)
? (new_portfolio_value_pre_floor - prev_equity) / prev_equity
: 0.0f;
/* Loss aversion: penalize losses more heavily than gains (prospect theory).
* This teaches the agent to avoid drawdowns, not just maximize returns. */
float reward;
if (step_return < 0.0f) {
reward = step_return * loss_aversion; /* loss_aversion ≈ 1.5 */
} else {
reward = step_return;
}
/* Suppress unused variables from old reward components */
(void)w_dsr; (void)w_pnl; (void)w_dd; (void)w_idle;
(void)dsr_t; (void)asym_pnl; (void)drawdown_penalty_t;
(void)hold_time_penalty; (void)trade_completion_reward;
(void)exiting_trade; (void)is_flat;
/* ---- Capital floor protection (PDT $25K rule) ---- */
/* If equity drops to 25% drawdown from peak, terminate the episode.
@@ -1022,14 +874,11 @@ extern "C" __global__ void experience_env_step(
int next_bar = bar_idx + 1;
int done = (next_bar >= total_bars || new_portfolio_value < capital_floor) ? 1 : 0;
/* ---- Update full portfolio state (PORTFOLIO_STRIDE=14) ---- */
/* ---- Update full portfolio state (PORTFOLIO_STRIDE=20) ---- */
ps[0] = position;
ps[1] = cash;
ps[2] = new_portfolio_value;
ps[3] = dsr_A;
ps[4] = dsr_B;
ps[5] = pnl_ema;
ps[6] = pnl_var;
/* ps[3:6] reserved — reward v5 does not use DSR/PnL EMA */
ps[7] = peak_equity;
ps[8] = flat_counter;
ps[9] = new_portfolio_value; /* prev_equity = current equity for next step */
@@ -1045,11 +894,9 @@ extern "C" __global__ void experience_env_step(
ps[19] = sum_sq_returns; /* Kelly continuous: Σ returns² */
/* ---- NO global reward clamp ---- */
/* Reward v2: each component is individually bounded. The composite
* is in ~[-2.5, +1.1] with default weights. The C51 support range
* (v_min, v_max) must be set to cover Q = reward / (1-gamma).
* With gamma=0.95 and reward in [-2.5, +1.1]: Q in [-50, +22].
* v_range should be dynamically computed from gamma. */
/* Reward v5: dense in [-small, +small], sparse in [-moderate, +moderate].
* The combined reward is unbounded in principle but practically small.
* C51 v_range should cover Q = reward / (1-gamma). */
/* ---- Write reward and done flag ---- */
/* Final NaN guard — if reward is NaN/Inf, write 0.0 instead of poisoning

2
crates/ml/src/cuda_pipeline/gpu_experience_collector.rs
View File

@@ -1275,7 +1275,7 @@ impl GpuExperienceCollector {
}
}
// ── 5. Environment step (reward v4: mark-to-market return) ──────
// ── 5. Environment step (reward v5: trade-aware hybrid) ──────
let max_pos = config.max_position;
let tx_cost = config.tx_cost_multiplier;
let rw_loss_av = config.loss_aversion;