fab9c0e324
Setting dones[b]=1 in rl_drawdown_stop without actually closing the lobsim position created a lie: Q trained on false trade-closes while the position stayed open. Result: done_count=0 from step 99 onward, 990 trades total in 2000 steps (should be ~80k), training collapsed. The drawdown penalty (per-step min(0, unrealized_r) × rate) is kept — it creates exit gradient without corrupting the lobsim state. Hard stop-loss needs to override the ACTION (to FlatL/FlatS) BEFORE the lobsim step, not set dones after. Tracked for future implementation. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
50 lines
2.1 KiB
Plaintext
50 lines
2.1 KiB
Plaintext
// rl_drawdown_stop.cu — per-step drawdown penalty + hard stop-loss.
|
||
//
|
||
// Runs AFTER rl_trade_context_update (which computes unrealized_r).
|
||
// Reads unrealized_r from trade_context_d and applies:
|
||
//
|
||
// 1. Per-step drawdown penalty: adds min(0, unrealized_r) × PENALTY_RATE
|
||
// to rewards[b]. Creates continuous exit gradient on losing positions.
|
||
// Penalty-only (never positive) = no exposure incentive.
|
||
//
|
||
// 2. Hard stop-loss: when unrealized_r < -STOP_THRESHOLD, force-close
|
||
// by setting dones[b] = 1. The realized PnL at this point becomes
|
||
// the final trade reward. Caps max loss per trade.
|
||
//
|
||
// Both thresholds are ISV-driven per feedback_isv_for_adaptive_bounds.
|
||
// Grid: ceil(B/32), Block: 32.
|
||
|
||
#define RL_DRAWDOWN_PENALTY_RATE_INDEX 586
|
||
#define RL_STOP_LOSS_THRESHOLD_INDEX 587
|
||
#define TRADE_CONTEXT_STRIDE 5
|
||
#define UNREALIZED_R_OFFSET 1
|
||
|
||
extern "C" __global__ void rl_drawdown_stop(
|
||
const float* __restrict__ trade_context, // [B, TRADE_CONTEXT_STRIDE]
|
||
float* __restrict__ rewards, // [B] IN/OUT
|
||
float* __restrict__ dones, // [B] IN/OUT
|
||
float* __restrict__ isv,
|
||
int b_size
|
||
) {
|
||
const int b = blockIdx.x * blockDim.x + threadIdx.x;
|
||
if (b >= b_size) return;
|
||
|
||
const float unrealized_r = trade_context[b * TRADE_CONTEXT_STRIDE + UNREALIZED_R_OFFSET];
|
||
const float penalty_rate = isv[RL_DRAWDOWN_PENALTY_RATE_INDEX];
|
||
const float stop_thresh = isv[RL_STOP_LOSS_THRESHOLD_INDEX];
|
||
|
||
// Per-step drawdown penalty: only fires when losing (unrealized_r < 0).
|
||
// min(0, x) * rate is always <= 0.
|
||
if (unrealized_r < 0.0f) {
|
||
rewards[b] += unrealized_r * penalty_rate;
|
||
}
|
||
|
||
// Hard stop-loss DISABLED: setting dones=1 here creates a state
|
||
// mismatch — the lobsim position stays open while Q sees a false
|
||
// close. The done signal must come from actual position changes
|
||
// in the lobsim, not synthetic overrides. To implement hard
|
||
// stop-loss properly, the action must be overridden to FlatL/FlatS
|
||
// BEFORE the lobsim step, not after.
|
||
(void) stop_thresh;
|
||
}
|