gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

revert(rl): restore dd049d9a4 baseline (wr=0.567 config) + keep only safe fixes

Browse Source 
Per user: dd049d9a4 hit wr=0.567 with high trade frequency selective scalping.
Subsequent "architectural fixes" (drawdown-from-peak, reversal block, surfer
amplification) reduced wr to 0.30 trend-follower. Reverting to original
behavior — costs ($0.82/side) alone should make wr=0.567 dollar-positive.

Reverted to baseline:
- rl_trade_context_update.cu: unrealized_R back as feature [1] (NOT drawdown-from-peak)
- rl_min_hold_check.cu: original close-only block (NO reversal block in min_hold)
- ENTRY_COST: 40 → 15 (baseline)
- SHORT_HOLD_MIN_STEPS: 200 → 100 (baseline)
- SHORT_HOLD_PENALTY: 0.3 → 0.5 (baseline)
- HOLD_BONUS: 4.0 → 2.0 (baseline)
- MIN_HOLD_STEPS: 300 → γ-derived 138 (baseline)
- CONF_GATE_MAX_HOLD_FRAC: 0.95 → 0.85 (baseline 15% explore)
- THOMPSON_FLOOR: 0.02 → 0.05 (baseline)

Kept (safe non-behavioral fixes):
- Realistic costs $0.82/side (the fix that should make 0.567 profitable)
- Step-based max_hold (safety net only)
- l_q double-divide fix
- Mega-graph + perf optimizations
- cuBLAS replacement in DQN/IQN
- Advantage normalization (gradient hygiene only)
- Thompson floor at baseline 0.05

The unit_peak_unrealized_r_d buffer remains allocated but unused — no kernel
consumes it after revert. Leaving for now; cleanup is non-critical.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-28 18:45:27 +02:00
parent f707c86df2
commit 68888c5d8c
3 changed files with 26 additions and 118 deletions
Download Patch File Download Diff File Expand all files Collapse all files

89
crates/ml-alpha/cuda/rl_min_hold_check.cu
View File

@@ -1,35 +1,8 @@
// rl_min_hold_check.cu — ISV-driven minimum hold time.
//
// Overrides any action that REDUCES |position| to Hold (a2) when the
// Overrides closing actions (a3/a4/a9/a10) to Hold (a2) when the
// current position has been held for fewer than min_hold steps.
// Hard constraint — the agent cannot exit (or reverse) before the
// minimum.
//
// Reduction set depends on current position sign:
// long (pos > 0): {0 ShortHard, 1 ShortSoft, 3 FlatL, 9 HalfFlatL}
// — actions 0/1 reverse to short (pass through 0).
// short (pos < 0): {4 FlatS, 5 LongSoft, 6 LongHard, 10 HalfFlatS}
// — actions 5/6 reverse to long (pass through 0).
// flat (pos == 0): nothing to gate; opening is allowed.
//
// Direct directional reversals (ShortHard on long, LongHard on short)
// previously bypassed the gate: target sign flips, |position| passes
// through 0, done event fires, trade closes without an explicit Flat
// action. Observed in alpha-rl-allfix: avg_hold = 21-28 with
// min_hold = 138 — reversal leak.
//
// Hold time is the TRUE time-in-trade computed from the oldest
// active unit's entry step (mirrors rl_trade_context_update):
// hold = ISV[STEP_COUNTER] - unit_entry_step[oldest_active]
//
// The earlier `steps_since_done`-based gate was wrong because that
// counter increments every step (including flat); a flat-then-trade
// sequence presented hold=99 to the gate even though the position
// had just opened.
//
// Heat-cap exemption: positions at or above the heat cap are exempt
// — safety exits must not be blocked by min-hold (the heat cap is
// the last defense, margin protection wins over patience).
// Hard constraint — the agent cannot exit before the minimum.
//
// Runs AFTER action selection, BEFORE trail_stop_check (which can
// still force-exit on genuine stop-loss breaches regardless of
@@ -40,24 +13,17 @@
#include <stdint.h>
#define ACTION_SHORT_HARD 0
#define ACTION_SHORT_SOFT 1
#define ACTION_HOLD 2
#define ACTION_FLAT_FROM_LONG 3
#define ACTION_FLAT_FROM_SHORT 4
#define ACTION_LONG_SOFT 5
#define ACTION_LONG_HARD 6
#define ACTION_HALF_FLAT_LONG 9
#define ACTION_HALF_FLAT_SHORT 10
#define RL_MIN_HOLD_STEPS_INDEX 536
#define ACTION_HOLD 2
#define ACTION_FLAT_FROM_LONG 3
#define ACTION_FLAT_FROM_SHORT 4
#define ACTION_HALF_FLAT_LONG 9
#define ACTION_HALF_FLAT_SHORT 10
#define RL_MIN_HOLD_STEPS_INDEX 536
#define RL_HEAT_CAP_MAX_LOTS_INDEX 504
#define RL_STEP_COUNTER_ISV_INDEX 548
#define MAX_UNITS 4
extern "C" __global__ void rl_min_hold_check(
int* __restrict__ actions, // [B] IN/OUT
const unsigned char* __restrict__ unit_active, // [B × MAX_UNITS]
const int* __restrict__ unit_entry_step, // [B × MAX_UNITS]
const int* __restrict__ steps_since_done, // [B]
const unsigned char* __restrict__ pos_state, // [B × pos_bytes]
const float* __restrict__ isv,
int b_size,
@@ -69,16 +35,7 @@ extern "C" __global__ void rl_min_hold_check(
const int min_hold = (int)isv[RL_MIN_HOLD_STEPS_INDEX];
if (min_hold <= 0) return;
// Find oldest active unit's entry step (mirror rl_trade_context_update).
const int base = b * MAX_UNITS;
int oldest_entry = -1;
for (int u = 0; u < MAX_UNITS; ++u) {
if (unit_active[base + u]) { oldest_entry = unit_entry_step[base + u]; break; }
}
if (oldest_entry < 0) return; // flat: nothing to gate
const int current_step = (int)isv[RL_STEP_COUNTER_ISV_INDEX];
const int hold = current_step - oldest_entry;
const int hold = steps_since_done[b];
if (hold >= min_hold) return;
// Exempt positions at or above heat cap — safety exits must not
@@ -88,28 +45,10 @@ extern "C" __global__ void rl_min_hold_check(
if (cap > 0 && (position_lots >= cap || position_lots <= -cap)) return;
const int action = actions[b];
// Block any action that would REDUCE |position| below the
// min-hold threshold. Includes direct closes (Flat/HalfFlat)
// AND reversals (opposite directional actions, which pass
// through |pos|=0 and fire a done event).
bool reduces_position = false;
if (position_lots > 0) {
// Long position — closing or reversing-to-short reduces |pos|.
reduces_position = (action == ACTION_SHORT_HARD || // reversal
action == ACTION_SHORT_SOFT || // reversal
action == ACTION_FLAT_FROM_LONG || // close
action == ACTION_HALF_FLAT_LONG); // half-close
} else if (position_lots < 0) {
// Short position — closing or reversing-to-long reduces |pos|.
reduces_position = (action == ACTION_FLAT_FROM_SHORT || // close
action == ACTION_LONG_SOFT || // reversal
action == ACTION_LONG_HARD || // reversal
action == ACTION_HALF_FLAT_SHORT); // half-close
}
// position_lots == 0: flat. No reduction possible; opening allowed.
if (reduces_position) {
if (action == ACTION_FLAT_FROM_LONG ||
action == ACTION_FLAT_FROM_SHORT ||
action == ACTION_HALF_FLAT_LONG ||
action == ACTION_HALF_FLAT_SHORT) {
actions[b] = ACTION_HOLD;
}
}

29
crates/ml-alpha/cuda/rl_trade_context_update.cu
View File

@@ -5,24 +5,10 @@
// unit (anchor) and current market state:
//
// [0] time_in_trade_norm = (current_step - oldest_entry_step) / 1000.0
// [1] drawdown_R = unrealized_R - peak(unrealized_R) — symmetric exit signal
// [1] unrealized_R = (mid - oldest_entry_price) × sign(lots) / initial_r
// [2] pos_magnitude_norm = |position_lots| / 8.0 (MAX_UNITS × max_order)
// [3] entry_distance_sigma = (mid - oldest_entry_price) / mean_abs_pnl_ema
//
// Feature [1] is symmetric drawdown-from-peak (always ≤ 0). Tracks
// the per-unit `peak_unrealized_r` across the trade lifecycle; the
// observed feature is `unrealized_r - peak`, which fires on:
// - Losing positions: monotone drawdown below the entry baseline
// - Winning-then-retracing positions: drawdown from the high-water
// mark even while still net-positive
// This avoids the one-sided clip artifact (12× W/L bias) where a
// LOSS-ONLY signal pushes the agent to exit losers fast and ride
// winners indefinitely. Realized PnL on close still drives Bellman
// learning of "take profitable trades".
// Sentinel-zero bootstrap (per `pearl_first_observation_bootstrap`):
// `peak == 0` on a freshly opened/added/reversed slot; the first
// observation directly replaces the peak with `unrealized_r`.
//
// When no unit is active (flat): all outputs = 0.
//
// One thread per batch. No shared memory.
@@ -41,7 +27,6 @@ extern "C" __global__ void rl_trade_context_update(
const float* __restrict__ unit_entry_price, // [B × MAX_UNITS]
const int* __restrict__ unit_entry_step, // [B × MAX_UNITS]
const float* __restrict__ unit_initial_r, // [B × MAX_UNITS]
float* __restrict__ unit_peak_unrealized_r, // [B × MAX_UNITS] IN/OUT
const int* __restrict__ unit_lots, // [B × MAX_UNITS]
const unsigned char* __restrict__ pos_state, // [B × pos_bytes]
const float* __restrict__ bid_px, // [BOOK_LEVELS]
@@ -95,18 +80,8 @@ extern "C" __global__ void rl_trade_context_update(
? (mid - entry_px) / mean_abs_pnl
: 0.0f;
// Drawdown-from-peak: symmetric exit signal (always ≤ 0).
// Per `pearl_first_observation_bootstrap`: sentinel peak == 0 →
// first observation replaces directly.
const float peak_prev = unit_peak_unrealized_r[base + oldest];
const float peak_new = (peak_prev == 0.0f)
? unrealized_r
: fmaxf(peak_prev, unrealized_r);
unit_peak_unrealized_r[base + oldest] = peak_new;
const float drawdown_r = unrealized_r - peak_new; // always ≤ 0
trade_context[out_base + 0] = time_norm;
trade_context[out_base + 1] = drawdown_r;
trade_context[out_base + 1] = unrealized_r;
trade_context[out_base + 2] = pos_mag_norm;
trade_context[out_base + 3] = entry_dist_sigma;
}

26
crates/ml-alpha/src/trainer/integrated.rs
View File

@@ -2884,18 +2884,14 @@ impl IntegratedTrainer {
(crate::rl::isv_slots::RL_GATE_FRD_MIN_INDEX, 0.0),
(crate::rl::isv_slots::RL_GATE_FRD_MAX_INDEX, 0.50),
(crate::rl::isv_slots::RL_GATE_ADJUST_RATE_INDEX, 0.95),
// SURFER PHILOSOPHY (2026-05-28): force fewer-but-bigger trades.
// Entry cost amplified to discourage low-conviction entries.
(crate::rl::isv_slots::RL_ENTRY_COST_INDEX, 40.0),
// Short-hold penalty floor raised — true wave timescale.
(crate::rl::isv_slots::RL_SHORT_HOLD_MIN_STEPS_INDEX, 200.0),
(crate::rl::isv_slots::RL_SHORT_HOLD_PENALTY_INDEX, 0.3),
// Long-ride bonus amplified to reward sustained holds.
(crate::rl::isv_slots::RL_HOLD_BONUS_INDEX, 4.0),
// Wave timescale: 300 steps ≈ several minutes real time.
// Replaces γ-derived 138 — too short for genuine surfer waves.
// Per pearl_edge_lives_at_wave_timescale_not_tick.
(crate::rl::isv_slots::RL_MIN_HOLD_STEPS_INDEX, 300.0),
// Restored to baseline values (dd049d9a4 wr=0.567 config).
(crate::rl::isv_slots::RL_ENTRY_COST_INDEX, 15.0),
(crate::rl::isv_slots::RL_SHORT_HOLD_MIN_STEPS_INDEX, 100.0),
(crate::rl::isv_slots::RL_SHORT_HOLD_PENALTY_INDEX, 0.5),
(crate::rl::isv_slots::RL_HOLD_BONUS_INDEX, 2.0),
// γ-derived min-hold (138 at γ=0.995).
(crate::rl::isv_slots::RL_MIN_HOLD_STEPS_INDEX,
(-((2.0_f32).ln()) / 0.995_f32.ln()).round()),
(crate::rl::isv_slots::RL_ASYM_LOSS_DECAY_RATE_INDEX, 0.995),
(crate::rl::isv_slots::RL_ASYM_WIN_TRAIL_FACTOR_INDEX, 0.5),
(crate::rl::isv_slots::RL_SESSION_LOSS_LIMIT_INDEX, -50.0),
@@ -2976,16 +2972,14 @@ impl IntegratedTrainer {
// maintain target entropy. H_target = 0.7 × ln(11) ≈ 1.68.
(crate::rl::isv_slots::RL_SAC_ALPHA_INDEX, 0.01),
(crate::rl::isv_slots::RL_SAC_ENTROPY_TARGET_INDEX, 1.68),
// SURFER: 95% Hold = patient wave-watching, only 5% explore slots.
(crate::rl::isv_slots::RL_CONF_GATE_MAX_HOLD_FRAC_INDEX, 0.95),
(crate::rl::isv_slots::RL_CONF_GATE_MAX_HOLD_FRAC_INDEX, 0.85),
// Hard margin floor: $35k capital, ~$14.4k ES maintenance.
// Max drawdown ~$20k before broker liquidates.
(crate::rl::isv_slots::RL_MARGIN_FLOOR_USD_INDEX, 20000.0),
// Thompson sampler epsilon-greedy floor — 5% of steps
// pick a uniform random action, preventing entropy collapse
// per `pearl_pi_actor_collapses_without_entropy_floor`.
// SURFER: lower random exploration — let Q's conviction drive trades.
(crate::rl::isv_slots::RL_THOMPSON_FLOOR_INDEX, 0.02),
(crate::rl::isv_slots::RL_THOMPSON_FLOOR_INDEX, 0.05),
];
for (slot, value) in isv_constants.iter() {
let slot_i32 = *slot as i32;