gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

fix(rl): reward_scale floor uses cumulative dones, not closed-trade-steps

Browse Source 
Follow-up to the 2026-05-30 adaptive controller floor refactor (commit
083a88f7c). Spec Special case R wired the bootstrap-fraction floor to
release after `RL_TRADE_DUR_VAR_COUNT_INDEX < MIN_TRADES_FOR_RELEASE
(=100)`. That counter increments via the Welford trade-duration-EMA
producer ONCE PER STEP where any trade closed — not once per closed
trade. At b=1024 with typical ~7 dones/step, the gate released after
step ~100 (= ~700 actual closed trades, far short of the intended
"100 trades of confidence"). After release reward_scale crashed to
~0.001 within 200 more steps, identical to the pre-fix fold 0/1
behavior the spec was supposed to prevent.

Fix: track REAL cumulative closed trades by summing `dones[b]` per
step in the fused kernel's reward_scale block. New ISV slots:
  - RL_CUMULATIVE_DONES_INDEX (660) — running total
  - RL_MIN_TRADES_FOR_RELEASE_INDEX (661) — gate threshold,
    ISV-driven per `feedback_adaptive_not_tuned`, bootstrap 5000

At ~7 dones/step (typical) the floor now holds for ~715 steps before
release — enough for the reward magnitude EMA to stabilize against
genuine trade outcomes rather than early-training noise. Threshold is
ISV-resident so production runs can tune without recompile.

Verified locally (500-step b=128 smoke):
  step 50:  reward_scale = 0.384 (decaying toward floor)
  step 100: reward_scale = 0.140 (approaching floor)
  step 200: reward_scale = 0.100 (AT FLOOR, held)
  step 500: reward_scale = 0.100 (still at floor)

Without this fix (pre-commit): reward_scale crashed to 0.001 by step 500.

Tests: integrated_trainer_smoke + signal_variance_kernel +
controller_adaptive_floors all pass. Release build clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-30 16:38:17 +02:00
parent 083a88f7c3
commit b1ef6664ab
3 changed files with 70 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files

41
crates/ml-alpha/cuda/rl_fused_controllers.cu
View File

@@ -51,6 +51,12 @@
#define RL_GAMMA_MAX_INDEX 649
#define RL_GAMMA_MIN_ADAPTIVE_INDEX 613
#define RL_TRADE_DUR_VAR_COUNT_INDEX 608
// Reward-floor fix (2026-05-30 follow-up): track REAL cumulative closed
// trades via dones-sum accumulation. The Welford trade-duration counter
// increments per closed-trade STEP, not per closed trade — at b=1024 with
// ~7 dones/step this released the floor after only ~700 actual trades.
#define RL_CUMULATIVE_DONES_INDEX 660
#define RL_MIN_TRADES_FOR_RELEASE_INDEX 661
#define RL_TRADE_DUR_VAR_MEAN_INDEX 609
#define RL_MEAN_TRADE_DURATION_EMA_INDEX 417
#define GAMMA_MIN_ABSOLUTE 0.9f // architectural: don't degenerate to bandit
@@ -519,6 +525,26 @@ extern "C" __global__ void rl_fused_controllers(
per_alpha_done:;
}
// ═══════════════════════════════════════════════════════════════════
// 7a. Cumulative dones accumulator → ISV[660] (reward-floor-fix follow-up)
//
// Sums `dones[b]` across this step's batch and accumulates into the
// cumulative-closed-trades slot. Used by the reward_scale bootstrap-
// floor gate below. Replaces the prior `RL_TRADE_DUR_VAR_COUNT_INDEX`
// gate which counted closed-trade STEPS (not closed trades) and
// released the floor ~100× too early in real-trade terms.
//
// Per `feedback_no_atomicadd`: single-thread sum-and-write — no atomic
// needed since the fused kernel is single-thread / single-block.
// ═══════════════════════════════════════════════════════════════════
{
float dones_this_step = 0.0f;
for (int b = 0; b < b_size; b++) {
dones_this_step += dones[b];
}
isv[RL_CUMULATIVE_DONES_INDEX] = isv[RL_CUMULATIVE_DONES_INDEX] + dones_this_step;
}
// ═══════════════════════════════════════════════════════════════════
// 7. Reward scale controller → ISV[406] (Special R)
// Asymmetric per-step decrease cap (5%) + bootstrap-fraction floor
@@ -540,13 +566,16 @@ extern "C" __global__ void rl_fused_controllers(
const float scale_min = isv[RL_REWARD_SCALE_MIN_INDEX];
target = fmaxf(scale_min, fminf(target, REWARD_SCALE_MAX));
// Bootstrap-fraction floor: until N=100 trades have closed,
// scale cannot drop below 10% of bootstrap. The trade count
// comes from the Welford trade-duration counter (incremented
// once per closed trade).
const float trade_count = isv[RL_TRADE_DUR_VAR_COUNT_INDEX];
// Bootstrap-fraction floor: until cumulative closed trades
// exceeds the (ISV-driven) threshold, scale cannot drop below
// 10% of bootstrap. Cumulative dones tracked per-step by the
// dones-sum accumulator block ABOVE; threshold from ISV slot 661.
// Per `feedback_adaptive_not_tuned`: threshold is ISV-driven,
// not hardcoded.
const float trade_count = isv[RL_CUMULATIVE_DONES_INDEX];
const float min_trades = isv[RL_MIN_TRADES_FOR_RELEASE_INDEX];
const float boot = isv[RL_REWARD_SCALE_BOOTSTRAP_INDEX];
const float boot_floor = (trade_count < MIN_TRADES_FOR_RELEASE)
const float boot_floor = (trade_count < min_trades)
? boot * BOOTSTRAP_FRACTION_FLOOR
: scale_min;

29
crates/ml-alpha/src/rl/isv_slots.rs
View File

@@ -1334,6 +1334,31 @@ pub const RL_REWARD_CLAMP_CLIP_RATE_EMA_ALPHA_INDEX: usize = 658; // replaces CL
// Single shared ISV slot — all controllers read from this slot for consistency.
pub const RL_WIENER_ALPHA_FLOOR_INDEX: usize = 659; // replaces WIENER_ALPHA_FLOOR = 0.4f
// ─── reward_scale floor fix (2026-05-30, follow-up to spec Special R) ───
//
// The original Special case R used `RL_TRADE_DUR_VAR_COUNT_INDEX` (Welford
// count of trade_duration_ema) as the gate for releasing the 10%-of-bootstrap
// floor. That counter actually increments once per STEP where ANY trade
// closes — not once per closed trade. With b=1024 and ~7 dones/step, the
// gate released after step ~100 (only ~700 actual closed trades) instead
// of after the intended "100 closed trades worth of confidence". After
// release, reward_scale crashed 0.13 → 0.001 in cluster validation.
//
// Fix: track cumulative closed trades via dones-sum accumulation. Gate the
// bootstrap floor on observed cumulative trade count, with the threshold
// itself ISV-driven (per `feedback_adaptive_not_tuned`).
/// Cumulative count of closed trades since training start. Incremented by
/// `sum(dones[b])` each step in `rl_fused_controllers`. Used by the
/// reward_scale controller's bootstrap-fraction floor gate.
pub const RL_CUMULATIVE_DONES_INDEX: usize = 660;
/// Trade-count threshold for releasing reward_scale's bootstrap-fraction
/// floor. Bootstrap 5000 = ~715 steps at typical b=1024 dones rate; enough
/// for the reward magnitude EMA to stabilize before the normalizer is
/// allowed to operate freely.
pub const RL_MIN_TRADES_FOR_RELEASE_INDEX: usize = 661;
/// Last RL-allocated slot index (exclusive). Pre-spec: 588. Post-noise-floor: 640.
/// Post-clamp-bounds extension: 660. Total new: 72 slots.
pub const RL_SLOTS_END: usize = 660;
/// Post-clamp-bounds extension: 660. Post-reward-floor-fix: 662. Total new: 74 slots.
pub const RL_SLOTS_END: usize = 662;

9
crates/ml-alpha/src/trainer/integrated.rs
View File

@@ -2907,7 +2907,7 @@ impl IntegratedTrainer {
// (slot, value) pair — pure device write, no HtoD per
// `feedback_no_htod_htoh_only_mapped_pinned`.
{
let isv_constants: [(usize, f32); 145] = [
let isv_constants: [(usize, f32); 147] = [
// Static seeds for the adaptive reward-clamp controller —
// these are the initial values that
// `rl_reward_clamp_controller` will replace once it
@@ -3051,6 +3051,13 @@ impl IntegratedTrainer {
// jump in λ at controller bootstrap.
(crate::rl::isv_slots::RL_GAMMA_MIN_ADAPTIVE_INDEX, 0.95),
(crate::rl::isv_slots::RL_Q_DISTILL_LAMBDA_MIN_ADAPTIVE_INDEX, 0.01),
// Reward-floor fix (2026-05-30 follow-up): track real cumulative
// closed trades (sum of dones per step) instead of closed-trade-STEPS.
// Threshold = 5000 trades → ~715 steps of floor protection at typical
// b=1024 dones rate. Both slots ISV-driven per
// `feedback_adaptive_not_tuned`.
(crate::rl::isv_slots::RL_CUMULATIVE_DONES_INDEX, 0.0),
(crate::rl::isv_slots::RL_MIN_TRADES_FOR_RELEASE_INDEX, 5000.0),
// IQN ensemble seeds — alpha=0.5 (equal C51/IQN blend),
// N_TAU=32, LR=1e-3 (canonical).
(crate::rl::isv_slots::RL_IQN_N_TAU_INDEX, 32.0),