d3175711b9
Action enum extended:
a9 = HalfFlatLong (close ⌈|pos|/2⌉ of long position, no-op if not long)
a10 = HalfFlatShort (close ⌈|pos|/2⌉ of short position, no-op if not short)
`actions_to_market_targets.cu` extended with a9/a10 handlers:
HalfFlatLong (pos > 0): side=1 sell, size=max(1, (position_lots+1)/2)
HalfFlatShort (pos < 0): side=0 buy, size=max(1, (|position_lots|+1)/2)
Round-up division ensures min 1 lot closes — single-lot positions
fully close on HalfFlat (the half rounds up to 1).
N_ACTIONS=9 → 11 propagated to all 10 .cu kernels:
argmax_expected_q, bellman_target_projection, dqn_distributional_q,
log_pi_at_action, ppo_clipped_surrogate, rl_action_kernel,
rl_entropy_coef_controller, rl_pi_action_kernel, rl_q_pi_agree_b,
rl_q_pi_distill_grad
Rust-side N_ACTIONS const bumped to 11 in src/rl/common.rs.
CLI alpha_rl_train.rs action_hist + windowed_act_hist refactored
to reference `N_ACTIONS` const instead of literal 11. Caught DURING
this commit's dogfood: an intermediate state had `[0u32; 11]` but
left `(0..9).contains(&a)` unchanged — HalfFlat samples silently
dropped (a9/a10 showed 0% in diag despite P_MIN=0.02 floor
guaranteeing 2% each). Fix uses N_ACTIONS const everywhere; new
pearl `feedback_use_consts_not_literals_for_structural_dims`
codifies the meta-pattern (Rust code mirroring kernel structural
dims must reference the const, NEVER duplicate the literal —
audit-isv only scans .cu files, this class of bug is currently
unaudited in .rs).
Audits PASS:
audit-isv: all kernel #defines allowlisted (BOOK_LEVELS,
ACTION_*, structural dims)
audit-wiring: all 4 actions in manifest (TrailTighten, TrailLoosen,
HalfFlatLong, HalfFlatShort) have consumers
Local 1k-step smoke (RTX 3050 Ti, 13.5s):
* Exit 0, 0 NaN/inf, 16000/16000 samples accounted for
* Action distribution: all 11 used in 7-11% range
* HalfFL=7.99%, HalfFS=7.51% — π samples them under multinomial
* Trail=19.34% — agent continues to value trail-stop actions
Trail-stop check (rl_trail_stop_check.cu) currently still routes
force-close through a3/a4 (FlatFromLong/Short) rather than per-unit
partial-flat via a9/a10. That routing upgrade is SP20 P5b follow-up
work — it requires the per-unit close_unit_index buffer wiring per
spec §3 P5. Adding a9/a10 to the action space is the foundational
prerequisite; consumer kernel uses come with P5b.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
56 lines
2.1 KiB
Plaintext
56 lines
2.1 KiB
Plaintext
// log_pi_at_action.cu — GPU-resident per-batch log π(action | s)
|
||
// computation for the PPO importance-ratio path (Phase R4 of the
|
||
// integrated RL trainer rebuild;
|
||
// see docs/superpowers/plans/2026-05-23-integrated-rl-trainer-rebuild.md).
|
||
//
|
||
// Replaces the host-side `pi_logits[action] − log_sum_exp(pi_logits)`
|
||
// loop the flawed Phase F shipped in step_with_lobsim — which
|
||
// violated `feedback_cpu_is_read_only` by DtoH-copying π logits and
|
||
// computing log-softmax on CPU.
|
||
//
|
||
// Per-batch: log_pi[b] = pi_logits[b, actions[b]] − log Σ_a exp(pi_logits[b, a]).
|
||
// Numerically stabilised via max-shift before exp.
|
||
//
|
||
// One thread per batch entry; element-wise. No reduction across
|
||
// batches, no atomics. N_ACTIONS = 9 fits in a per-thread sequential
|
||
// loop comfortably.
|
||
|
||
#define N_ACTIONS 11
|
||
|
||
// Inputs:
|
||
// pi_logits [b_size, N_ACTIONS] row-major
|
||
// actions [b_size] index into [0, N_ACTIONS)
|
||
// Outputs:
|
||
// log_pi_out [b_size]
|
||
extern "C" __global__ void log_pi_at_action(
|
||
const float* __restrict__ pi_logits,
|
||
const int* __restrict__ actions,
|
||
float* __restrict__ log_pi_out,
|
||
int b_size
|
||
) {
|
||
const int b = blockIdx.x * blockDim.x + threadIdx.x;
|
||
if (b >= b_size) return;
|
||
|
||
const int row_off = b * N_ACTIONS;
|
||
float max_l = -INFINITY;
|
||
#pragma unroll
|
||
for (int i = 0; i < N_ACTIONS; ++i) {
|
||
const float l = pi_logits[row_off + i];
|
||
if (l > max_l) max_l = l;
|
||
}
|
||
float sum_exp = 0.0f;
|
||
#pragma unroll
|
||
for (int i = 0; i < N_ACTIONS; ++i) {
|
||
sum_exp += expf(pi_logits[row_off + i] - max_l);
|
||
}
|
||
// log-sum-exp identity: log Σ exp(x) = max + log Σ exp(x − max)
|
||
const float log_norm = max_l + logf(sum_exp);
|
||
|
||
const int chosen = actions[b];
|
||
// No bounds check on `chosen` per `feedback_no_quickfixes`: the
|
||
// upstream Thompson kernel (rl_action_kernel) writes into
|
||
// [0, N_ACTIONS) by construction. A bounds check here would mask
|
||
// an upstream contract violation; let it crash visibly.
|
||
log_pi_out[b] = pi_logits[row_off + chosen] - log_norm;
|
||
}
|