jgrusewski e5ced809aa feat(rl): split KL into static reward + dynamic advantage ...

Static: β×log(π_ref(a)) added to reward, stored in PER. Q sees
consistent Hold preference across replays.

Dynamic: -β×log(π_θ(a)) added to advantage only (not stored in PER).
Provides entropy-regularized signal (SAC-like) that adapts to
current policy. Low π_θ → positive advantage (explore). High π_θ →
near-zero (don't over-concentrate).

Result: entropy STABLE 0.94-1.29 through 5000 steps (no collapse).
Hold 62-100%. wr=0.36 (best local). Removed KL gradient kernel
from training step (KL is now entirely in the reward/advantage).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

2026-05-27 11:46:13 +02:00

..

abs_copy.cu

…

actions_to_market_targets.cu

feat(rl): per-batch outcome EMA, vol-adjusted trail, ISV-driven P_MIN

2026-05-24 21:59:34 +02:00

adamw_step.cu

perf(rl): fuse adamw_increment into adamw_step + tau into iqn_forward

2026-05-26 16:46:10 +02:00

apply_reward_scale.cu

…

argmax_expected_q.cu

…

attention_pool.cu

…

aux_heads.cu

…

aux_loss.cu

…

aux_trunk.cu

…

aux_vec_add.cu

…

bce_loss_multi_horizon.cu

…

bellman_target_projection.cu

perf(rl): cuBLAS SGEMM for DQN/IQN + Bellman/outcome kernel fusion

2026-05-26 17:17:56 +02:00

bucket_transition_kernels.cu

…

cfc_step_per_branch.cu

…

cfc_step.cu

…

compute_advantage_return.cu

feat(rl): split KL into static reward + dynamic advantage

2026-05-27 11:46:13 +02:00

dqn_distributional_q.cu

feat(rl): /B normalize loss reporting — batch-invariant JSONL metrics

2026-05-26 23:41:03 +02:00

dqn_target_soft_update.cu

…

ema_update_on_done.cu

…

ema_update_per_step.cu

…

extract_realized_pnl_delta.cu

…

gpu_log_helpers.cuh

…

gpu_log_ids.h

…

gpu_log_ring.cu

…

gpu_sample_and_gather.cu

fix(rl): wire BCE + aux losses into GPU loader path + event-based sync

2026-05-26 22:18:30 +02:00

grad_h_accumulate.cu

…

grad_norm.cu

…

heads_block_diagonal_fwd.cu

…

horizon_lambda.cu

…

layer_norm.cu

…

log_pi_at_action.cu

…

mamba2_alpha_kernel.cu

…

multi_horizon_heads.cu

…

output_smoothness.cu

…

ppo_clipped_surrogate.cu

wip(rl): entropy gradient + controller fix — still collapsing

2026-05-27 01:05:42 +02:00

ppo_log_ratio_abs_max_b.cu

…

projection.cu

…

reduce_axis0.cu

…

rl_action_kernel.cu

…

rl_adversarial_boost.cu

feat(cuda): curriculum weights E8 + adversarial regime boost kernels

2026-05-26 12:00:21 +02:00

rl_asymmetric_trail_decay.cu

fix(rl): asymmetric trail win threshold 100% → 25% of initial_r

2026-05-25 20:59:44 +02:00

rl_atom_support_update.cu

…

rl_confidence_gate.cu

feat(rl): KL reference policy — proven RLHF pattern for Hold preservation

2026-05-27 01:58:01 +02:00

rl_curriculum_weights.cu

feat(cuda): curriculum weights E8 + adversarial regime boost kernels

2026-05-26 12:00:21 +02:00

rl_encoder_context_broadcast.cu

feat(rl): encoder input expansion 40 → 56 dims (trade-context + multires)

2026-05-24 22:35:26 +02:00

rl_ensemble_action_value.cu

feat(rl): IQN loss + backward + ensemble action selection

2026-05-25 21:03:04 +02:00

rl_entropy_coef_controller.cu

wip(rl): entropy gradient + controller fix — still collapsing

2026-05-27 01:05:42 +02:00

rl_frd_fwd.cu

…

rl_frd_gate.cu

perf(rl): device-resident step counter + fused controllers (-9 launches)

2026-05-25 21:35:50 +02:00

rl_frd_layer1_bwd.cu

…

rl_frd_layer2_bwd.cu

…

rl_frd_softmax_ce_grad.cu

…

rl_fused_controllers.cu

wip(rl): entropy gradient + controller fix — still collapsing

2026-05-27 01:05:42 +02:00

rl_fused_reward_pipeline.cu

fix(rl): trail distance units mismatch — was 400× too small

2026-05-26 00:24:48 +02:00

rl_gamma_controller.cu

feat(rl): tier 2 wave-scale — γ=0.995, PER 32k, min-hold 100

2026-05-25 15:26:58 +02:00

rl_gate_threshold_controller.cu

perf(rl): device-resident step counter + fused controllers (-9 launches)

2026-05-25 21:35:50 +02:00

rl_hindsight_forward.cu

feat(cuda): bidirectional HER kernels — track, inject, forward

2026-05-26 08:41:05 +02:00

rl_hindsight_inject.cu

feat(cuda): bidirectional HER kernels — track, inject, forward

2026-05-26 08:41:05 +02:00

rl_hindsight_track.cu

feat(cuda): bidirectional HER kernels — track, inject, forward

2026-05-26 08:41:05 +02:00

rl_increment_step.cu

perf(rl): device-resident step counter + fused controllers (-9 launches)

2026-05-25 21:35:50 +02:00

rl_iqn_backward.cu

perf(rl): cuBLAS SGEMM for DQN/IQN + Bellman/outcome kernel fusion

2026-05-26 17:17:56 +02:00

rl_iqn_forward.cu

perf(rl): cuBLAS SGEMM for DQN/IQN + Bellman/outcome kernel fusion

2026-05-26 17:17:56 +02:00

rl_iqn_loss.cu

feat(rl): IQN complementary Q-head — quantile embedding + Huber loss

2026-05-25 20:35:06 +02:00

rl_isv_write.cu

…

rl_kl_approx_b.cu

…

rl_kl_reference_grad.cu

fix: rename RL_KL_TARGET_INDEX → RL_KL_REF_TARGET_INDEX (avoid dupe)

2026-05-27 09:59:55 +02:00

rl_kurtosis_streaming.cu

…

rl_l2_diff_norm.cu

…

rl_l2_norm.cu

…

rl_lr_controller.cu

…

rl_min_hold_check.cu

fix(rl): min-hold exempts positions at heat cap level

2026-05-25 13:40:53 +02:00

rl_multires_features_update.cu

refactor(rl): pre-allocate 56 replay-step gradient buffers for Graph C

2026-05-25 23:00:40 +02:00

rl_noisy_linear_backward.cu

feat(rl): Phase 3 noisy linear layers — state-dependent exploration

2026-05-25 21:09:41 +02:00

rl_noisy_linear_forward.cu

feat(rl): Phase 3 noisy linear layers — state-dependent exploration

2026-05-25 21:09:41 +02:00

rl_outcome_bwd.cu

feat(rl): outcome head backward + diag JSONL enrichment

2026-05-26 11:47:19 +02:00

rl_outcome_ce.cu

feat(rl): P1+P2 kernels — PopArt, spectral norm, outcome head, Q-bias, per-branch LR

2026-05-26 09:36:21 +02:00

rl_outcome_fused.cu

perf(rl): cuBLAS SGEMM for DQN/IQN + Bellman/outcome kernel fusion

2026-05-26 17:17:56 +02:00

rl_outcome_fwd.cu

feat(rl): P1+P2 kernels — PopArt, spectral norm, outcome head, Q-bias, per-branch LR

2026-05-26 09:36:21 +02:00

rl_outcome_label.cu

feat(rl): P1+P2 kernels — PopArt, spectral norm, outcome head, Q-bias, per-branch LR

2026-05-26 09:36:21 +02:00

rl_per_alpha_controller.cu

…

rl_per_branch_lr.cu

feat(rl): wire P1+P2 — PopArt, spectral norm/decouple, outcome head, Q-bias, per-branch LR

2026-05-26 09:49:57 +02:00

rl_per_push_flush.cu

feat(cuda): coalesced PER push — ring + flush kernels (Grid=B, Block=128)

2026-05-26 01:41:15 +02:00

rl_per_push_ring.cu

feat(cuda): coalesced PER push — ring + flush kernels (Grid=B, Block=128)

2026-05-26 01:41:15 +02:00

rl_per_sample.cu

feat(cuda): GPU-resident PER kernels — push, sample, update, tree_rebuild

2026-05-26 00:25:31 +02:00

rl_per_tree_rebuild.cu

feat(cuda): GPU-resident PER kernels — push, sample, update, tree_rebuild

2026-05-26 00:25:31 +02:00

rl_per_update_priority.cu

feat(cuda): GPU-resident PER kernels — push, sample, update, tree_rebuild

2026-05-26 00:25:31 +02:00

rl_pi_action_kernel.cu

feat(rl): per-batch outcome EMA, vol-adjusted trail, ISV-driven P_MIN

2026-05-24 21:59:34 +02:00

rl_popart_normalize.cu

feat(rl): P1+P2 kernels — PopArt, spectral norm, outcome head, Q-bias, per-branch LR

2026-05-26 09:36:21 +02:00

rl_popart_v_correct.cu

feat(rl): P1+P2 kernels — PopArt, spectral norm, outcome head, Q-bias, per-branch LR

2026-05-26 09:36:21 +02:00

rl_position_heat_check.cu

fix(rl): heat cap uses >= not > (pos=8 at cap=8 must trigger)

2026-05-25 11:00:56 +02:00

rl_ppo_clip_controller.cu

…

rl_ppo_ratio_clamp_controller.cu

…

rl_q_bias_correction.cu

feat(rl): P1+P2 kernels — PopArt, spectral norm, outcome head, Q-bias, per-branch LR

2026-05-26 09:36:21 +02:00

rl_q_distill_lambda_controller.cu

fix(rl): asymmetric Q→π distillation λ controller — ramp fast, decay slow

2026-05-25 11:54:17 +02:00

rl_q_pi_agree_b.cu

fix(rl): q_pi_agree metric → cosine similarity instead of argmax match

2026-05-25 12:04:26 +02:00

rl_q_pi_distill_grad.cu

feat(rl): KL penalty as REWARD not gradient — fixes structural imbalance

2026-05-27 11:38:09 +02:00

rl_recent_outcome_update.cu

feat(rl): per-batch outcome EMA, vol-adjusted trail, ISV-driven P_MIN

2026-05-24 21:59:34 +02:00

rl_reward_clamp_controller.cu

…

rl_reward_scale_controller.cu

fix(rl): per-step ±2% clamp on reward_scale movement

2026-05-25 13:50:25 +02:00

rl_reward_shaping.cu

feat(rl): longer horizon γ=0.99, min-hold 50, ride bonus for winners

2026-05-25 15:00:21 +02:00

rl_rollout_steps_controller.cu

…

rl_sample_noise.cu

test(rl): GPU oracle tests for IQN + NoisyNet + PRNG advance fix

2026-05-25 22:37:40 +02:00

rl_sample_tau.cu

test(rl): GPU oracle tests for IQN + NoisyNet + PRNG advance fix

2026-05-25 22:37:40 +02:00

rl_session_risk_check.cu

feat(rl): asymmetric trail + session risk — structural P&L edge

2026-05-25 19:53:54 +02:00

rl_spectral_decouple.cu

feat(rl): P1+P2 kernels — PopArt, spectral norm, outcome head, Q-bias, per-branch LR

2026-05-26 09:36:21 +02:00

rl_spectral_norm.cu

feat(rl): P1+P2 kernels — PopArt, spectral norm, outcome head, Q-bias, per-branch LR

2026-05-26 09:36:21 +02:00

rl_step_counter_update.cu

…

rl_streaming_clamp_init.cu

…

rl_target_tau_controller.cu

feat(rl): revert to V-advantage + normalization, ISV-driven TAU_MAX

2026-05-27 00:10:57 +02:00

rl_trade_context_update.cu

perf(rl): device-resident step counter + fused controllers (-9 launches)

2026-05-25 21:35:50 +02:00

rl_trail_mutate.cu

…

rl_trail_stop_check.cu

…

rl_unit_state_update.cu

fix(rl): trail distance units mismatch — was 400× too small

2026-05-26 00:24:48 +02:00

rl_var_over_abs_mean_streaming.cu

…

rl_write_u64.cu

refactor(rl): pre-allocate 56 replay-step gradient buffers for Graph C

2026-05-25 23:00:40 +02:00

smoothness_lambda_controller.cu

…

snap_feature_assemble.cu

feat(rl): encoder input expansion 40 → 56 dims (trade-context + multires)

2026-05-24 22:35:26 +02:00

snapshot_aos_to_soa.cu

perf(rl): GPU AoS→SoA scatter replaces host-side snapshot staging

2026-05-26 19:18:26 +02:00

v_head_fwd_bwd.cu

…

variable_selection.cu

…