Files
foxhunt/crates/ml-alpha/build.rs
jgrusewski 9ea7692abd feat(ml-alpha): Phase 7b F5 — state-conditional action availability mask
Adds the no-op-set / direction / trail-at-cap action availability mask
specified in §3.5 of docs/superpowers/specs/2026-06-04-bellman-target-foundation-reshape.md.

## What lands

* New kernel `cuda/rl_state_action_mask.cu` (~115 LOC) modelled on
  `rl_band_mask.cu`'s lattice — Grid=(B), Block=(1,1,1), single thread
  per block, ISV-gated, reads `pos_state[b*pos_bytes+0..4]` for
  `position_lots` and the existing per-batch per-unit trail buffers
  (`unit_trail_distance`, `unit_initial_r`, `unit_active`).
* New ISV slot `RL_F5_STATE_MASK_ENABLED_INDEX = 823` (binary master
  gate; bootstrap 0.0 = OFF; preserves Phase 7a `43e7b6383`
  bit-equality). `RL_SLOTS_END` bumped 823 → 824.
* Trainer integration: `launch_rl_state_action_mask` public wrapper +
  inline call sites at the two policy-sampling paths
  (`step_with_lobsim` and `step_with_lobsim_gpu_body`), inserted BEFORE
  `rl_pi_action_kernel` so the sampler sees the masked logits. Both
  sites sit inside the prefill graph capture (device-side master gate
  preserves bit-equality across off↔on flips). `isv_constants` array
  size bumped 274 → 275.
* `build.rs` registers the new cubin.

## Mask semantics (when slot 823 > 0.5)

* `position_lots == 0` (flat) → mask {Hold=2, FlatFromLong=3,
  FlatFromShort=4, TrailTighten=7, TrailLoosen=8, HalfFlatLong=9,
  HalfFlatShort=10}. Surviving support: {ShortLarge=0, ShortSmall=1,
  LongSmall=5, LongLarge=6} — agent is FORCED to open.
* `position_lots > 0` (long) → mask all short-side actions
  {ShortLarge=0, ShortSmall=1, FlatFromShort=4, HalfFlatShort=10}.
  Same-side opens (5/6) remain available; pyramid resolution stays in
  `actions_to_market_targets.cu`.
* `position_lots < 0` (short) → symmetric.
* Any active unit at trail-cap (`trail_distance ≥ unit_initial_r *
  RL_TRAIL_MAX_INITIAL_R_RATIO * (1 − 1e-3)`) → additionally mask
  TrailLoosen (mq2pc-specific gate per spec §3.5 trail-at-cap branch).

## Composition with F2 (Q-distill hinged advantage)

F2 computes `π_target` from unmasked E_Q; F5 sets `pi_logits[masked] =
−INFINITY` so `softmax(pi_logits)` has EXACTLY zero mass on masked
actions (F5-G1 design requirement). The distill gradient
`(π_θ − π_target)` evaluates to `(0 − π_target_masked)` at masked
actions, which naturally drives the target off those actions —
gradient consistency without re-masking inside `rl_q_pi_distill_grad`.

## Action enum (verified against actions_to_market_targets.cu and
   crates/ml-alpha/src/rl/common.rs)

| id | name           | masked from flat | masked from long | masked from short |
|----|----------------|------------------|------------------|-------------------|
|  0 | ShortLarge     |                  | ✓                |                   |
|  1 | ShortSmall     |                  | ✓                |                   |
|  2 | Hold           | ✓                |                  |                   |
|  3 | FlatFromLong   | ✓                |                  | ✓                 |
|  4 | FlatFromShort  | ✓                | ✓                |                   |
|  5 | LongSmall      |                  |                  | ✓                 |
|  6 | LongLarge      |                  |                  | ✓                 |
|  7 | TrailTighten   | ✓                |                  |                   |
|  8 | TrailLoosen    | ✓                | (cap-only)       | (cap-only)        |
|  9 | HalfFlatLong   | ✓                |                  | ✓                 |
| 10 | HalfFlatShort  | ✓                | ✓                |                   |

(There are NO separate PyramidLong/PyramidShort actions in foxhunt;
pyramid logic resolves inside `actions_to_market_targets.cu` when
same-side opens fire from an existing position.)

## Surfer-principle trade-off (acknowledged per spec §3.5 + §4.1.4)

F5 destroys patience-while-waiting-for-setup by construction. F2
preserves the surfer principle mathematically (Open mass = 0 when
E_Q(Open) ≤ baseline); F5 sacrifices it for choice-set enforcement.
Bootstrap 0.0 keeps F5 OFF until F2 alone fails the Tier 1.5 / Tier 2
verdict AND the operator explicitly accepts the trend-follower
regression. Reversible at runtime via ISV re-seed.

## Verification

* `cargo build --release --example alpha_rl_train -p ml-alpha` clean.
* Determinism (band off, F5 off / band on, F5 off / F5 ON via
  temporary bootstrap=1.0): all three PASS — `determinism-check.sh
  --quick` reports byte-equal `eval_summary.json` and
  `alpha_rl_train_summary.json` plus checksum-equal diag rows.
* Invariants (band_invariants 11/11, eval_diag_emission 1/1,
  multi_head_policy_invariants 18/18, phase_5_invariants 3/3): 33/33
  PASS.

## STOP-on-surprise — F5 mechanism vs downstream gate-stack interaction

50-step smoke with F5 ON (bootstrap=1.0, then reverted) surfaced an
EXPECTED interaction documented in spec §3.5 expected-failure-mode #5:

* F5 mask itself is mechanically correct — `pi_logits[masked] = −INF`,
  softmax mass is zero, `rl_pi_action_kernel` samples only from
  {0,1,5,6} from flat.
* BUT the downstream gates that run AFTER `rl_pi_action_kernel` —
  `rl_confidence_gate` (lines 67-74 read `pos_state`, override opens
  from flat to Hold when C51 confidence is low), `rl_frd_gate`,
  `rl_session_risk_check`, `rl_min_hold_check` — re-introduce Hold
  into the executed action histogram. F5-G1 (`hold_frac_flat == 0`)
  is therefore NOT achieved by this commit alone.

The F5 kernel does what the spec asks; the gate-stack interaction is
the orthogonal wiring the spec called out as out-of-scope for F5
correctness. A follow-up plan should either suppress those gates'
Hold override when `isv[823] > 0.5 AND position_lots == 0`, OR
re-launch the F5 mask AFTER the gate stack (single extra device-side
kernel invocation). The mask kernel and ISV slot land here so the
follow-up is purely a wiring task. Per the STOP-on-unexpected-finding
discipline (feedback_investigation_first_falsification_methodology),
no further fix is applied in this dispatch.

## Files

* `crates/ml-alpha/cuda/rl_state_action_mask.cu` (new, ~115 LOC)
* `crates/ml-alpha/build.rs` (kernel registration)
* `crates/ml-alpha/src/rl/isv_slots.rs` (slot 823 + RL_SLOTS_END bump)
* `crates/ml-alpha/src/trainer/integrated.rs` (cubin include, struct
  fields, ctor load + struct init, `launch_rl_state_action_mask`
  wrapper, ISV bootstrap entry + array size bump 274→275, two inline
  call sites at `step_with_lobsim` and `step_with_lobsim_gpu_body`)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-05 01:00:01 +02:00

304 lines
30 KiB
Rust
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
//! Pre-compile all ml-alpha CUDA kernels into arch-specific cubins.
//!
//! Per `feedback_no_nvrtc.md`: no runtime kernel compilation.
//! Per `pearl_build_rs_rerun_if_env_changed.md`: every `std::env::var`
//! is paired with `cargo:rerun-if-env-changed`.
use std::path::{Path, PathBuf};
use std::process::Command;
const KERNELS: &[&str] = &[
"mamba2_alpha_kernel", // Mamba2 SSM scan kernel (used by PerceptionTrainer's encoder prefix)
"snap_feature_assemble",
"cfc_step",
"multi_horizon_heads",
"projection",
"bce_loss_multi_horizon", // Kendall σ-weighted multi-horizon BCE (axis A)
"adamw_step",
"grad_norm",
"horizon_lambda", // ISV-driven per-horizon gradient scaler (EMA + lambda)
"layer_norm", // Phase 1: trunk pre-CfC normalisation
"variable_selection", // Phase 2D: TFT-style per-feature gating
"attention_pool", // Phase 3: single-Q learned content summary at CfC k=0
"reduce_axis0", // Phase B: cross-batch param-grad reducer
"output_smoothness", // CRT.train: per-horizon adjacent-position prob-jitter penalty
"smoothness_lambda_controller", // CRT.train: ISV-driven λ controller anchored on h30 jitter
"gpu_log_ring", // GPU diagnostic log ring — tick kernel + log_record helper
"bucket_transition_kernels", // Per-horizon CfC Phase 1→2 transition: tau_sort, bucket_assign, bucket_iqr, channels_in_bucket, heads_compact, zero_off_bucket (ALPHA fix 2026-05-21)
"cfc_step_per_branch", // Per-horizon CfC Phase 2: fused per-(batch, branch) fwd + bwd over [25,25,25,25,28] buckets
"heads_block_diagonal_fwd", // Per-horizon CfC Phase 2: heads w_skip projection with compact ragged storage (640→128 floats)
"aux_trunk", // SDD-3 Layer B3: smaller single-bucket CfC trunk (AUX_HIDDEN=64) for outcome-supervision (D-labels)
"aux_heads", // SDD-3 Layer B4: per-direction linear regression heads on AuxTrunk output (long + short, N_AUX_HORIZONS each)
"aux_loss", // SDD-3 Layer B4: Huber loss + grad for aux trade-outcome regression targets (NaN-masked)
"aux_vec_add", // SDD-3 Layer B5: element-wise dst += src for aux→encoder gradient accumulation (lifted stop-grad)
"dqn_distributional_q", // RL Phase C: C51 distributional Q-head fwd + Bellman TD bwd for integrated RL trainer
"rl_gamma_controller", // RL Phase C: ISV controller emitting γ to ISV[RL_GAMMA_INDEX=400]
"rl_target_tau_controller", // RL Phase C: ISV controller emitting τ to ISV[RL_TARGET_TAU_INDEX=401]
"ppo_clipped_surrogate", // RL Phase D: PPO clipped-surrogate + entropy bonus + value MSE fwd/bwd
"ppo_loss_reduce_b", // F4.1 (2026-05-31): [B] → [1] block tree-reduce mean of per-batch PPO loss + entropy loss (replaces atomicAdd; eliminates l_pi step-count staleness bug — see ppo_clipped_surrogate.cu header)
"rl_ppo_clip_controller", // RL Phase D: ISV controller emitting ε to ISV[RL_PPO_CLIP_INDEX=402]
"rl_entropy_coef_controller", // RL Phase D: ISV controller emitting entropy bonus weight to ISV[RL_ENTROPY_COEF_INDEX=403]
"v_head_fwd_bwd", // RL Phase E.2: scalar V(s) head fwd + MSE bwd (linear layer; per-batch scratch + reduce_axis0)
"grad_h_accumulate", // RL Phase E.2: element-wise grad_h_encoder += λ × grad_h_head accumulator (one head at a time, serialised by stream)
"bellman_target_projection", // RL Phase E.2-DEFER: C51 categorical projection of Bellman target Z(s_{t+1}, a*) onto the discrete support, reads γ from ISV[400]; replaces host-side build_synthetic_bellman_target stand-in
"rl_bellman_target_saturation_reduce", // B-9 (2026-06-01): cross-batch tree-reduce of per-batch saturation tallies from bellman_target_projection / _fused; emits ISV slots 726-729 (top/bot rate + max/min pre-proj)
"rl_q_distribution_stats", // B-10 (2026-06-01) G1: Q-distribution informativeness; two entry points `rl_q_distribution_per_batch` (per-block per-action softmax + entropy + E_Q over online Q logits) + `rl_q_distribution_reduce` (cross-batch tree-reduce → ISV slots 730/731/732)
"rl_ppo_diagnostic_stats_reduce",// B-10 (2026-06-01) G3+G4: cross-batch tree-reduce of per-batch advantage + ratio + surrogate scratches written by ppo_clipped_surrogate.cu; reads var_pre_norm slot 612 for σ_used; emits ISV slots 735-742
"rl_lr_controller", // RL Phase E.2-DEFER: per-head learning-rate ISV emitter — bootstraps ISV[412..417] with 1e-3 (BCE/Q/π/V/aux); replaces hardcoded PHASE_E2_DEFAULT_LR
"rl_rollout_steps_controller", // RL Phase E.3b: rollout-buffer-length ISV emitter — emits ISV[RL_N_ROLLOUT_STEPS_INDEX=404] from var(advantage)/|mean A| EMA; bootstraps 2048
"rl_per_alpha_controller", // RL Phase E.3b: PER priority-exponent ISV emitter — emits ISV[RL_PER_ALPHA_INDEX=405] from TD-error kurtosis EMA; bootstraps 0.6
"rl_reward_scale_controller", // RL Phase R1 (rebuild): reward-standardisation scale ISV emitter — emits ISV[RL_REWARD_SCALE_INDEX=406] from mean |realized_pnl_usd| EMA; bootstraps 1.0
"ema_update_on_done", // RL Phase R3: generic done-gated EMA producer (slot-parameterised) for closed-trade-magnitude EMAs (mean_abs_pnl, q_divergence, td_kurtosis)
"ema_update_per_step", // RL Phase R3: generic per-step EMA producer (slot-parameterised) for continuous EMAs (kl_pi, entropy_observed, advantage_var_ratio, trade_duration)
"compute_advantage_return", // RL Phase R3: element-wise A_t = r + γ(1-done)·V(s_{t+1}) V(s_t), R_t = r + γ(1-done)·V(s_{t+1}); reads γ from ISV[400]
"gae_backward_sweep", // Phase 1B-A (2026-06-02): GAE backward sweep over [B × T] rollout trajectories — A_t = δ_t + γλ·A_{t+1}·non_terminal, returns_t = A_t + V_t; deterministic single-thread-per-batch sequential sweep; replaces compute_advantage_return when wired in Phase 1B-B+
"rollout_pack", // Phase 1B-B (2026-06-02): per-step f32→u8 dones packer; writes `dones_f32 [B]` into rollout buffer's `dones_u8_bt [B × T]` at offset `b * T + t_cursor`; single-thread-per-batch, deterministic
"rl_action_kernel", // RL Phase R4: Thompson sampler over C51 atoms; one block per batch, N_ACTIONS threads; per-batch xorshift32 PRNG state; replaces host Thompson loop per feedback_cpu_is_read_only
"argmax_expected_q", // RL Phase R4: argmax over expected Q per action; Bellman-target argmax (Double-DQN); pairs with rl_action_kernel per pearl_thompson_for_distributional_action_selection
"log_pi_at_action", // RL Phase R4: per-batch log π(action_b) via log-softmax + lookup; PPO importance-ratio path
"dqn_target_soft_update", // RL Phase R5: element-wise target[i] = (1-τ)·target + τ·current, reads τ from ISV[401]; closes defect #4 (no target-net soft update in flawed branch)
"extract_realized_pnl_delta", // RL Phase R6: GPU-pure reward + done extraction from device Pos array; replaces host read_pos loop per feedback_cpu_is_read_only
"apply_reward_scale", // RL Phase R6: element-wise rewards *= ISV[RL_REWARD_SCALE_INDEX=406]; closes the F.3b host roundtrip
"actions_to_market_targets", // RL Phase R6: 9-action grid → LobSim market_targets[B*2] on device; replaces host submit_market loop per feedback_cpu_is_read_only
"abs_copy", // RL Phase R7a: element-wise dst[b] = fabsf(src[b]); feeds |reward| into ema_update_on_done for the MEAN_ABS_PNL_EMA slot
"rl_var_over_abs_mean_streaming",// EMA-streaming var/|mean| (folds across STEPS, fixes b_size=1 → ISV[421] feeding rl_rollout_steps
"rl_kurtosis_streaming", // EMA-streaming kurtosis M4/M2² (folds across STEPS, fixes b_size=1) → ISV[422] feeding rl_per_alpha
"rl_kl_approx_b", // Schulman-style KL = mean(log π_old log π_new) → kl_pi_ema (ISV[419]) feeding rl_ppo_clip
"rl_l2_diff_norm", // ‖W_online W_target‖₂ → q_divergence_ema (ISV[418]) feeding rl_target_tau
"rl_step_counter_update", // per-batch trade-duration counter + done-gated emit → mean_trade_duration_ema (ISV[417]) feeding rl_gamma
"rl_l2_norm", // ‖x‖₂ single-buffer reduction → q/pi/v grad-norm EMAs (ISV[424..427]) feeding rl_lr_controller
// (entropy_observed_ema, ISV[420], feeds rl_entropy_coef directly via ema_update_per_step's internal mean reduce on entropy_d — no separate kernel needed.)
"rl_ppo_ratio_clamp_controller", // RL R9: PPO ratio clamp ceiling at ISV[440], anchored on ε at ISV[402] — bounds catastrophic unclipped-branch surrogate
"ppo_log_ratio_abs_max_b", // RL R9 diag: per-batch max|log π_new log π_old| → ISV[441]; surfaces ratio-clamp activity in diag JSONL
"rl_streaming_clamp_init", // RL R9: device-side seeder for streaming-kernel output clamp ceilings (ISV[447], ISV[448]) — no HtoD
"rl_isv_write", // RL R9: generic single-slot ISV seeder for tunable design constants — no HtoD
"rl_q_pi_agree_b", // RL R9 audit: per-batch fraction (argmax Q == argmax π) → ISV[407] EMA; wires previously-dead slot
"rl_pi_action_kernel", // audit Option B: π drives action selection via multinomial sampling from softmax(pi_logits); Q becomes pure critic
"rl_reward_clamp_controller", // audit 2026-05-24: adaptive [-LOSS, +WIN] clamp from positive-tail EMA; replaces static [-3, +1] that crushed winning-trade signal in rmgm5
"rl_atom_support_update", // audit 2026-05-24 followup: refreshes atom_supports_d from ISV V_MIN/V_MAX so C51 atom span adapts with reward clamp (Q learning was capped at V_MAX=1.0)
"rl_kl_reference_grad",
"rl_q_pi_distill_grad", // audit 2026-05-24 vj5f6 followup: KL(softmax(E_Q/τ) || π_new) gradient ADDED to pi_grad_logits — couples Q's improved C51 calibration to action selection (was decoupled per Option B)
"rl_pi_grad_blend", // Phase 3D-C (2026-06-03): PPO surrogate × Q-distill blend operator (replaces zero-fill before Q-distill +=); restores direct PG signal to π
"action_entropy_per_step", // POST-gate action entropy EMA for SAC α/τ co-tuning
"rl_q_distill_lambda_controller",// audit 2026-05-24 rljzl followup: adaptive λ_distill via Schulman bounded step on KL_EMA vs target
"rl_unit_state_update", // SP20 P1+P5 audit fix: per-unit trade state machine — detects open/close/reverse transitions, sets up unit slot 0 entry+trail
"rl_trail_mutate", // SP20 P1+P5 audit fix (a7/a8 dead): TrailTighten/TrailLoosen mutate unit_trail_distance bounded MIN/MAX, symmetric reciprocal adjust
"rl_trail_stop_check", // SP20 P1+P5 audit fix: per-unit trail breach check; OVERRIDE action to FlatFromLong/Short on breach (close routes through existing flat plumbing)
"rl_frd_fwd", // SP20 P3: Forward-Return-Distribution head fwd — 2-layer MLP [HIDDEN_DIM → FRD_HIDDEN_DIM → FRD_N_HORIZONS × FRD_N_ATOMS]; ReLU hidden cached for bwd; softmax + CE happen in bwd
"rl_frd_softmax_ce_grad", // SP20 P3 F.3a: per-(batch, horizon) softmax + CE loss + dL/dlogits; 1 block per (b, h), 21 threads; label = -1 sentinel masks the row
"rl_frd_layer2_bwd", // SP20 P3 F.3b: FRD head layer-2 backward — dW2 (per-batch scratch), db2 (per-batch scratch), dhidden (per-batch overwrite); 1 block per batch, 64 threads
"rl_frd_layer1_bwd", // SP20 P3 F.3c: FRD head layer-1 backward — dW1 (per-batch scratch), db1 (per-batch scratch), dh_t (per-batch overwrite); applies ReLU mask via cached post-ReLU hidden; 1 block per batch, 128 threads
"rl_position_heat_check", // SP20 P6: position heat cap — force-flat when |position_lots| exceeds ISV-driven max; last defense before actions_to_market_targets
"rl_confidence_gate", // P8: override opening actions to Hold when C51 distributional Q is insufficiently confident (LCB < threshold)
"rl_frd_gate", // P9: override opening actions to Hold when FRD head predicts insufficient favorable probability mass
"rl_recent_outcome_update", // P10: per-batch signed outcome EMA for anti-martingale sizing
"rl_trade_context_update", // P1: per-batch trade-arc features (time_in_trade, unrealized_R, pos_mag, entry_dist)
"rl_multires_features_update", // P0: per-batch multi-resolution streaming features (3 horizons × 4 features)
"rl_encoder_context_broadcast", // P2: broadcast per-batch context (16 dims) into encoder input [B,K,56] at cols 40-55
"rl_gate_threshold_controller", // adaptive gate thresholds from trade frequency (dones EMA)
"rl_reward_shaping", // surfer-philosophy: entry cost + short-hold penalty + hold bonus
"rl_min_hold_check", // hard minimum hold time — override close actions to Hold before N steps
"rl_asymmetric_trail_decay", // auto-tighten losers, auto-widen winners — structural P&L asymmetry
"rl_session_risk_check", // session-level loss limit circuit breaker
"rl_iqn_forward", // IQN distributional Q-head: quantile embedding + action-value projection; complementary to C51
"rl_iqn_loss", // IQN quantile Huber loss: ρ_τ(δ) = |τ - 1(δ<0)| × Huber(δ, κ=1.0); forward + backward
"rl_iqn_backward", // IQN backward through forward pass: grad_output → grad_w_out/b_out/w_embed/b_embed per-batch scratch
"rl_dueling_q_forward", // Phase 4 (2026-05-30): Independent dueling Q head forward — V[B] + A[B,N] + composed_Q[B,N] = V + A mean_a A; parallel to C51/IQN, zero shared state per spec 2026-05-30-phase4-independent-dueling-head-design.md
"rl_dueling_q_bellman_target", // Phase 4: argmax over target composed_Q + Bellman target = r + γ^n × (1-done) × max_Q
"rl_dueling_q_loss_and_grad", // Phase 4: Huber loss on (target online_composed_Q[taken]) + grad_composed
"rl_dueling_q_decompose_and_bwd", // Phase 4: decompose grad_composed → grad_V + grad_A via mean-subtraction Jacobian + per-batch weight gradients
"rl_v_blend", // Phase 4.4 (2026-05-30): elementwise V_used = α V_scalar + (1α) V_dq, α from ISV
"rl_v_blend_alpha_controller", // Phase 4.4: ISV-adaptive Schulman-bounded controller on α from observed |V_dq V_scalar| / |V_scalar| tracking ratio
"rl_advantage_normalize", // Phase 4.5 (2026-05-30): per-batch advantage normalization (A mean)/std with ε² variance floor — standard PPO practice, self-adaptive (no tuned params)
"rl_signal_variance_update", // Adaptive controller floors (2026-05-30): Welford online variance for controller input EMAs; replaces hardcoded NOISE_FLOOR_FRAC constants in 8 controllers — sample_var = M² / (count-1); see spec 2026-05-30-adaptive-controller-floor-design
"rl_win_rate_ema_update", // Adaptive risk management (2026-05-30): Layer 4 (Kelly) input — observed win_rate EMA from per-batch trade outcomes
"rl_avg_win_loss_ema_update", // Adaptive risk management (2026-05-30): Layer 4 (Kelly) input — avg_win/avg_loss USD EMAs from per-batch closed-trade pnl
"rl_inventory_variance_update", // Adaptive risk management (2026-05-30): Layer 3 (inventory β) input — |net_position| variance EMA across batches
"rl_cmdp_constraints_check", // Adaptive risk management (2026-05-30): Layer 1 hard gates — session DD + cooldown + consecutive-loss tracking; writes override flags to ISV
"rl_iqn_action_tau_controller", // Adaptive risk management (2026-05-30): Layer 2 — adapts IQN action-selection quantile τ from session drawdown signal
"rl_inventory_beta_controller", // Adaptive risk management (2026-05-30): Layer 3 — adapts inventory penalty β from observed inventory variance vs reward magnitude
"rl_kelly_fraction_controller", // Adaptive risk management (2026-05-30): Layer 4 — half-Kelly position-size multiplier from observed win_rate × R-multiple; warmup-gated
"rl_surfer_scaffold_controller", // Reward-policy realignment v5 (2026-06-01): adaptive Phase 5 shaping weight ∈ [0,1] — fades surfer-bias scaffold as agent crosses break-even, re-engages on edge-decay PH alerts
"rl_eval_warmup_decay", // v9 (2026-05-31): defensive eval-boundary calibration — overrides Kelly/τ_min/entropy_min/ε_min during warmup, linearly decays back to normal; runs every step
"rl_regime_flat_count", // F1.2 (2026-05-31): block-reduce per-account lots[b] → flat_count single int; prereq for regime_observer (F1.3)
"rl_regime_observer", // F1.3 (2026-05-31): unified regime state-machine — dead-zone, Welford PnL variance, tail-event, recovery_factor/eps_live
"rl_ensemble_action_value", // C51+IQN ensemble: E_ensemble = α×E_C51 + (1-α)×E_IQN; α from ISV[544]
"rl_noisy_linear_forward", // NoisyNet: factored noisy linear forward — y = (mu_w + sigma_w ⊙ eps_w) × x + (mu_b + sigma_b ⊙ eps_b); state-dependent exploration for C51/IQN final projection
"rl_noisy_linear_backward", // NoisyNet: factored noisy linear backward — grad_mu_w/sigma_w/mu_b/sigma_b per-batch scratch for reduce_axis0
"rl_sample_tau", // CUDA graph prereq: device-side xorshift32 tau ~ U(0,1) for IQN; replaces host ChaCha8 + mapped-pinned upload
"rl_sample_noise", // CUDA graph prereq: device-side factored noise f(rand) for NoisyLinear; replaces host ChaCha8 + mapped-pinned upload
"rl_write_u64", // CUDA graph prereq: single-thread u64 scalar write for device-resident ts_ns (graph-captured kernels read via pointer)
"rl_fused_reward_pipeline", // P3: 7→1 fused per-batch reward extraction + shaping + EMA + outcome update
"rl_per_push_ring", // GPU PER: coalesced n-step ring write + flush decision (Grid=B, Block=128)
"rl_per_push_flush", // GPU PER: coordinated coalesced replay write with prefix-sum slot allocation (Grid=B, Block=128)
"rl_per_sample", // GPU PER: stratified proportional sampling via sum-tree walk + gather
"rl_per_update_priority", // GPU PER: write |TD|^α to tree leaves + block-wide max reduction
"rl_per_tree_rebuild", // GPU PER: bottom-up parallel sum-tree rebuild (no atomics)
"rl_hindsight_track", // HER Phase 1: per-step mid-price ring + peak tracking for backward hindsight
"rl_hindsight_inject", // HER Phase 2: backward inject — synthetic replay push on done if peak >> actual
"rl_hindsight_forward", // HER Phase 3: forward continuation — evaluates closed trades after lookahead
"rl_increment_step", // device-resident step counter bump (ISV[548] += 1.0); graph-safe prereq — removes scalar current_step from all downstream kernel args
"rl_fused_controllers", // fused kernel: 10 RL ISV controllers in one launch (gamma, tau, ppo_clip, entropy_coef, rollout_steps, per_alpha, reward_scale, ppo_ratio_clamp, gate_threshold, q_distill_lambda) — saves 9 kernel launch overheads (~40-80μs/step)
"rl_popart_normalize", // PopArt: Welford-EMA reward normalization (replaces apply_reward_scale)
"rl_popart_v_correct", // PopArt: V-head output correction after stats shift
"rl_spectral_norm", // Spectral norm: power iteration σ_max estimate + W rescale
"rl_spectral_decouple", // Spectral decoupling: L2 penalty on logit magnitudes
"rl_q_bias_correction", // Q-bias: EMA of (Q_pred - actual_return) → Bellman correction
"rl_per_branch_lr", // Per-branch LR: adaptive per-head learning rate scaling
"rl_outcome_fwd", // Outcome aux: linear forward h_t → 3-class logits
"rl_outcome_ce", // Outcome aux: softmax CE loss + gradient (masked by sentinel -1)
"rl_outcome_label", // Outcome aux: assign labels from reward/done (Profit/Timeout/Loss)
"rl_outcome_bwd", // Outcome aux: backward through linear layer → grad_W/b/h_t
"rl_outcome_fused", // Outcome aux: fused fwd + CE + bwd — eliminates 2 global round-trips (logits, grad_logits kept in smem)
"rl_curriculum_weights", // E8: per-segment difficulty-weighted softmax from Sharpe → PER weights
"rl_adversarial_boost", // Adversarial: boost PER priority for negative-reward transitions
"rl_outcome_bwd", // Outcome aux: single linear layer backward — dW (per-batch), db (per-batch), dh_t (per-batch); 1 block per batch, 128 threads
"snapshot_aos_to_soa", // AoS→SoA scatter: one thread per snapshot reads contiguous Mbp10RawInput, writes into 10 SoA device buffers; replaces host nested loops + 10 DtoD copies
"gpu_sample_and_gather", // GPU-resident batch sampler: random file+anchor sampling + AoS→SoA gather from pre-uploaded dataset; eliminates ALL per-step CPU data loading
"rl_deterministic_checksum", // Determinism foundation Phase 1 (2026-06-02): provably deterministic sum-of-squares (single-block / single-thread / f64 accumulation) for per-step component checksums in diag; localizes non-determinism source. Spec: docs/superpowers/specs/2026-06-02-determinism-foundation.md §1.1
"multi_head_policy_forward", // Phase 2A-A (2026-06-03): K-head policy logit + softmax + mixture combination; per-batch deterministic sequential mixture sum. Spec: docs/superpowers/specs/2026-06-02-multi-head-policy-with-r-multiple.md §R.6
"multi_head_policy_gate_forward",// Phase 2A-A (2026-06-03): gating head from raw regime features [B × 6] (parallel channel bypassing VSN/Mamba2). Spec ADDENDUM 2026-06-03 §R.3 Option B
"multi_head_policy_backward", // Phase 2A-B (2026-06-03): backward through mixture + per-head softmax + gating softmax. Two kernels: `_backward_pi` and `_backward_gate`. Per-batch grad scratch; caller reduces via reduce_axis0. Spec ADDENDUM 2026-06-03 §R.6
"multi_head_policy_aux_prior", // Phase 2A-B (2026-06-03): per-head auxiliary KL prior gradient — additive into grad_pi_logits_k. β read from ISV slot 764. Spec ADDENDUM 2026-06-03 §R.4
"multi_head_policy_aggregate_diag", // Phase 2A-D (2026-06-03): device-aggregated gate-and-head diagnostics. Reduces forward outputs (gate_probs, pi_probs_k) over B → 25 ISV slots (gate_probs_mean[K] / gate_argmax_mass[K] / gate_entropy_mean / per_head_entropy_mean[K]) for the multi-head specialization verdict signal. Tree-reduce, no atomicAdd.
"rl_gate_lr_multiplier_controller", // Phase 2A-D fix B1.3 (2026-06-03): adaptive gate-LR multiplier controller. Reads gate_entropy_mean (slot 781) via EMA → escalates +0.3 %/step when entropy_ema > 0.85·log(K) (under-learning), decays 5 %/step when < 0.20·log(K) (collapse risk), leaves alone in healthy band. Single-thread launch (1,1,1)/(1,1,1).
"rl_band_head_forward", // Phase 4-A (2026-06-03): No-transaction-band head forward — two-stage launch: (1) `rl_band_head_linear_fwd` produces [B × 2] pre-activation band logits via tree-reduce over HIDDEN_DIM; (2) `rl_band_apply_activation` applies asymmetric ±|tanh| × N_max_eff to enforce b_l ≤ 0 ≤ b_u per Davis-Norman optimality. Spec: docs/superpowers/specs/2026-06-03-no-transaction-band-architecture.md §1.1.
"rl_band_mask", // Phase 4-A: override actions[b]→Hold when position_lots[b] ∈ [b_l, b_u]. Master-gated at slot 799 (RL_BAND_ENABLED_INDEX). Grid=(B), Block=(1) — mirrors rl_confidence_gate launch shape; runs OUTSIDE graph capture per spec §9.5.
"rl_state_action_mask", // Phase 7b F5 (2026-06-05): state-conditional action availability mask. Sets pi_logits[b][a]=-INF for state-illegal actions BEFORE rl_pi_action_kernel samples. Master-gated at slot 823 (RL_F5_STATE_MASK_ENABLED_INDEX, bootstrap 0.0 = OFF). Grid=(B), Block=(1). Spec docs/superpowers/specs/2026-06-04-bellman-target-foundation-reshape.md §3.5.
"rl_band_turnover_loss", // Phase 4-A: turnover regularizer (Option b) with sigmoid surrogate for differentiable boundary gradient. Emits per-batch loss + per-batch grad on (b_l, b_u). Phase 4-A wires for OBSERVABILITY only; full encoder backward chain lands in Phase 4-B.
"rl_band_frac_aggregate", // Phase 4-B (2026-06-04): per-step `frac_not_masked` reducer (single-block tree-reduce over batch) → writes to RL_BAND_FRAC_NOT_MASKED_OBSERVED_INDEX (slot 812). Consumed by `rl_band_turnover_controller`.
"rl_band_turnover_controller", // Phase 4-B: adaptive turnover-target controller. Single-thread launch (1×1×1); first-observation bootstrap for EMA at slot 809 + Schulman-bounded asymmetric adapter for slot 811 from slot 812 input. Spec §3.1 Option (c).
"rl_band_head_backward", // Phase 4-B: backward through ±|tanh|×N_max activation + linear projection → per-batch grad_w/grad_b scratch + grad_h_t (OVERWRITE). Caller reduces via reduce_axis0 + folds grad_h_t into encoder via grad_h_accumulate_scaled. Spec §3.3.
];
// Cache bust v31 — five new reduce / derive kernels populate the input
// EMAs for the previously-frozen controllers (entropy_coef,
// rollout_steps, per_alpha, ppo_clip, target_tau, gamma). Each kernel
// is a single-block reduction (tree-reduce, grid-stride, or per-batch
// state update). The trainer launches each one immediately after its
// source signal is populated:
// * `rl_var_over_abs_mean_b` after compute_advantage_return
// * `rl_kurtosis_b` after dqn_distributional_q_bwd
// * `rl_kl_approx_b` after PPO surrogate forward
// * `rl_l2_diff_norm` after target-net soft update
// * `rl_step_counter_update` after extract_realized_pnl_delta
// The scalar output is then consumed by ema_update_per_step (continuous
// EMAs) or ema_update_on_done (done-gated EMAs) at the right ISV slot.
// `entropy_observed_ema` reuses ema_update_per_step's built-in
// per-batch mean reduce on entropy_d directly.
fn main() {
println!("cargo:rerun-if-changed=build.rs");
// Track shared headers so .cuh / .h edits trigger rebuilds of every
// .cu that #includes them. Without these, an edit to a helper header
// leaves a stale cubin.
println!("cargo:rerun-if-changed=cuda/gpu_log_ids.h");
println!("cargo:rerun-if-changed=cuda/gpu_log_helpers.cuh");
println!("cargo:rerun-if-env-changed=CARGO_FEATURE_CUDA");
if std::env::var("CARGO_FEATURE_CUDA").is_err() {
eprintln!(" ml-alpha: cuda feature disabled, skipping kernel build");
return;
}
println!("cargo:rerun-if-env-changed=CUDA_HOME");
println!("cargo:rerun-if-env-changed=CUDA_COMPUTE_CAP");
let nvcc = match find_nvcc() {
Some(p) => p,
None => {
eprintln!(" ml-alpha: nvcc not found, skipping kernel build (set CUDA_HOME or install CUDA toolkit)");
return;
}
};
let out = PathBuf::from(std::env::var("OUT_DIR").expect("OUT_DIR not set by cargo"));
// Detect GPU arch: CUDA_COMPUTE_CAP env > nvidia-smi query > default 86
let arch = detect_arch(&nvcc);
eprintln!(" ml-alpha: compiling kernels for sm_{arch}");
for k in KERNELS {
let src = PathBuf::from(format!("cuda/{k}.cu"));
if !src.exists() {
eprintln!(" ml-alpha: skipping {k} — source not yet present");
continue;
}
println!("cargo:rerun-if-changed={}", src.display());
let cubin = out.join(format!("{k}.cubin"));
compile(&nvcc, &src, &cubin, &arch);
}
}
fn detect_arch(_nvcc: &Path) -> String {
// 1. Explicit env override
if let Ok(cap) = std::env::var("CUDA_COMPUTE_CAP") {
return cap;
}
// 2. Query the GPU on this machine
if let Ok(output) = Command::new("nvidia-smi")
.args(["--query-gpu=compute_cap", "--format=csv,noheader"])
.output()
{
if output.status.success() {
let s = String::from_utf8_lossy(&output.stdout);
let cap = s.trim().replace('.', "");
if !cap.is_empty() {
return cap;
}
}
}
// 3. Default to sm_86 (RTX 3050 Ti local dev)
"86".to_string()
}
fn compile(nvcc: &Path, src: &Path, cubin: &Path, arch: &str) {
let status = Command::new(nvcc)
.args([
"-cubin",
&format!("-arch=sm_{arch}"),
"-O3",
"--use_fast_math",
"--ftz=true",
"--fmad=true",
"-o",
cubin.to_str().unwrap(),
src.to_str().unwrap(),
])
.status()
.unwrap_or_else(|e| panic!("nvcc spawn failed for {}: {e}", src.display()));
if !status.success() {
panic!(
"nvcc failed for {} (exit {})",
src.display(),
status.code().unwrap_or(-1)
);
}
eprintln!(
" ml-alpha: compiled {} -> {} (sm_{arch})",
src.display(),
cubin.display()
);
}
fn find_nvcc() -> Option<PathBuf> {
if let Ok(home) = std::env::var("CUDA_HOME") {
let p = PathBuf::from(home).join("bin/nvcc");
if p.exists() {
return Some(p);
}
}
for cand in ["/usr/local/cuda/bin/nvcc", "/usr/bin/nvcc"] {
let p = PathBuf::from(cand);
if p.exists() {
return Some(p);
}
}
Command::new("nvcc")
.arg("--version")
.output()
.ok()
.filter(|o| o.status.success())
.map(|_| PathBuf::from("nvcc"))
}