Files
foxhunt/crates/ml-alpha/build.rs
jgrusewski 56efd96cb2 feat(rl): C51 distributional Q-head + PER replay + 2 ISV controllers (Phase C)
Adds the DQN component of the integrated RL trainer per the plan at
docs/superpowers/plans/2026-05-22-integrated-rl-trainer.md.

What this commit lands:
- DqnHead: linear projection h_t [B, HIDDEN_DIM] -> atom logits
  [B, N_ACTIONS=9, Q_N_ATOMS=21] with parallel target-network weights.
  Xavier x 0.01 init (initial softmax-over-atoms approx uniform),
  scoped_init_seed-guarded per pearl_scoped_init_seed_for_reproducibility.
- dqn_distributional_q.cu: forward (one block per (batch, action), one
  thread per atom) + Bellman categorical-CE backward against a pre-projected
  target distribution. Atom-softmax fused into backward.
- ReplayBuffer (rl/replay.rs): capacity-bounded PER with priority^alpha
  sampling, random replacement, and TD-error priority update. O(N)
  cumulative-sum sampling; Phase E may upgrade to a GPU sum-tree once
  capacity profiling demands it.
- rl_gamma_controller.cu: ISV[RL_GAMMA_INDEX=400] producer; gamma
  adapts toward 0.5^(1/mean_trade_duration) via Wiener-alpha blend
  (floor 0.4 per pearl_wiener_alpha_floor_for_nonstationary), clamped
  to [0.90, 0.999]. Bootstrap gamma = 0.99 on sentinel.
- rl_target_tau_controller.cu: ISV[RL_TARGET_TAU_INDEX=401] producer;
  tau adapts multiplicatively from Q-divergence ratio vs anchor 0.01,
  Wiener-alpha blend with floor 0.4, clamped to [0.001, 0.05].
  Bootstrap tau = 0.005 on sentinel.
- Action enum + try_from_u32 in rl/common.rs (matches existing ml DQN
  action grid for cross-system policy comparability).
- C51 atom support constants Q_V_MIN / Q_V_MAX in rl/common.rs (kept
  for Phase E's projection kernel; backward in this commit operates in
  categorical domain on a pre-projected target).

What this commit DEFERS to Phase E:
- soft_update_target kernel (struct fields w_target_d / b_target_d are
  wired and read by the Bellman backward in this commit; the writer
  lives in Phase E alongside the training-loop tau driver).
- Categorical projection kernel that reads gamma from ISV[400] and
  produces the target_dist input to the backward kernel.
- Toy bandit test activation (tests/dqn_toy.rs is #[ignore]-gated; the
  type contract is locked here, the training loop wires in Phase E).
- atomicAdd in the per-batch CE accumulator (Phase E replaces with the
  warp-shuffle + shared reduce pattern from aux_loss.cu when batches
  reach production sizes; B <= 32 toy contention is negligible).

Per pearl_controller_anchors_isv_driven and feedback_isv_for_adaptive_bounds:
gamma and tau are NOT hardcoded constants. They live in ISV[400] /
ISV[401], emitted by the controller kernels above, and Phase E consumers
read via __ldg(isv + INDEX). Bootstrap values (0.99, 0.005) appear only
in the controller kernel as first-observation defaults, NOT baked into
the loss kernel.

Validation:
- SQLX_OFFLINE=true cargo check -p ml-alpha --lib  -> clean (1m 03s)
- SQLX_OFFLINE=true cargo check --workspace --lib   -> clean (42s)
- SQLX_OFFLINE=true cargo test -p ml-alpha --lib rl::replay -> 3 pass
- Cubins built for all 3 new kernels (sm_80 default).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-22 22:45:23 +02:00

131 lines
5.8 KiB
Rust
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
//! Pre-compile all ml-alpha CUDA kernels into arch-specific cubins.
//!
//! Per `feedback_no_nvrtc.md`: no runtime kernel compilation.
//! Per `pearl_build_rs_rerun_if_env_changed.md`: every `std::env::var`
//! is paired with `cargo:rerun-if-env-changed`.
use std::path::{Path, PathBuf};
use std::process::Command;
const KERNELS: &[&str] = &[
"mamba2_alpha_kernel", // Mamba2 SSM scan kernel (used by PerceptionTrainer's encoder prefix)
"snap_feature_assemble",
"cfc_step",
"multi_horizon_heads",
"projection",
"bce_loss_multi_horizon", // Kendall σ-weighted multi-horizon BCE (axis A)
"adamw_step",
"grad_norm",
"horizon_lambda", // ISV-driven per-horizon gradient scaler (EMA + lambda)
"layer_norm", // Phase 1: trunk pre-CfC normalisation
"variable_selection", // Phase 2D: TFT-style per-feature gating
"attention_pool", // Phase 3: single-Q learned content summary at CfC k=0
"reduce_axis0", // Phase B: cross-batch param-grad reducer
"output_smoothness", // CRT.train: per-horizon adjacent-position prob-jitter penalty
"smoothness_lambda_controller", // CRT.train: ISV-driven λ controller anchored on h30 jitter
"gpu_log_ring", // GPU diagnostic log ring — tick kernel + log_record helper
"bucket_transition_kernels", // Per-horizon CfC Phase 1→2 transition: tau_sort, bucket_assign, bucket_iqr, channels_in_bucket, heads_compact, zero_off_bucket (ALPHA fix 2026-05-21)
"cfc_step_per_branch", // Per-horizon CfC Phase 2: fused per-(batch, branch) fwd + bwd over [25,25,25,25,28] buckets
"heads_block_diagonal_fwd", // Per-horizon CfC Phase 2: heads w_skip projection with compact ragged storage (640→128 floats)
"aux_trunk", // SDD-3 Layer B3: smaller single-bucket CfC trunk (AUX_HIDDEN=64) for outcome-supervision (D-labels)
"aux_heads", // SDD-3 Layer B4: per-direction linear regression heads on AuxTrunk output (long + short, N_AUX_HORIZONS each)
"aux_loss", // SDD-3 Layer B4: Huber loss + grad for aux trade-outcome regression targets (NaN-masked)
"aux_vec_add", // SDD-3 Layer B5: element-wise dst += src for aux→encoder gradient accumulation (lifted stop-grad)
"dqn_distributional_q", // RL Phase C: C51 distributional Q-head fwd + Bellman TD bwd for integrated RL trainer
"rl_gamma_controller", // RL Phase C: ISV controller emitting γ to ISV[RL_GAMMA_INDEX=400]
"rl_target_tau_controller", // RL Phase C: ISV controller emitting τ to ISV[RL_TARGET_TAU_INDEX=401]
];
// Cache bust v20 (2026-05-22): RL Phase C — dqn_distributional_q.cu (C51 fwd + categorical-CE bwd over 9 actions × 21 atoms), rl_gamma_controller.cu (γ→ISV[400], anchored on mean trade duration), rl_target_tau_controller.cu (τ→ISV[401], anchored on Q-divergence). Per pearl_controller_anchors_isv_driven, γ + τ are not constants — Wiener-α adaptive with first-observation bootstrap.
fn main() {
println!("cargo:rerun-if-changed=build.rs");
// Track shared headers so .cuh / .h edits trigger rebuilds of every
// .cu that #includes them. Without these, an edit to a helper header
// leaves a stale cubin.
println!("cargo:rerun-if-changed=cuda/gpu_log_ids.h");
println!("cargo:rerun-if-changed=cuda/gpu_log_helpers.cuh");
println!("cargo:rerun-if-env-changed=CARGO_FEATURE_CUDA");
if std::env::var("CARGO_FEATURE_CUDA").is_err() {
eprintln!(" ml-alpha: cuda feature disabled, skipping kernel build");
return;
}
println!("cargo:rerun-if-env-changed=CUDA_COMPUTE_CAP");
println!("cargo:rerun-if-env-changed=CUDA_HOME");
let cap = std::env::var("CUDA_COMPUTE_CAP").unwrap_or_else(|_| "80".to_string());
let arch = format!("sm_{cap}");
let nvcc = match find_nvcc() {
Some(p) => p,
None => {
eprintln!(" ml-alpha: nvcc not found, skipping kernel build (set CUDA_HOME or install CUDA toolkit)");
return;
}
};
let out = PathBuf::from(std::env::var("OUT_DIR").expect("OUT_DIR not set by cargo"));
for k in KERNELS {
let src = PathBuf::from(format!("cuda/{k}.cu"));
if !src.exists() {
eprintln!(" ml-alpha: skipping {k} — source not yet present");
continue;
}
println!("cargo:rerun-if-changed={}", src.display());
let cubin = out.join(format!("{k}.cubin"));
compile(&nvcc, &src, &cubin, &arch);
}
}
fn compile(nvcc: &Path, src: &Path, cubin: &Path, arch: &str) {
let status = Command::new(nvcc)
.args([
"-cubin",
&format!("-arch={arch}"),
"-O3",
"--use_fast_math",
"--ftz=true",
"--fmad=true",
"-o",
cubin.to_str().unwrap(),
src.to_str().unwrap(),
])
.status()
.unwrap_or_else(|e| panic!("nvcc spawn failed for {}: {e}", src.display()));
if !status.success() {
panic!(
"nvcc failed for {} (exit {})",
src.display(),
status.code().unwrap_or(-1)
);
}
eprintln!(
" ml-alpha: compiled {} -> {} ({arch})",
src.display(),
cubin.display()
);
}
fn find_nvcc() -> Option<PathBuf> {
if let Ok(home) = std::env::var("CUDA_HOME") {
let p = PathBuf::from(home).join("bin/nvcc");
if p.exists() {
return Some(p);
}
}
for cand in ["/usr/local/cuda/bin/nvcc", "/usr/bin/nvcc"] {
let p = PathBuf::from(cand);
if p.exists() {
return Some(p);
}
}
Command::new("nvcc")
.arg("--version")
.output()
.ok()
.filter(|o| o.status.success())
.map(|_| PathBuf::from("nvcc"))
}