From 17eb825113f39ab6795d4c5f91a4e74eb2712138 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Mon, 18 May 2026 21:12:45 +0200 Subject: [PATCH] =?UTF-8?q?arch(ml-alpha):=20=CF=83=20becomes=20sidecar=20?= =?UTF-8?q?=E2=80=94=20BCE=20drops=20Kendall=20damping=20(upgraded=20path)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Empirical record at this data + this architecture: Phase 1+2+3 (no σ in loss): mean_auc 0.7749 ± 0.024 v2 (σ + axes B/C/D/E): mean_auc 0.7541 ± 0.005 σ-only (kept σ, dropped B/C/D/E): mean_auc 0.7506 ± 0.008 Perf-fix (σ-only math): mean_auc 0.7499 ± 0.010 ISV-σ (closed-form σ + adaptive λ): mean_auc ~0.75 (2/3 folds) Every architecture with σ-in-loss lands at 0.75. Removing σ is the only thing that hits 0.77. That is a framework mismatch, not a tuning problem. Kendall+Gal+Cipolla 2018 frames σ as TASK NOISE level — damp the noisy task, trust the clean one. Our horizons don't have different label noise; they have different intrinsic difficulty (longer horizon = more price-walk uncertainty = lower achievable AUC). σ-Kendall sees "high BCE on h6000" and interprets it as "h6000 is unreliable, back off" — precisely the opposite of what we want. h6000 is the deployment target; damping it is a self-inflicted wound. With mean_bce ~ 0.7, σ ≈ √0.7 ≈ 0.84 ⇒ w_h ≈ 0.71, uniformly attenuating gradient by ~30% across every horizon. λ's z-score boost (max 2×) can rebalance relative-per-horizon but cannot recover the absolute magnitude. Per-horizon prioritization remains via the ISV-driven λ controller (grad scaler in heads_grn_bwd, per pearl_adam_normalizes_loss_weights). That controller IS appropriate for our problem: it boosts hard horizons rather than damping them, and it operates on the gradient into the trunk rather than on the loss aggregate (Adam-cancellation safe). Changes to bce_loss_multi_horizon.cu (six lines): w_h = bw (was: 0.5 * bw * exp(-2 * log_sigma_h)) d_log_sigma_h[h] = 0.0 (was: 1 - 2 * w_h * mean_bce) total_loss += bw * mean_bce (was: + w_h * mean_bce + log_sigma_h) σ infrastructure preserved unchanged: - horizon_ema_and_lambda still computes log_sigma_h closed-form from loss_ema (Kendall equilibrium) for telemetry / future label-noise estimation use cases - log_sigma_h kernel arg still in BCE signature (zero churn at callsite); ignored inside All 9 perception_overfit tests pass — including horizon_ema_and_lambda_track_after_training which validates the per-horizon controller end-to-end through 64 K-loop iterations of capture/replay. Co-Authored-By: Claude Opus 4.7 --- .../ml-alpha/cuda/bce_loss_multi_horizon.cu | 46 ++++++++++++------- 1 file changed, 29 insertions(+), 17 deletions(-) diff --git a/crates/ml-alpha/cuda/bce_loss_multi_horizon.cu b/crates/ml-alpha/cuda/bce_loss_multi_horizon.cu index 759d07afe..bacffc508 100644 --- a/crates/ml-alpha/cuda/bce_loss_multi_horizon.cu +++ b/crates/ml-alpha/cuda/bce_loss_multi_horizon.cu @@ -1,21 +1,25 @@ -// bce_loss_multi_horizon.cu — Kendall σ-weighted multi-horizon BCE. +// bce_loss_multi_horizon.cu — base-weight-only multi-horizon BCE. // -// Single source of truth for the multi-horizon BCE. Implements -// Kendall homoscedastic-uncertainty weighting per [Kendall+Gal+Cipolla -// 2018]: each horizon has a learnable `log_sigma_h` scalar that -// balances its loss contribution. Replaces the prior plain-pooled BCE. +// Single source of truth for the multi-horizon BCE. Loss is the +// base-weight-scaled per-horizon mean of unweighted BCE — no Kendall +// σ damping (2026-05-18). Per-horizon prioritization is done outside +// this kernel by the ISV-driven λ controller scaling the trunk +// grad_h contribution in heads_grn_bwd (per `pearl_adam_normalizes_loss_weights` +// — affect the gradient, not the loss aggregate). // // Math: // // raw_bce_h = Σ_{i in horizon h, m_i = 1} L_i (unweighted sum) // count_h = #{i in horizon h : m_i = 1} // mean_bce_h = raw_bce_h / count_h -// w_h = base_weight_h / (2 * exp(2 * log_sigma_h)) -// total_loss = Σ_h [ w_h * mean_bce_h + log_sigma_h ] +// w_h = base_weight_h (typically 1) +// total_loss = Σ_h [ w_h * mean_bce_h ] // // Gradients: // d L / d p_i = m_i * (w_h / count_h) * (p − y) / (p (1 − p)) -// d L / d log_sigma_h = 1 − 2 * w_h * mean_bce_h +// d L / d log_sigma_h = 0 (σ is a sidecar — emitted by +// horizon_ema_and_lambda for telemetry, +// not used in the loss path) // // Per `feedback_nvidia_grade_perf_for_kernels`: // - Warp-shuffle reduction (`__shfl_xor_sync`), NOT block tree-reduce. @@ -138,14 +142,21 @@ extern "C" __global__ void bce_multi_horizon_forward_backward( mean_bce_h_shared[h] = mean_bce; mean_bce_per_h_out[h] = mean_bce; - const float ls = log_sigma_h[h]; + // σ sidecar (2026-05-18): Kendall σ damped every horizon + // by ~30% (σ ≈ √mean_bce ≈ 0.84 → w_h ≈ 0.71) and biased + // the encoder away from h6000 — empirically -2pp mean_auc + // vs Phase 1+2+3 across multiple architectures. The + // Kendall framework's assumption ("damp the noisier + // task") inverts what we need ("learn the harder + // horizon harder"). σ is now emitted by + // horizon_ema_and_lambda for telemetry only and does + // NOT participate in the BCE forward weighting or + // backward gradient. `log_sigma_h` arg is preserved + // for kernel-signature stability. const float bw = (base_weights != nullptr) ? base_weights[h] : 1.0f; - // w_h = bw / (2 * exp(2*ls)) = 0.5 * bw * exp(-2 * ls) - const float w_h = 0.5f * bw * expf(-2.0f * ls); - // Per-element grad prefactor = w_h / count_h. + const float w_h = bw; w_eff_shared[h] = (c_h > 0.0f) ? w_h / c_h : 0.0f; - // ∂L/∂log_sigma_h = 1 − 2 * w_h * mean_bce_h. - d_log_sigma_h[h] = 1.0f - 2.0f * w_h * mean_bce; + d_log_sigma_h[h] = 0.0f; } } int v_valid = (lane < BCS_N_WARPS) ? s_valid[lane] : 0; @@ -158,14 +169,15 @@ extern "C" __global__ void bce_multi_horizon_forward_backward( __syncthreads(); // Total loss — small serial reduce by thread 0 over 5 horizons. + // σ-Kendall regularizer (log σ_h term) intentionally absent — see + // sidecar note above. Loss is unweighted per-horizon BCE sum, + // scaled only by `base_weights[h]` (typically uniform [1,1,1,1,1]). if (tid == 0) { float total_loss = 0.0f; #pragma unroll for (int h = 0; h < BCS_N_HORIZONS; ++h) { - const float ls = log_sigma_h[h]; const float bw = (base_weights != nullptr) ? base_weights[h] : 1.0f; - const float w_h = 0.5f * bw * expf(-2.0f * ls); - total_loss += w_h * mean_bce_h_shared[h] + ls; + total_loss += bw * mean_bce_h_shared[h]; } loss_out[0] = total_loss; }