gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(sp18 v2 P3.T1-T5): adaptive HOLD_REWARD_POS/NEG_CAP producer kernel

Browse Source 
Lifts the Phase 2 caps from sentinel-driven cold-start (5.0/-10.0
fixed) to p99(|step_ret|) over Long/Short trade closes × 1.5 safety
factor with Wiener-optimal alpha blend. The Phase 2 consumer
(compute_sp18_hold_opportunity_cost) sees producer-driven slots
[483]/[484] from epoch 1 onward; sentinel branch in the consumer
remains as the cold-start fallback path (zero-trade-close epoch ⇒
producer early-returns ⇒ slots stay at sentinel ⇒ consumer falls
through to the +5/-10 macro defaults — bit-identical to pre-Phase-3).

Mirrors SP14 P0-A reward_cap_update_kernel structural template with
three differences: (1) filter (is_close && step_ret != 0) — both
winners AND losers (the consumer needs the magnitude scale of *all*
Long/Short closes); (2) Welford-derived Wiener-α (slots [487..493))
replaces fixed α=0.01, with floor at WELFORD_ALPHA_MIN=0.4 per
pearl_wiener_alpha_floor_for_nonstationary (the policy-realised
distribution is intrinsically non-stationary as the policy adapts);
(3) bounds [0.5, 50.0] (vs. position-side [1.0, 50.0]).

Atomic single-commit per feedback_no_partial_refactor:
- crates/ml/src/cuda_pipeline/hold_reward_cap_update_kernel.cu (NEW)
- crates/ml/build.rs cubin manifest entry
- HoldRewardCapUpdateOps in gpu_aux_trunk.rs (new struct + impl)
- HOLD_REWARD_CAP_UPDATE_CUBIN static + struct field +
  launch_hold_reward_cap_update method + constructor instantiation +
  field-init in gpu_dqn_trainer.rs (5 sites)
- Per-epoch boundary launch in training_loop.rs right AFTER
  launch_reward_cap_update (shared step_ret/trade_close source buffers,
  independent ISV slot pairs)
- HEALTH_DIAG[N]: hold_reward_cap [pos={:.4} neg={:.4} fire_rate={:.4}]
- 3 GPU oracle tests (T5 producer-drives-slots, Pearl-A REPLACE,
  no-closes preserves-isv) — all pass on local RTX 3050 Ti
- Phase 3 close-out sections in docs/sp18-wireup-audit.md and
  docs/dqn-wire-up-audit.md

Pearls applied: feedback_no_atomicadd, pearl_first_observation_bootstrap,
pearl_wiener_optimal_adaptive_alpha, pearl_wiener_alpha_floor_for_nonstationary,
pearl_no_host_branches_in_captured_graph, pearl_symmetric_clamp_audit,
pearl_audit_unboundedness_for_implicit_asymmetry (NEG = -2 × POS at
producer time, single source of truth), feedback_isv_for_adaptive_bounds,
pearl_fused_per_group_statistics_oracle.

Validation: cargo check --workspace clean; 3 GPU oracle tests pass on
local RTX 3050 Ti; scripts/audit_sp18_consumers.sh --check exits 0
(no fingerprint drift in tracked sections).

Plan: docs/superpowers/plans/2026-05-08-sp18-reward-shape-hold-attractor.md
Phase 4-5 (B-leg target-net forward + q_next replacement) follows.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-09 11:05:03 +02:00
parent 1f4cc0f207
commit 5ea5aa9b8e
8 changed files with 1455 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

134
crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs
View File

@@ -2168,6 +2168,27 @@ pub(crate) static AVG_WIN_HOLD_TIME_UPDATE_CUBIN: &[u8] = include_bytes!(concat!
/// `gpu_aux_trunk::RewardCapUpdateOps`. Per-epoch boundary launch.
pub(crate) static REWARD_CAP_UPDATE_CUBIN: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/reward_cap_update_kernel.cubin"));
/// SP18 v2 Phase 3 (2026-05-09): adaptive HOLD_REWARD_POS/NEG_CAP producer
/// cubin. Single-block 256-thread kernel that sweeps the per-epoch
/// `step_ret_per_sample` + `trade_close_per_sample` buffers (same source
/// `reward_cap_update_kernel` reads from), filters by `(is_close &&
/// step_ret != 0)` to extract Long/Short close magnitudes (DD3=b),
/// computes a Welford `mean + Z_99 × sigma` p99 estimator with
/// `max(p99, max_observed)` conservative takeover × HOLD_REWARD_CAP_
/// SAFETY_FACTOR=1.5, applies Wiener-optimal α blend (Welford
/// accumulators in slots [487..493); floor at WELFORD_ALPHA_MIN=0.4 per
/// `pearl_wiener_alpha_floor_for_nonstationary`), and writes both
/// `ISV[HOLD_REWARD_POS_CAP_INDEX=483]` and
/// `ISV[HOLD_REWARD_NEG_CAP_INDEX=484]` (NEG = −2 × POS, DD5=b mirrored
/// asymmetry — same Kahneman 2:1 ratio as the position-side cap, single
/// source of truth at producer time per
/// `pearl_audit_unboundedness_for_implicit_asymmetry`). Pearl-A
/// first-observation bootstrap on POS (sentinels POS=5.0/NEG=-10.0
/// match SP14 P0-A pattern). Loaded by
/// `gpu_aux_trunk::HoldRewardCapUpdateOps`. Per-epoch boundary launch
/// right AFTER `launch_reward_cap_update`.
pub(crate) static HOLD_REWARD_CAP_UPDATE_CUBIN: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/hold_reward_cap_update_kernel.cubin"));
/// Class A P1-Producer (2026-05-08): adaptive Bayesian Kelly priors producer cubin.
/// Single-block 256-thread kernel that sweeps the per-fold-end
/// `portfolio_state[n_envs, PS_STRIDE]` buffer (same source the
@@ -7784,6 +7805,28 @@ pub struct GpuDqnTrainer {
/// foundation of training and shouldn't move fast.
reward_cap_update_ops: super::gpu_aux_trunk::RewardCapUpdateOps,
/// SP18 v2 Phase 3 (2026-05-09): adaptive HOLD_REWARD_POS/NEG_CAP
/// producer Ops handle. Pre-loads the `CudaFunction` at construction
/// per `pearl_no_host_branches_in_captured_graph.md`. Sweeps the
/// per-epoch `step_ret_per_sample` + `trade_close_per_sample`
/// buffers (same source `reward_cap_update_ops` reads from), filters
/// by `(is_close && step_ret != 0)` to extract Long/Short close
/// magnitudes per spec DD3=b, computes a Welford p99 estimator ×
/// 1.5 safety factor → POS cap, derives Wiener-optimal α from
/// Welford accumulators in slots [487..493), and writes both
/// `ISV[HOLD_REWARD_POS_CAP_INDEX=483]` and
/// `ISV[HOLD_REWARD_NEG_CAP_INDEX=484]` (NEG = −2 × POS, DD5=b
/// mirrored asymmetry, same Kahneman 2:1 ratio as the position-
/// side cap). Block-tree-reduce (no atomicAdd); Pearl-A first-
/// observation bootstrap; Wiener-α floor at 0.4 per
/// `pearl_wiener_alpha_floor_for_nonstationary` since the policy-
/// realised distribution is intrinsically non-stationary as the
/// policy adapts. Per-epoch boundary launch right AFTER
/// `reward_cap_update_ops` so both cap producers share the same
/// step_ret/trade_close source buffers; no shared state — they
/// drive independent ISV slot pairs.
hold_reward_cap_update_ops: super::gpu_aux_trunk::HoldRewardCapUpdateOps,
/// Class A P1-Producer (2026-05-08): adaptive Bayesian Kelly priors
/// producer. Sweeps `portfolio_state[n_envs, PS_STRIDE]` (same source
/// `kelly_cap_update_kernel` reads), aggregates the realized
@@ -15565,6 +15608,86 @@ impl GpuDqnTrainer {
Ok(())
}
/// SP18 v2 Phase 3 (2026-05-09): launch the adaptive HOLD_REWARD_POS/
/// NEG_CAP producer. Sweeps the per-epoch `step_ret_per_sample` +
/// `trade_close_per_sample` buffers (same source `launch_reward_cap_update`
/// reads from), filters by `(is_close && step_ret != 0)` to extract
/// Long/Short close magnitudes (per spec DD3=b — Hold/Flat closes
/// don't generate non-zero step_ret per `experience_kernels.cu:2855`),
/// computes a Welford `mean + Z_99 × sigma` p99 estimator ×
/// `HOLD_REWARD_CAP_SAFETY_FACTOR=1.5`, applies Wiener-optimal α
/// blend (Welford accumulators in slots [487..493); floor at
/// `WELFORD_ALPHA_MIN=0.4` per
/// `pearl_wiener_alpha_floor_for_nonstationary`), and writes both
/// `ISV[HOLD_REWARD_POS_CAP_INDEX=483]` (POS cap) and
/// `ISV[HOLD_REWARD_NEG_CAP_INDEX=484]` (NEG = −2 × POS, DD5=b
/// mirrored asymmetry — same Kahneman 2:1 ratio as the position-
/// side `launch_reward_cap_update`, single source of truth at
/// producer time per
/// `pearl_audit_unboundedness_for_implicit_asymmetry`).
///
/// Per-epoch boundary launch — call AFTER `launch_reward_cap_update`
/// so both cap producers share the same source buffers (no shared
/// state; they drive independent ISV slot pairs at [452]/[453]
/// and [483]/[484]).
///
/// Pearls applied:
/// - `pearl_first_observation_bootstrap.md` (sentinel POS=5.0/
/// NEG=-10.0 match SP14 P0-A pattern for bit-identical cold-start)
/// - `pearl_no_host_branches_in_captured_graph.md`
/// - `feedback_no_atomicadd.md` (block-tree-reduce in shmem only)
/// - `pearl_symmetric_clamp_audit.md` (bilateral clamp on POS, NEG)
/// - `feedback_isv_for_adaptive_bounds.md` (POS bounds [0.5, 50.0]
/// are dimensional safety floors, not tuning)
/// - `pearl_wiener_optimal_adaptive_alpha.md` (Welford-derived α)
/// - `pearl_wiener_alpha_floor_for_nonstationary.md` (α ≥ 0.4)
/// - `pearl_fused_per_group_statistics_oracle.md` (sum/sumsq/count/
/// max accumulated in ONE fused reduction)
pub(crate) fn launch_hold_reward_cap_update(
&self,
step_ret_dev_ptr: u64,
trade_close_dev_ptr: u64,
total_samples: i32,
) -> Result<(), MLError> {
use crate::cuda_pipeline::sp14_isv_slots::{
HOLD_REWARD_CAP_SAFETY_FACTOR, HOLD_REWARD_NEG_CAP_INDEX,
HOLD_REWARD_POS_CAP_INDEX, HOLD_REWARD_POS_CAP_MAX_BOUND,
HOLD_REWARD_POS_CAP_MIN_BOUND, HRC_DIFF_M2_INDEX, HRC_DIFF_MEAN_INDEX,
HRC_PREV_TARGET_INDEX, HRC_SAMPLE_COUNT_INDEX, HRC_TARGET_M2_INDEX,
HRC_TARGET_MEAN_INDEX, REWARD_NEG_TO_POS_RATIO, SENTINEL_HOLD_REWARD_NEG_CAP,
SENTINEL_HOLD_REWARD_POS_CAP,
};
debug_assert!(
self.isv_signals_dev_ptr != 0,
"launch_hold_reward_cap_update: isv_signals_dev_ptr must be allocated"
);
self.hold_reward_cap_update_ops.launch(
&self.stream,
step_ret_dev_ptr,
trade_close_dev_ptr,
total_samples,
self.isv_signals_dev_ptr,
HOLD_REWARD_POS_CAP_INDEX as i32,
HOLD_REWARD_NEG_CAP_INDEX as i32,
HRC_TARGET_MEAN_INDEX as i32,
HRC_TARGET_M2_INDEX as i32,
HRC_DIFF_MEAN_INDEX as i32,
HRC_DIFF_M2_INDEX as i32,
HRC_PREV_TARGET_INDEX as i32,
HRC_SAMPLE_COUNT_INDEX as i32,
SENTINEL_HOLD_REWARD_POS_CAP,
SENTINEL_HOLD_REWARD_NEG_CAP,
HOLD_REWARD_CAP_SAFETY_FACTOR,
REWARD_NEG_TO_POS_RATIO,
HOLD_REWARD_POS_CAP_MIN_BOUND,
HOLD_REWARD_POS_CAP_MAX_BOUND,
)?;
Ok(())
}
/// Class A P1-Producer (2026-05-08): launch the adaptive Bayesian Kelly
/// priors producer. Per-fold-end (per-epoch boundary in current
/// scheduling) launch — Bayesian prior is a *prior belief* and should
@@ -24277,6 +24400,15 @@ impl GpuDqnTrainer {
let reward_cap_update_ops =
super::gpu_aux_trunk::RewardCapUpdateOps::new(&stream)?;
// SP18 v2 Phase 3 (2026-05-09): adaptive HOLD_REWARD_POS/NEG_CAP
// producer. Pre-loads the `CudaFunction` handle at construction
// per `pearl_no_host_branches_in_captured_graph.md`. Per-epoch
// boundary launch (Wiener-α blend with floor at 0.4 — non-
// stationary control loop tracks the policy-realised Long/Short
// close magnitude distribution as the policy adapts).
let hold_reward_cap_update_ops =
super::gpu_aux_trunk::HoldRewardCapUpdateOps::new(&stream)?;
// Class A P1-Producer (2026-05-08): adaptive Bayesian Kelly priors
// producer. Pre-loads the `CudaFunction` handle at construction per
// `pearl_no_host_branches_in_captured_graph.md`. Per-fold-end (per-epoch
@@ -25157,6 +25289,8 @@ impl GpuDqnTrainer {
avg_win_hold_time_update_ops,
// Class A P0-A (2026-05-08): adaptive reward-cap producer.
reward_cap_update_ops,
// SP18 v2 Phase 3 (2026-05-09): adaptive Hold-reward-cap producer.
hold_reward_cap_update_ops,
// Class A P1-Producer (2026-05-08): adaptive Bayesian Kelly priors producer.
kelly_bayesian_priors_update_ops,
// Class A audit-fix Batch 4-A (2026-05-08): adaptive DD saturation floor producer.