gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(alpha): wire stacker-threshold controller into smoke rollout-end

Browse Source 
Phase E.2 Task 17. Loads stacker_threshold_controller.cubin at smoke
startup, initialises ISV[544] (TRADE_RATE_TARGET) to 0.08 (CLI flag
--trade-rate-target, never reset), allocates a 3-float Wiener state
buffer for slot 545's Pearl A+D state.

Invokes the controller at every episode end with:
  rollout_trade_count    = count of non-Wait actions in the episode
  rollout_total_decisions = actions_host.len() (= ep_len)
  rollout_realized_sharpe = ep_terminal_R / RANDOM_BASELINE_STD
                            (per-rollout analog of the rvr metric;
                             lets the Kelly-atten controller respond
                             to in-policy performance vs the baseline
                             noise floor)

CLI args added:
  --trade-rate-target  default 0.08 (8% per-step trade rate target)
  --k-threshold        default 0.01
  --k-atten            default 0.005
  --target-sharpe      default 0.5
  --wiener-alpha-floor default 0.4
  --ctl-alpha-meta     default 0.1

Periodic log line extended:
  ep ... | KC q/H/rvr/ΔQ ... | CTL thresh=... obs=... atten=...

Final JSON adds:
  final_stacker_threshold
  final_trade_rate_observed_ema
  final_stacker_kelly_attenuation
  trade_rate_target

Smoke run (H=600, 1000 episodes) verifies the controller is alive:
  ISV[543] STACKER_THRESHOLD:        0.000 → 0.5000 (saturated at ceiling)
  ISV[545] TRADE_RATE_OBSERVED_EMA:  0.000 → 0.712
  ISV[546] STACKER_KELLY_ATTENUATION:0.000 → 0.100 (hit floor)
  Verdict: PASS — rvr=+1.043σ (unchanged from Task 12b PASS, expected
           since smoke doesn't yet CONSUME slots 543/546).

Tuning notes (calibration for production, not bugs):
  • Threshold saturating at 0.5 → policy trades ~85% (target 8%, off by
    10×). Either re-calibrate target_trade_rate from realistic backtest
    behaviour, or raise the clamp ceiling. Current ε-greedy with low
    threshold-consumption gate produces high trade rate.
  • Kelly atten hit floor (0.1) because rollout_sharpe (~-0.004) is far
    below target_sharpe=0.5. The target needs to match the rollout
    metric's scale, OR the metric should be time-normalised. The
    current ep_terminal_R / baseline_std proxy is meaningful but its
    scale doesn't match a typical annualised Sharpe target.

These tuning items don't gate Milestone E.2 — the producer-side
controller is correctly driving the ISV slots; *consuming* those slots
(threshold gate on alpha signal, Kelly-cap multiplier) is Phase E.3
work (alpha + execution composition).

Phase E.2 Tasks 16 + 17 close-out: kernel + launcher + GPU smoke test
+ wired into smoke binary + initialisation + verified end-to-end. Tasks
19-22 (NoisyNet) are gated on Task 12 FAIL, which we passed — skipped.
Task 18 (alpha-trust ablation, ~9-18 hours compute) deferred to a
dedicated session if needed.
This commit is contained in:
jgrusewski
2026-05-15 17:35:35 +02:00
parent 85792ed28a
commit 91383507fc
2 changed files with 188 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

90
crates/ml/examples/alpha_dqn_h600_smoke.rs
View File

@@ -139,6 +139,27 @@ struct Cli {
/// Episodes between kill-criteria pipeline launches.
#[arg(long, default_value_t = 50)]
kill_criteria_every: usize,
/// Phase E.2 stacker-threshold controller — target trade rate (ISV[544]).
/// Set once at training start, never reset (TrainingPersist anchor).
#[arg(long, default_value_t = 0.08)]
trade_rate_target: f32,
/// Stacker-threshold controller P-gain on rate error.
#[arg(long, default_value_t = 0.01)]
k_threshold: f32,
/// Kelly-attenuation controller P-gain on Sharpe error.
#[arg(long, default_value_t = 0.005)]
k_atten: f32,
/// Target rollout Sharpe for the Kelly-attenuation controller.
#[arg(long, default_value_t = 0.5)]
target_sharpe: f32,
/// Pearl D Wiener-α floor on the observed-rate EMA (slot 545). 0.4 per
/// `pearl_wiener_alpha_floor_for_nonstationary` — controller co-adapts
/// with the policy, so the EMA needs to stay responsive.
#[arg(long, default_value_t = 0.4)]
wiener_alpha_floor: f32,
/// Meta-α for the Wiener variance updates inside the controller.
#[arg(long, default_value_t = 0.1)]
ctl_alpha_meta: f32,
/// Reward normalization scale. Rewards are divided by this before
/// being passed to the Munchausen target — the network learns
/// normalized Q-values. Default 1000 ≈ |median reward| at H=600
@@ -512,6 +533,14 @@ fn main() -> Result<()> {
let pearls_kernel = pearls_module.load_function("apply_pearls_ad_kernel")
.context("pearls load")?;
// Phase E.2 Task 16: stacker-threshold + Kelly-attenuation controller
let ctl_module = ctx
.load_cubin(ml::cuda_pipeline::alpha_kernels::STACKER_THRESHOLD_CONTROLLER_CUBIN.to_vec())
.context("controller cubin load")?;
let ctl_kernel = ctl_module
.load_function("stacker_threshold_controller_update")
.context("controller kernel load")?;
// --- Load env data ---
let fill_model = load_fill_model(&cli.fill_coeffs)?;
info!("Loaded FillModel from {}", cli.fill_coeffs.display());
@@ -595,12 +624,19 @@ fn main() -> Result<()> {
// bug fixes. See alpha_random_baseline.json for the source numbers.
isv_host[RANDOM_BASELINE_MEAN_INDEX] = -5191.53;
isv_host[RANDOM_BASELINE_STD_INDEX] = 4963.62;
// Phase E.2 Task 17: trade-rate target anchor for the stacker-threshold
// controller. Set at training start, never reset across folds
// (TrainingPersist). Default 0.08 = ~8% per-step trade rate.
isv_host[ml::cuda_pipeline::alpha_isv_slots::TRADE_RATE_TARGET_INDEX] =
cli.trade_rate_target;
let mut isv_dev = stream.clone_htod(&isv_host).context("upload isv")?;
// Wiener state for the 4 kill-criteria slots: 4 × [sample_var, diff_var, x_lag] = 12 floats.
let mut wiener_dev = stream.alloc_zeros::<f32>(12).context("alloc wiener")?;
// Scratch buffer for kill-criteria producer output: 4 floats.
let mut kc_scratch_dev = stream.alloc_zeros::<f32>(4).context("alloc kc_scratch")?;
// Phase E.2: Wiener state for slot 545 (observed-rate EMA): 3 floats.
let mut ctl_wiener_dev = stream.alloc_zeros::<f32>(3).context("alloc ctl wiener")?;
// --- Allocate per-episode batch buffers (sized to horizon, reused) ---
let h = cli.horizon as i32;
@@ -822,6 +858,47 @@ fn main() -> Result<()> {
}
}
// Phase E.2 Task 17: Stacker-threshold controller at rollout end.
// Trade rate = (count of non-Wait actions) / total decisions in the
// episode. Realized Sharpe ≈ mean(returns) / std(returns) over the
// window — for a single rollout this collapses to the rollout's
// terminal reward divided by the random baseline std (a per-rollout
// analog of the rvr metric, lets the controller decide whether to
// tighten or loosen Kelly).
let trade_count_ep: f32 = actions_host
.iter()
.filter(|&&a| a != 0) // 0 == Wait
.count() as f32;
let decisions_ep = actions_host.len() as f32;
// Single-episode Sharpe proxy: terminal reward / baseline std.
// Note: terminal reward is the LAST element of rewards_host
// (sum of step rewards from env.step into ep.cumulative_pnl).
let ep_terminal_r = *rewards_host.last().unwrap_or(&0.0);
let baseline_std = isv_host[RANDOM_BASELINE_STD_INDEX].max(1e-6);
let rollout_sharpe = ep_terminal_r / baseline_std;
unsafe {
let (isv_ptr, _g0) = isv_dev.device_ptr_mut(&stream);
let (w_ptr, _g1) = ctl_wiener_dev.device_ptr_mut(&stream);
ml::cuda_pipeline::alpha_kernels::launch_stacker_threshold_controller(
&stream,
&ctl_kernel,
trade_count_ep,
decisions_ep,
rollout_sharpe,
cli.target_sharpe,
cli.k_threshold,
cli.k_atten,
cli.wiener_alpha_floor,
cli.ctl_alpha_meta,
ml::cuda_pipeline::alpha_isv_slots::STACKER_THRESHOLD_INDEX as i32,
ml::cuda_pipeline::alpha_isv_slots::TRADE_RATE_TARGET_INDEX as i32,
ml::cuda_pipeline::alpha_isv_slots::TRADE_RATE_OBSERVED_EMA_INDEX as i32,
ml::cuda_pipeline::alpha_isv_slots::STACKER_KELLY_ATTENUATION_INDEX as i32,
isv_ptr,
w_ptr,
)?;
}
// Target network hard update every K episodes: w_target ← w_online.
// Standard DQN stabilizer — V_soft(s') bootstrap reads w_target,
// so updating it slowly breaks the chase-its-own-tail divergence
@@ -901,9 +978,14 @@ fn main() -> Result<()> {
isv[RETURN_VS_RANDOM_EMA_INDEX],
isv[EARLY_Q_MOVEMENT_EMA_INDEX],
];
// Phase E.2: controller-driven slots for visibility.
let ctl_threshold = isv[ml::cuda_pipeline::alpha_isv_slots::STACKER_THRESHOLD_INDEX];
let ctl_obs_rate = isv[ml::cuda_pipeline::alpha_isv_slots::TRADE_RATE_OBSERVED_EMA_INDEX];
let ctl_atten = isv[ml::cuda_pipeline::alpha_isv_slots::STACKER_KELLY_ATTENUATION_INDEX];
info!(
"ep {:>4} | ε={:.3} | rollout_R_mean={:>+9.1} | KC: q_spread={:.4} entropy={:.4} rvr={:+.4} early_mvmt={:.4}",
ep + 1, eps, rollout_r_mean, kc[0], kc[1], kc[2], kc[3]
"ep {:>4} | ε={:.3} | R_mean={:>+9.1} | KC q={:.3} H={:.3} rvr={:+.3} ΔQ={:.3} | CTL thresh={:.4} obs={:.3} atten={:.3}",
ep + 1, eps, rollout_r_mean, kc[0], kc[1], kc[2], kc[3],
ctl_threshold, ctl_obs_rate, ctl_atten,
);
kc_logs.push((ep + 1, kc));
}
@@ -965,6 +1047,10 @@ fn main() -> Result<()> {
"target_update_every": cli.target_update_every,
"grad_clip": cli.grad_clip,
"q_init_norm": q_init_norm,
"final_stacker_threshold": isv_final[ml::cuda_pipeline::alpha_isv_slots::STACKER_THRESHOLD_INDEX],
"final_trade_rate_observed_ema": isv_final[ml::cuda_pipeline::alpha_isv_slots::TRADE_RATE_OBSERVED_EMA_INDEX],
"final_stacker_kelly_attenuation": isv_final[ml::cuda_pipeline::alpha_isv_slots::STACKER_KELLY_ATTENUATION_INDEX],
"trade_rate_target": cli.trade_rate_target,
"q_spread_ema": kc_final[0],
"action_entropy_ema": kc_final[1],
"return_vs_random_ema": kc_final[2],