feat(alpha): wire slot 543 consumption — 2D threshold × cost sweep
Phase E.3 Task 23 follow-up. Adds the confidence-threshold gate that
consumes the controller's ISV[543] output. Both binaries:
fn epsilon_greedy_gated(q, alpha_confidence, threshold, eps, rng) -> u8 {
if alpha_confidence < threshold { return 0; /* Wait */ }
epsilon_greedy(q, eps, rng)
}
State[1] is the env's alpha_confidence = |sigmoid(alpha_logit) - 0.5|
which is in [0, 0.5]; threshold is also clamped [0, 0.5], so direct
comparison is valid.
alpha_dqn_h600_smoke (closed-loop with controller):
Adds current_threshold: f32 cache, initialised to 0.0 (no gate),
refreshed via stream.clone_dtoh(&isv_dev) after each per-episode
controller invocation. Action selector reads current_threshold for
the NEXT episode's step decisions.
alpha_compose_backtest (2D sweep):
Adds --threshold-grid CLI flag (default [0.0, 0.05, 0.10, 0.15, 0.20,
0.25] — Phase 1d.4 pattern). Eval loop becomes 2D (threshold × cost).
Per-bin includes avg_n_trades for trade-rate visibility. End-of-run
prints BEST per-cost = max Sharpe_ann across τ.
Results (1000 train ep, 300 eval ep × 5 τ × 5 costs):
cost τ=0.00 best τ Sharpe lift trades/ep saved
------- ---------- --------- ----------- ---------------
0.0000 -41.78 -15.72 (τ=0.20) +26.1 477 → 168 (-65%)
0.0625 -71.46 -21.30 (τ=0.25) +50.2 476 → 138 (-71%)
0.1250 -86.78 -29.17 (τ=0.20) +57.6 482 → 167 (-65%)
0.2500 -108.57 -42.12 (τ=0.25) +66.5 480 → 132 (-73%)
0.5000 -146.76 -54.86 (τ=0.25) +91.9 478 → 136 (-72%)
Win rate at cost=0: 7.7% (no gate) → 20.3% (τ=0.20).
The gate architecture is VALIDATED: monotone improvement in win rate +
Sharpe + trade-rate reduction across all costs. The control loop
(controller → slot 543 → policy gate → observed rate feedback) is
sound. But the policy is STILL negative-Sharpe at every cost.
Phase 1d.4 baseline at half-tick: -4.0 (ours: -29.17). 25-pt gap.
Root cause of the remaining gap: the Q-network was TRAINED without
gate awareness. It learned Q-values for the over-trading regime. The
eval-only gate filters those decisions but can't fix miscalibrated
Q-values. Phase 1d.4 baseline beats us because its policy
(always-market-when-confident) is INHERENTLY gated by design — no
mismatched Q-values to fix.
Next iteration to close the 25-pt gap: train WITH gate on, so the
Q-network learns weights for the gated policy class. This means:
either (a) controller runs during training (smoke pattern) and the
threshold develops endogenously, or (b) fixed --train-threshold CLI
during training. Either way, the Q-network sees Wait-at-low-confidence
during the learning phase and adapts.
Files touched:
crates/ml/examples/alpha_dqn_h600_smoke.rs (gate + threshold cache)
crates/ml/examples/alpha_compose_backtest.rs (gate + 2D sweep)
config/ml/alpha_compose_backtest.json (2D verdict)
This commit is contained in:
jgrusewski
@@ -84,6 +84,12 @@ struct Cli {
|
||||
/// Phase 1d.4 used [0.0, 0.0625, 0.125, 0.25, 0.50].
|
||||
#[arg(long, value_delimiter = ',', default_value = "0.0,0.0625,0.125,0.25,0.5")]
|
||||
cost_grid: Vec<f32>,
|
||||
/// Comma-separated alpha-confidence threshold grid for the gate at eval.
|
||||
/// Direct analogue of Phase 1d.4's `--threshold` sweep — at each
|
||||
/// threshold, the policy is forced to Wait when |sigmoid(alpha)−0.5|
|
||||
/// < threshold. Threshold 0 = no gate (original Task 23 behaviour).
|
||||
#[arg(long, value_delimiter = ',', default_value = "0.0,0.05,0.10,0.15,0.20,0.25")]
|
||||
threshold_grid: Vec<f32>,
|
||||
#[arg(long, default_value_t = 1)]
|
||||
trade_size: i32,
|
||||
#[arg(long, default_value_t = 0xCAFEBABE_u64)]
|
||||
@@ -152,9 +158,25 @@ fn epsilon_greedy(q: &[f32], eps: f32, rng: &mut SmokeRng) -> u8 {
|
||||
}
|
||||
}
|
||||
|
||||
/// Phase E.3 confidence-gated ε-greedy. If `alpha_confidence < threshold`,
|
||||
/// force action=0 (Wait). Otherwise standard ε-greedy.
|
||||
fn epsilon_greedy_gated(
|
||||
q: &[f32],
|
||||
alpha_confidence: f32,
|
||||
threshold: f32,
|
||||
eps: f32,
|
||||
rng: &mut SmokeRng,
|
||||
) -> u8 {
|
||||
if alpha_confidence < threshold {
|
||||
return 0;
|
||||
}
|
||||
epsilon_greedy(q, eps, rng)
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, serde::Serialize)]
|
||||
struct CostBin {
|
||||
cost: f32,
|
||||
threshold: f32,
|
||||
n_episodes: usize,
|
||||
mean_reward: f32,
|
||||
std_reward: f32,
|
||||
@@ -163,6 +185,7 @@ struct CostBin {
|
||||
/// the eval window's mid_price index span × bar_seconds.
|
||||
sharpe_annualised: f32,
|
||||
win_rate: f32,
|
||||
avg_n_trades: f32,
|
||||
p05: f32,
|
||||
p50: f32,
|
||||
p95: f32,
|
||||
@@ -434,16 +457,20 @@ fn main() -> Result<()> {
|
||||
info!("Training complete. Frozen policy ready for eval.");
|
||||
|
||||
// ---------------------------------------------------------------
|
||||
// Phase 2: COST SWEEP — frozen-policy greedy eval per cost.
|
||||
// Phase 2: 2D SWEEP — frozen-policy greedy eval per (threshold, cost).
|
||||
// Direct analogue of Phase 1d.4's threshold × cost table.
|
||||
// ---------------------------------------------------------------
|
||||
info!("=== Eval phase: {} episodes per cost × {} costs ===",
|
||||
cli.n_eval_episodes, cli.cost_grid.len());
|
||||
info!("=== Eval phase: {} episodes × {} thresholds × {} costs ===",
|
||||
cli.n_eval_episodes, cli.threshold_grid.len(), cli.cost_grid.len());
|
||||
let eval_max_start = (n_total - n_train).saturating_sub(cli.horizon + 1).max(1);
|
||||
let mut bins: Vec<CostBin> = Vec::with_capacity(cli.cost_grid.len());
|
||||
let mut bins: Vec<CostBin> =
|
||||
Vec::with_capacity(cli.cost_grid.len() * cli.threshold_grid.len());
|
||||
for &threshold in &cli.threshold_grid {
|
||||
for &cost in &cli.cost_grid {
|
||||
env.config.cost_per_contract = cost;
|
||||
let mut rewards: Vec<f32> = Vec::with_capacity(cli.n_eval_episodes);
|
||||
let mut win_count = 0_usize;
|
||||
let mut trade_count_total: u64 = 0;
|
||||
for _ in 0..cli.n_eval_episodes {
|
||||
let start_cursor =
|
||||
n_train + (episode_rng.next_u64() as usize) % eval_max_start;
|
||||
@@ -451,6 +478,7 @@ fn main() -> Result<()> {
|
||||
env.reset_at(env_seed, start_cursor);
|
||||
let mut state = EpisodeState::new();
|
||||
let mut terminal_r = 0.0_f32;
|
||||
let mut ep_n_trades = 0_u32;
|
||||
loop {
|
||||
let s_vec = env.state(&state).to_vec();
|
||||
stream.memcpy_htod(&s_vec, &mut single_state_dev)?;
|
||||
@@ -469,7 +497,16 @@ fn main() -> Result<()> {
|
||||
stream.synchronize()?;
|
||||
let q_host = stream.clone_dtoh(&single_q_dev)?;
|
||||
let mut greedy_rng = SmokeRng::new(0); // unused — eps=0
|
||||
let action = epsilon_greedy(&q_host, 0.0, &mut greedy_rng);
|
||||
let action = epsilon_greedy_gated(
|
||||
&q_host,
|
||||
s_vec[1], // alpha_confidence
|
||||
threshold,
|
||||
0.0,
|
||||
&mut greedy_rng,
|
||||
);
|
||||
if action != 0 {
|
||||
ep_n_trades += 1;
|
||||
}
|
||||
match env.step(action, &mut state) {
|
||||
Some((_, reward, done)) => {
|
||||
if done {
|
||||
@@ -481,6 +518,7 @@ fn main() -> Result<()> {
|
||||
}
|
||||
}
|
||||
rewards.push(terminal_r);
|
||||
trade_count_total += ep_n_trades as u64;
|
||||
if terminal_r > 0.0 {
|
||||
win_count += 1;
|
||||
}
|
||||
@@ -494,14 +532,10 @@ fn main() -> Result<()> {
|
||||
/ n;
|
||||
let std = var.sqrt().max(1e-6);
|
||||
let sharpe_per = (mean / std) as f32;
|
||||
// Annualised Sharpe via per-episode Sharpe × sqrt(episodes/year).
|
||||
// Episode duration ≈ horizon × bar_seconds. ES.FUT MBP-10 snapshots
|
||||
// at snapshot_interval=50 ≈ 12 sec/bar on average; 600-bar horizon
|
||||
// ≈ 7200 s ≈ 2 hours. Episodes per year ≈ 252 × 6.5 × 3600 / 7200
|
||||
// ≈ 819. Sharpe_ann ≈ Sharpe_per × sqrt(819) ≈ 28.6.
|
||||
let episodes_per_year = 252.0 * 6.5 * 3600.0 / (cli.horizon as f64 * 12.0);
|
||||
let sharpe_ann = (sharpe_per as f64 * episodes_per_year.sqrt()) as f32;
|
||||
let win_rate = win_count as f32 / cli.n_eval_episodes as f32;
|
||||
let avg_n_trades = trade_count_total as f32 / cli.n_eval_episodes as f32;
|
||||
let mut sorted = rewards.clone();
|
||||
sorted.sort_by(|a, b| a.partial_cmp(b).unwrap_or(std::cmp::Ordering::Equal));
|
||||
let pick = |q: f64| -> f32 {
|
||||
@@ -510,50 +544,72 @@ fn main() -> Result<()> {
|
||||
};
|
||||
let bin = CostBin {
|
||||
cost,
|
||||
threshold,
|
||||
n_episodes: cli.n_eval_episodes,
|
||||
mean_reward: mean as f32,
|
||||
std_reward: std as f32,
|
||||
sharpe_per_episode: sharpe_per,
|
||||
sharpe_annualised: sharpe_ann,
|
||||
win_rate,
|
||||
avg_n_trades,
|
||||
p05: pick(0.05),
|
||||
p50: pick(0.50),
|
||||
p95: pick(0.95),
|
||||
};
|
||||
info!(
|
||||
" cost={:>6.4} mean={:>+9.2} std={:>8.2} Sharpe/ep={:+.3} Sharpe_ann={:+.3} win_rate={:.3}",
|
||||
cost, mean, std, sharpe_per, sharpe_ann, win_rate
|
||||
" τ={:.2} cost={:>6.4} mean={:>+9.2} Sharpe/ep={:+.3} Sharpe_ann={:+.3} win={:.3} trades/ep={:.1}",
|
||||
threshold, cost, mean, sharpe_per, sharpe_ann, win_rate, avg_n_trades
|
||||
);
|
||||
bins.push(bin);
|
||||
}
|
||||
}
|
||||
|
||||
// --- Print table ---
|
||||
info!("");
|
||||
info!("=== Phase E.3 cost-sweep table (vs Phase 1d.4 baseline) ===");
|
||||
info!("=== Phase E.3 2D sweep (threshold × cost, vs Phase 1d.4 baseline) ===");
|
||||
info!(
|
||||
" {:>6} {:>8} {:>9} {:>9} {:>10} {:>11} {:>8}",
|
||||
"cost", "n_ep", "mean_R", "std_R", "Sharpe/ep", "Sharpe_ann", "win_rate"
|
||||
" {:>6} {:>6} {:>9} {:>9} {:>10} {:>11} {:>8} {:>10}",
|
||||
"τ", "cost", "mean_R", "std_R", "Sharpe/ep", "Sharpe_ann", "win_rate", "trades/ep"
|
||||
);
|
||||
for b in &bins {
|
||||
info!(
|
||||
" {:>6.4} {:>8} {:>9.2} {:>9.2} {:>10.4} {:>11.4} {:>8.3}",
|
||||
b.cost, b.n_episodes, b.mean_reward, b.std_reward,
|
||||
b.sharpe_per_episode, b.sharpe_annualised, b.win_rate
|
||||
" {:>6.3} {:>6.4} {:>9.2} {:>9.2} {:>10.4} {:>11.4} {:>8.3} {:>10.2}",
|
||||
b.threshold, b.cost, b.mean_reward, b.std_reward,
|
||||
b.sharpe_per_episode, b.sharpe_annualised, b.win_rate, b.avg_n_trades
|
||||
);
|
||||
}
|
||||
info!("");
|
||||
info!("BEST per-cost (max Sharpe_ann across all τ at each cost):");
|
||||
for &cost in &cli.cost_grid {
|
||||
let best = bins
|
||||
.iter()
|
||||
.filter(|b| (b.cost - cost).abs() < 1e-6)
|
||||
.max_by(|a, b| {
|
||||
a.sharpe_annualised
|
||||
.partial_cmp(&b.sharpe_annualised)
|
||||
.unwrap_or(std::cmp::Ordering::Equal)
|
||||
});
|
||||
if let Some(b) = best {
|
||||
info!(
|
||||
" cost={:>6.4} best τ={:.3} Sharpe_ann={:+.3} win={:.3} trades/ep={:.1}",
|
||||
b.cost, b.threshold, b.sharpe_annualised, b.win_rate, b.avg_n_trades
|
||||
);
|
||||
}
|
||||
}
|
||||
info!("");
|
||||
info!("Phase 1d.4 baseline for comparison: +4.4 annualised at cost=0,");
|
||||
info!(" -4.0 annualised at cost=0.125.");
|
||||
|
||||
// --- Save JSON ---
|
||||
let json = serde_json::json!({
|
||||
"phase": "E.3 Task 23",
|
||||
"phase": "E.3 Task 23 (2D sweep)",
|
||||
"horizon": cli.horizon,
|
||||
"train_frac": cli.train_frac,
|
||||
"n_train_episodes": cli.n_train_episodes,
|
||||
"n_eval_episodes": cli.n_eval_episodes,
|
||||
"train_cost": cli.train_cost,
|
||||
"cost_grid": cli.cost_grid,
|
||||
"threshold_grid": cli.threshold_grid,
|
||||
"bins": bins,
|
||||
});
|
||||
let mut f = File::create(&cli.out_path)?;
|
||||
|
||||
@@ -310,6 +310,24 @@ impl SmokeRng {
|
||||
}
|
||||
}
|
||||
|
||||
/// Phase E.3 confidence-gated ε-greedy. If `alpha_confidence < threshold`,
|
||||
/// force action=0 (Wait) regardless of Q. Otherwise standard ε-greedy.
|
||||
/// Both args are expected in [0, 0.5] (the state's
|
||||
/// `alpha_confidence = |sigmoid(alpha_logit) − 0.5|`, and the controller
|
||||
/// slot 543 which is clamped to [0, 0.5]). Threshold=0 disables the gate.
|
||||
fn epsilon_greedy_gated(
|
||||
q: &[f32],
|
||||
alpha_confidence: f32,
|
||||
threshold: f32,
|
||||
eps: f32,
|
||||
rng: &mut SmokeRng,
|
||||
) -> u8 {
|
||||
if alpha_confidence < threshold {
|
||||
return 0; // Wait
|
||||
}
|
||||
epsilon_greedy(q, eps, rng)
|
||||
}
|
||||
|
||||
/// Picks an action via ε-greedy: with probability `eps` random, otherwise
|
||||
/// argmax over `q`. Returns the action index in `[0, N_ACTIONS)`.
|
||||
fn epsilon_greedy(q: &[f32], eps: f32, rng: &mut SmokeRng) -> u8 {
|
||||
@@ -508,6 +526,12 @@ fn main() -> Result<()> {
|
||||
// --- Training loop ---
|
||||
let mut episode_rng = SmokeRng::new(cli.seed.wrapping_add(0xFEED));
|
||||
let mut action_counts_host = vec![0_i32; N_ACTIONS];
|
||||
// Phase E.3: cached threshold from controller (slot 543). Updated after
|
||||
// each rollout-end controller invocation; used in action selection for
|
||||
// the NEXT episode. Initialised to 0 (no gate) so the first episode's
|
||||
// exploration is unrestricted, letting the controller accumulate enough
|
||||
// observations to drive a meaningful threshold.
|
||||
let mut current_threshold: f32 = 0.0;
|
||||
let mut recent_returns: Vec<f32> = Vec::new();
|
||||
let max_start = n_rows.saturating_sub(cli.horizon + 1).max(1);
|
||||
|
||||
@@ -547,7 +571,13 @@ fn main() -> Result<()> {
|
||||
}
|
||||
stream.synchronize().context("sync after per-step fwd")?;
|
||||
let q_host = stream.clone_dtoh(&single_q_dev).context("dtoh Q")?;
|
||||
let action = epsilon_greedy(&q_host, eps, &mut episode_rng);
|
||||
let action = epsilon_greedy_gated(
|
||||
&q_host,
|
||||
s_vec[1], // alpha_confidence (state index 1)
|
||||
current_threshold,
|
||||
eps,
|
||||
&mut episode_rng,
|
||||
);
|
||||
|
||||
let (_next_state_arr, reward, done) = env.step(action, &mut state)
|
||||
.ok_or_else(|| anyhow::anyhow!("step returned None"))?;
|
||||
@@ -745,6 +775,12 @@ fn main() -> Result<()> {
|
||||
w_ptr,
|
||||
)?;
|
||||
}
|
||||
// Phase E.3: refresh CPU-side threshold cache from device after the
|
||||
// controller update, for the NEXT episode's gating decisions.
|
||||
stream.synchronize().context("sync after controller")?;
|
||||
let isv_now = stream.clone_dtoh(&isv_dev).context("dtoh isv post-controller")?;
|
||||
current_threshold =
|
||||
isv_now[ml::cuda_pipeline::alpha_isv_slots::STACKER_THRESHOLD_INDEX];
|
||||
|
||||
// Target network hard update every K episodes: w_target ← w_online.
|
||||
// Standard DQN stabilizer — V_soft(s') bootstrap reads w_target,
|
||||
|
||||
Reference in New Issue
Block a user