test(dqn): Phase 2 Test 2.C — mag/ord/urg branches unaffected (parity vs Boltzmann ref)
Plan C Phase 2 T8. The plan-prescribed pre-T2 snapshot approach was impossible (T2 had already landed); replacement strategy (ii) from the dispatch brief — behavioral parity vs an analytical Boltzmann reference computed in Rust — is used. Setup forces direction = Long (d=2) deterministically via peaked C51 logits (Long peaked at v=+0.8 atom; other directions at v=-0.5), so the kernel's Hold/Flat → mag_idx=0 short-circuit doesn't mask the magnitude branch's Boltzmann sampling. q_values are crafted with each branch (mag/ord/urg) peaked at a single bin with magnitude 1.0: Mag Q = [0.0, 0.0, 1.0] peak at Full (mag=2) Order Q = [1.0, 0.0, 0.0] peak at Market (ord=0) Urgency Q = [0.0, 1.0, 0.0] peak at urg=1 With q_range=1.0 in all three branches, tau collapses to 1.0 and the analytical Boltzmann probabilities are: P(best) = 1/(1 + 2/e) ≈ 0.5767 P(other) = 1/e/(1 + 2/e) ≈ 0.2117 Tolerance: at batch=8192 the 1-σ Bernoulli noise is ~0.0055 for p≈0.58; ±5% absolute tolerance covers ~9σ. Algorithmic divergence (e.g. an inadvertent strict-argmax substitution) would shift P(best) to 1.0 — trivially detected by the ±5% tolerance. Assertions: - dir_idx == Long for every sample (eval argmax E[Q] over peaked C51) - mag/ord/urg histograms each within ±5% of the Boltzmann reference Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
@@ -1442,3 +1442,187 @@ fn test_2b_production_matches_standalone() {
|
||||
stand_active,
|
||||
);
|
||||
}
|
||||
|
||||
/// Phase 2 Test 2.C — Magnitude/order/urgency picks unaffected by direction
|
||||
/// Thompson (behavioral parity vs hand-computed Boltzmann reference).
|
||||
///
|
||||
/// The plan-prescribed approach (snapshot pre-T2 kernel output) is impossible
|
||||
/// — T2 has already landed (commit 52a2663a2). Replacement strategy (ii)
|
||||
/// from the dispatch brief: assert mag/ord/urg picks against an analytical
|
||||
/// reference computed in Rust from the same Q-values.
|
||||
///
|
||||
/// The mag/ord/urg branches are unchanged across Plan C T2: Boltzmann softmax
|
||||
/// with adaptive tau = max(q_range, floor) where floor = 0.01 (mag) or 0.5
|
||||
/// (ord/urg). In EVAL mode (eps=0) all three branches still take the
|
||||
/// Boltzmann path (NOT strict-argmax — the kernel deliberately keeps
|
||||
/// stochasticity in eval to avoid lock-in).
|
||||
///
|
||||
/// Setup:
|
||||
/// - C51 logits forcing direction = Long (d=2) deterministically. This avoids
|
||||
/// the kernel's Hold/Flat → mag_idx=0 short-circuit which would mask
|
||||
/// the Boltzmann sampling on the magnitude branch.
|
||||
/// - Magnitude Q = [0.0, 0.0, 1.0] — peak at Full (mag=2). q_range=1.0,
|
||||
/// tau=max(1.0, 0.01)=1.0. Boltzmann: P(Full) = 1/(1+2/e) ≈ 0.577,
|
||||
/// P(Small) = P(Half) = e^-1/(1+2/e) ≈ 0.211 each.
|
||||
/// - Order Q = [1.0, 0.0, 0.0] — peak at Market (ord=0). q_range=1.0,
|
||||
/// tau=max(1.0, 0.5)=1.0. Same Boltzmann shape: P(Market) ≈ 0.577.
|
||||
/// - Urgency Q = [0.0, 1.0, 0.0] — peak at urg=1. q_range=1.0, tau=1.0.
|
||||
/// Same Boltzmann shape: P(urg=1) ≈ 0.577.
|
||||
///
|
||||
/// Assertions:
|
||||
/// - dir_idx is Long (2) for every sample (eval argmax E[Q] over the
|
||||
/// peaked C51 logits).
|
||||
/// - mag/ord/urg picks each follow the analytical Boltzmann distribution
|
||||
/// within finite-sample tolerance (±5% on P(best)).
|
||||
#[test]
|
||||
#[ignore]
|
||||
fn test_2c_mag_ord_urg_branches_unaffected() {
|
||||
let stream = make_test_stream();
|
||||
let n_atoms: i32 = 21;
|
||||
let v_min = -1.0_f32;
|
||||
let v_max = 1.0_f32;
|
||||
let delta_z = (v_max - v_min) / (n_atoms as f32 - 1.0);
|
||||
|
||||
// C51 logits: peak Long at v=+0.8 atom (E[Q] ≈ +0.8); other directions
|
||||
// peaked at v=-0.5 (E[Q] ≈ -0.5). Eval argmax E[Q] picks Long.
|
||||
let mut tile = vec![0.0_f32; (PROD_B0 as usize) * (n_atoms as usize)];
|
||||
let long_peak = (((0.8 - v_min) / delta_z).round() as usize)
|
||||
.min((n_atoms - 1) as usize);
|
||||
let other_peak = (((-0.5 - v_min) / delta_z).round() as usize)
|
||||
.min((n_atoms - 1) as usize);
|
||||
tile[0 * n_atoms as usize + other_peak] = 10.0; // Short
|
||||
tile[1 * n_atoms as usize + other_peak] = 10.0; // Hold
|
||||
tile[2 * n_atoms as usize + long_peak] = 10.0; // Long (best)
|
||||
tile[3 * n_atoms as usize + other_peak] = 10.0; // Flat
|
||||
|
||||
let batch: usize = 8192;
|
||||
let mut full_logits = Vec::with_capacity(batch * tile.len());
|
||||
for _ in 0..batch {
|
||||
full_logits.extend_from_slice(&tile);
|
||||
}
|
||||
let support_host = fill_linear_support(batch, v_min, v_max, delta_z);
|
||||
|
||||
// q_values: [direction[4], magnitude[3], order[3], urgency[3]] per sample.
|
||||
let mut q_values_host = vec![0.0_f32; batch * PROD_Q_STRIDE];
|
||||
for i in 0..batch {
|
||||
let base = i * PROD_Q_STRIDE;
|
||||
// Direction Q (unused — direction is decided by C51 Thompson/argmax)
|
||||
// Magnitude Q (b1=3): [0.0, 0.0, 1.0] — peak at Full
|
||||
q_values_host[base + 4 + 0] = 0.0;
|
||||
q_values_host[base + 4 + 1] = 0.0;
|
||||
q_values_host[base + 4 + 2] = 1.0;
|
||||
// Order Q (b2=3): [1.0, 0.0, 0.0] — peak at Market
|
||||
q_values_host[base + 7 + 0] = 1.0;
|
||||
q_values_host[base + 7 + 1] = 0.0;
|
||||
q_values_host[base + 7 + 2] = 0.0;
|
||||
// Urgency Q (b3=3): [0.0, 1.0, 0.0] — peak at urg=1
|
||||
q_values_host[base + 10 + 0] = 0.0;
|
||||
q_values_host[base + 10 + 1] = 1.0;
|
||||
q_values_host[base + 10 + 2] = 0.0;
|
||||
}
|
||||
|
||||
let mut fixture = ProdActionSelectFixture::new(
|
||||
stream,
|
||||
batch,
|
||||
n_atoms,
|
||||
&q_values_host,
|
||||
&full_logits,
|
||||
&support_host,
|
||||
);
|
||||
|
||||
let actions = fixture.launch(/*eval_mode=*/ true, /*timestep=*/ 0);
|
||||
|
||||
// ── Hand-computed Boltzmann reference ────────────────────────────────
|
||||
//
|
||||
// For each branch with peak Q at one bin and zeros elsewhere:
|
||||
// tau = max(q_range, floor)
|
||||
// P(best) = exp(0) / sum_a exp((q_a - q_max) / tau)
|
||||
// = 1.0 / (1.0 + 2 * exp(-1))
|
||||
// = 1.0 / (1.0 + 2/e)
|
||||
// ≈ 0.5767
|
||||
// P(non-best) = exp(-1) / (1 + 2/e) ≈ 0.2117 each
|
||||
//
|
||||
// tau = q_range = 1.0 in all three branches (q_max=1, q_min=0; range
|
||||
// strictly above the 0.01 / 0.5 floors).
|
||||
let p_best_boltzmann = 1.0_f32 / (1.0_f32 + 2.0_f32 * (-1.0_f32).exp());
|
||||
let p_other_boltzmann = (-1.0_f32).exp() / (1.0_f32 + 2.0_f32 * (-1.0_f32).exp());
|
||||
|
||||
// ── Histograms ────────────────────────────────────────────────────────
|
||||
let mut dir_hist = [0_u32; PROD_B0 as usize];
|
||||
let mut mag_hist = [0_u32; PROD_B1 as usize];
|
||||
let mut ord_hist = [0_u32; PROD_B2 as usize];
|
||||
let mut urg_hist = [0_u32; PROD_B3 as usize];
|
||||
for action in &actions {
|
||||
dir_hist[decode_dir(*action) as usize] += 1;
|
||||
mag_hist[decode_mag(*action) as usize] += 1;
|
||||
ord_hist[decode_ord(*action) as usize] += 1;
|
||||
urg_hist[decode_urg(*action) as usize] += 1;
|
||||
}
|
||||
|
||||
let p = |h: u32| h as f32 / batch as f32;
|
||||
println!(
|
||||
"Test 2.C dir hist: {:?} (expect Long=1.0)",
|
||||
dir_hist,
|
||||
);
|
||||
println!(
|
||||
"Test 2.C mag hist: {:?} P(Full)={:.4} P(Small)={:.4} P(Half)={:.4} \
|
||||
(Boltzmann ref: P(best)={:.4}, P(other)={:.4})",
|
||||
mag_hist, p(mag_hist[2]), p(mag_hist[0]), p(mag_hist[1]),
|
||||
p_best_boltzmann, p_other_boltzmann,
|
||||
);
|
||||
println!(
|
||||
"Test 2.C ord hist: {:?} P(Market)={:.4} (Boltzmann ref: {:.4})",
|
||||
ord_hist, p(ord_hist[0]), p_best_boltzmann,
|
||||
);
|
||||
println!(
|
||||
"Test 2.C urg hist: {:?} P(urg=1)={:.4} (Boltzmann ref: {:.4})",
|
||||
urg_hist, p(urg_hist[1]), p_best_boltzmann,
|
||||
);
|
||||
|
||||
// Direction: every sample picks Long (eval argmax E[Q]).
|
||||
assert_eq!(
|
||||
dir_hist[2], batch as u32,
|
||||
"All samples must pick Long (d=2) under the controlled C51 logits — \
|
||||
got dir hist {:?}",
|
||||
dir_hist,
|
||||
);
|
||||
|
||||
// Tolerance: at batch=8192, 1-σ on a Bernoulli proportion is
|
||||
// sqrt(p*(1-p)/n) ≈ 0.0055 for p≈0.58. Use ±5% absolute tolerance
|
||||
// (covers ~9σ — safely above any finite-sample noise).
|
||||
const TOL: f32 = 0.05;
|
||||
|
||||
let p_full = p(mag_hist[2]);
|
||||
let p_small = p(mag_hist[0]);
|
||||
let p_half = p(mag_hist[1]);
|
||||
assert!(
|
||||
(p_full - p_best_boltzmann).abs() < TOL,
|
||||
"Magnitude P(Full)={:.4} deviates from Boltzmann reference {:.4} by > {:.2}",
|
||||
p_full, p_best_boltzmann, TOL,
|
||||
);
|
||||
assert!(
|
||||
(p_small - p_other_boltzmann).abs() < TOL,
|
||||
"Magnitude P(Small)={:.4} deviates from Boltzmann reference {:.4} by > {:.2}",
|
||||
p_small, p_other_boltzmann, TOL,
|
||||
);
|
||||
assert!(
|
||||
(p_half - p_other_boltzmann).abs() < TOL,
|
||||
"Magnitude P(Half)={:.4} deviates from Boltzmann reference {:.4} by > {:.2}",
|
||||
p_half, p_other_boltzmann, TOL,
|
||||
);
|
||||
|
||||
let p_market = p(ord_hist[0]);
|
||||
assert!(
|
||||
(p_market - p_best_boltzmann).abs() < TOL,
|
||||
"Order P(Market)={:.4} deviates from Boltzmann reference {:.4} by > {:.2}",
|
||||
p_market, p_best_boltzmann, TOL,
|
||||
);
|
||||
|
||||
let p_urg1 = p(urg_hist[1]);
|
||||
assert!(
|
||||
(p_urg1 - p_best_boltzmann).abs() < TOL,
|
||||
"Urgency P(urg=1)={:.4} deviates from Boltzmann reference {:.4} by > {:.2}",
|
||||
p_urg1, p_best_boltzmann, TOL,
|
||||
);
|
||||
}
|
||||
|
||||
|
||||
@@ -2119,5 +2119,29 @@ A sanity assertion also verifies P(Long+Short) ≥ 0.20 in both
|
||||
histograms — without this a coincidental shared-mode collapse would
|
||||
produce KS ≈ 0 without actually exercising Thompson behaviour.
|
||||
|
||||
### Test 2.C — Mag/ord/urg branches unaffected (Task T8)
|
||||
|
||||
`crates/ml/src/trainers/dqn/distributional_q_tests.rs::test_2c_mag_ord_urg_branches_unaffected`
|
||||
asserts that the magnitude/order/urgency branches of the production
|
||||
kernel still follow the unchanged Boltzmann softmax with adaptive tau.
|
||||
The plan-prescribed pre-T2 snapshot approach was impossible (T2 had
|
||||
already landed); replacement strategy (ii) — behavioral parity vs an
|
||||
analytical Boltzmann reference — is used.
|
||||
|
||||
Setup forces direction = Long (d=2) deterministically via peaked C51
|
||||
logits (Long peaked at v=+0.8 atom; other directions at v=-0.5), so
|
||||
the kernel's Hold/Flat → mag_idx=0 short-circuit doesn't mask the
|
||||
magnitude branch's Boltzmann sampling. q_values are crafted with each
|
||||
branch (mag/ord/urg) peaked at a single bin with magnitude 1.0; with
|
||||
q_range=1.0 in all three branches, tau collapses to 1.0 and the
|
||||
analytical Boltzmann probabilities are P(best) = 1/(1 + 2/e) ≈ 0.5767
|
||||
and P(other) = 1/e/(1 + 2/e) ≈ 0.2117.
|
||||
|
||||
At batch=8192 the 1-σ Bernoulli noise is ~0.0055 for p≈0.58; ±5%
|
||||
absolute tolerance covers ~9σ — safely above any finite-sample noise.
|
||||
Algorithmic divergence in mag/ord/urg branches (e.g. an inadvertent
|
||||
strict-argmax substitution) would shift P(best) to 1.0, which the
|
||||
±5% tolerance trivially detects.
|
||||
|
||||
|
||||
Plan C T11 follow-up A.1 (2026-04-29): reset `prev_epoch_q_mean` and `adaptive_tau` in `DQNTrainer::reset_for_fold` (`crates/ml/src/trainers/dqn/trainer/mod.rs`). Q-drift kill criterion's prev-baseline carried across fold boundaries because `q_value_history` was cleared but its derived `prev_epoch_q_mean` was not — the ratio at fold-N epoch 0 was computed against fold-(N-1) epoch 5's q_mean. Companion `adaptive_tau` modulation state also reset to `hyperparams.tau` baseline. Independent bug fix; applies regardless of Plan C Thompson outcome. Identified by the Goal-1 q-drift research (researcher report 2026-04-29) when reconciling a52d99613's q_mean=5.003 fold-1 spike (no kill — criterion was added later in 1c917e3cb) against Plan C's fold-0 ep2 trigger.
|
||||
|
||||
Reference in New Issue
Block a user