From cc55c8a25c104263738f4e9411fdf30dc5eb0cad Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Wed, 29 Apr 2026 18:20:59 +0200 Subject: [PATCH] =?UTF-8?q?test(dqn):=20Phase=202=20Test=202.C=20=E2=80=94?= =?UTF-8?q?=20mag/ord/urg=20branches=20unaffected=20(parity=20vs=20Boltzma?= =?UTF-8?q?nn=20ref)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Plan C Phase 2 T8. The plan-prescribed pre-T2 snapshot approach was impossible (T2 had already landed); replacement strategy (ii) from the dispatch brief — behavioral parity vs an analytical Boltzmann reference computed in Rust — is used. Setup forces direction = Long (d=2) deterministically via peaked C51 logits (Long peaked at v=+0.8 atom; other directions at v=-0.5), so the kernel's Hold/Flat → mag_idx=0 short-circuit doesn't mask the magnitude branch's Boltzmann sampling. q_values are crafted with each branch (mag/ord/urg) peaked at a single bin with magnitude 1.0: Mag Q = [0.0, 0.0, 1.0] peak at Full (mag=2) Order Q = [1.0, 0.0, 0.0] peak at Market (ord=0) Urgency Q = [0.0, 1.0, 0.0] peak at urg=1 With q_range=1.0 in all three branches, tau collapses to 1.0 and the analytical Boltzmann probabilities are: P(best) = 1/(1 + 2/e) ≈ 0.5767 P(other) = 1/e/(1 + 2/e) ≈ 0.2117 Tolerance: at batch=8192 the 1-σ Bernoulli noise is ~0.0055 for p≈0.58; ±5% absolute tolerance covers ~9σ. Algorithmic divergence (e.g. an inadvertent strict-argmax substitution) would shift P(best) to 1.0 — trivially detected by the ±5% tolerance. Assertions: - dir_idx == Long for every sample (eval argmax E[Q] over peaked C51) - mag/ord/urg histograms each within ±5% of the Boltzmann reference Co-Authored-By: Claude Opus 4.7 (1M context) --- .../trainers/dqn/distributional_q_tests.rs | 184 ++++++++++++++++++ docs/dqn-wire-up-audit.md | 24 +++ 2 files changed, 208 insertions(+) diff --git a/crates/ml/src/trainers/dqn/distributional_q_tests.rs b/crates/ml/src/trainers/dqn/distributional_q_tests.rs index 58f11968b..c640a652c 100644 --- a/crates/ml/src/trainers/dqn/distributional_q_tests.rs +++ b/crates/ml/src/trainers/dqn/distributional_q_tests.rs @@ -1442,3 +1442,187 @@ fn test_2b_production_matches_standalone() { stand_active, ); } + +/// Phase 2 Test 2.C — Magnitude/order/urgency picks unaffected by direction +/// Thompson (behavioral parity vs hand-computed Boltzmann reference). +/// +/// The plan-prescribed approach (snapshot pre-T2 kernel output) is impossible +/// — T2 has already landed (commit 52a2663a2). Replacement strategy (ii) +/// from the dispatch brief: assert mag/ord/urg picks against an analytical +/// reference computed in Rust from the same Q-values. +/// +/// The mag/ord/urg branches are unchanged across Plan C T2: Boltzmann softmax +/// with adaptive tau = max(q_range, floor) where floor = 0.01 (mag) or 0.5 +/// (ord/urg). In EVAL mode (eps=0) all three branches still take the +/// Boltzmann path (NOT strict-argmax — the kernel deliberately keeps +/// stochasticity in eval to avoid lock-in). +/// +/// Setup: +/// - C51 logits forcing direction = Long (d=2) deterministically. This avoids +/// the kernel's Hold/Flat → mag_idx=0 short-circuit which would mask +/// the Boltzmann sampling on the magnitude branch. +/// - Magnitude Q = [0.0, 0.0, 1.0] — peak at Full (mag=2). q_range=1.0, +/// tau=max(1.0, 0.01)=1.0. Boltzmann: P(Full) = 1/(1+2/e) ≈ 0.577, +/// P(Small) = P(Half) = e^-1/(1+2/e) ≈ 0.211 each. +/// - Order Q = [1.0, 0.0, 0.0] — peak at Market (ord=0). q_range=1.0, +/// tau=max(1.0, 0.5)=1.0. Same Boltzmann shape: P(Market) ≈ 0.577. +/// - Urgency Q = [0.0, 1.0, 0.0] — peak at urg=1. q_range=1.0, tau=1.0. +/// Same Boltzmann shape: P(urg=1) ≈ 0.577. +/// +/// Assertions: +/// - dir_idx is Long (2) for every sample (eval argmax E[Q] over the +/// peaked C51 logits). +/// - mag/ord/urg picks each follow the analytical Boltzmann distribution +/// within finite-sample tolerance (±5% on P(best)). +#[test] +#[ignore] +fn test_2c_mag_ord_urg_branches_unaffected() { + let stream = make_test_stream(); + let n_atoms: i32 = 21; + let v_min = -1.0_f32; + let v_max = 1.0_f32; + let delta_z = (v_max - v_min) / (n_atoms as f32 - 1.0); + + // C51 logits: peak Long at v=+0.8 atom (E[Q] ≈ +0.8); other directions + // peaked at v=-0.5 (E[Q] ≈ -0.5). Eval argmax E[Q] picks Long. + let mut tile = vec![0.0_f32; (PROD_B0 as usize) * (n_atoms as usize)]; + let long_peak = (((0.8 - v_min) / delta_z).round() as usize) + .min((n_atoms - 1) as usize); + let other_peak = (((-0.5 - v_min) / delta_z).round() as usize) + .min((n_atoms - 1) as usize); + tile[0 * n_atoms as usize + other_peak] = 10.0; // Short + tile[1 * n_atoms as usize + other_peak] = 10.0; // Hold + tile[2 * n_atoms as usize + long_peak] = 10.0; // Long (best) + tile[3 * n_atoms as usize + other_peak] = 10.0; // Flat + + let batch: usize = 8192; + let mut full_logits = Vec::with_capacity(batch * tile.len()); + for _ in 0..batch { + full_logits.extend_from_slice(&tile); + } + let support_host = fill_linear_support(batch, v_min, v_max, delta_z); + + // q_values: [direction[4], magnitude[3], order[3], urgency[3]] per sample. + let mut q_values_host = vec![0.0_f32; batch * PROD_Q_STRIDE]; + for i in 0..batch { + let base = i * PROD_Q_STRIDE; + // Direction Q (unused — direction is decided by C51 Thompson/argmax) + // Magnitude Q (b1=3): [0.0, 0.0, 1.0] — peak at Full + q_values_host[base + 4 + 0] = 0.0; + q_values_host[base + 4 + 1] = 0.0; + q_values_host[base + 4 + 2] = 1.0; + // Order Q (b2=3): [1.0, 0.0, 0.0] — peak at Market + q_values_host[base + 7 + 0] = 1.0; + q_values_host[base + 7 + 1] = 0.0; + q_values_host[base + 7 + 2] = 0.0; + // Urgency Q (b3=3): [0.0, 1.0, 0.0] — peak at urg=1 + q_values_host[base + 10 + 0] = 0.0; + q_values_host[base + 10 + 1] = 1.0; + q_values_host[base + 10 + 2] = 0.0; + } + + let mut fixture = ProdActionSelectFixture::new( + stream, + batch, + n_atoms, + &q_values_host, + &full_logits, + &support_host, + ); + + let actions = fixture.launch(/*eval_mode=*/ true, /*timestep=*/ 0); + + // ── Hand-computed Boltzmann reference ──────────────────────────────── + // + // For each branch with peak Q at one bin and zeros elsewhere: + // tau = max(q_range, floor) + // P(best) = exp(0) / sum_a exp((q_a - q_max) / tau) + // = 1.0 / (1.0 + 2 * exp(-1)) + // = 1.0 / (1.0 + 2/e) + // ≈ 0.5767 + // P(non-best) = exp(-1) / (1 + 2/e) ≈ 0.2117 each + // + // tau = q_range = 1.0 in all three branches (q_max=1, q_min=0; range + // strictly above the 0.01 / 0.5 floors). + let p_best_boltzmann = 1.0_f32 / (1.0_f32 + 2.0_f32 * (-1.0_f32).exp()); + let p_other_boltzmann = (-1.0_f32).exp() / (1.0_f32 + 2.0_f32 * (-1.0_f32).exp()); + + // ── Histograms ──────────────────────────────────────────────────────── + let mut dir_hist = [0_u32; PROD_B0 as usize]; + let mut mag_hist = [0_u32; PROD_B1 as usize]; + let mut ord_hist = [0_u32; PROD_B2 as usize]; + let mut urg_hist = [0_u32; PROD_B3 as usize]; + for action in &actions { + dir_hist[decode_dir(*action) as usize] += 1; + mag_hist[decode_mag(*action) as usize] += 1; + ord_hist[decode_ord(*action) as usize] += 1; + urg_hist[decode_urg(*action) as usize] += 1; + } + + let p = |h: u32| h as f32 / batch as f32; + println!( + "Test 2.C dir hist: {:?} (expect Long=1.0)", + dir_hist, + ); + println!( + "Test 2.C mag hist: {:?} P(Full)={:.4} P(Small)={:.4} P(Half)={:.4} \ + (Boltzmann ref: P(best)={:.4}, P(other)={:.4})", + mag_hist, p(mag_hist[2]), p(mag_hist[0]), p(mag_hist[1]), + p_best_boltzmann, p_other_boltzmann, + ); + println!( + "Test 2.C ord hist: {:?} P(Market)={:.4} (Boltzmann ref: {:.4})", + ord_hist, p(ord_hist[0]), p_best_boltzmann, + ); + println!( + "Test 2.C urg hist: {:?} P(urg=1)={:.4} (Boltzmann ref: {:.4})", + urg_hist, p(urg_hist[1]), p_best_boltzmann, + ); + + // Direction: every sample picks Long (eval argmax E[Q]). + assert_eq!( + dir_hist[2], batch as u32, + "All samples must pick Long (d=2) under the controlled C51 logits — \ + got dir hist {:?}", + dir_hist, + ); + + // Tolerance: at batch=8192, 1-σ on a Bernoulli proportion is + // sqrt(p*(1-p)/n) ≈ 0.0055 for p≈0.58. Use ±5% absolute tolerance + // (covers ~9σ — safely above any finite-sample noise). + const TOL: f32 = 0.05; + + let p_full = p(mag_hist[2]); + let p_small = p(mag_hist[0]); + let p_half = p(mag_hist[1]); + assert!( + (p_full - p_best_boltzmann).abs() < TOL, + "Magnitude P(Full)={:.4} deviates from Boltzmann reference {:.4} by > {:.2}", + p_full, p_best_boltzmann, TOL, + ); + assert!( + (p_small - p_other_boltzmann).abs() < TOL, + "Magnitude P(Small)={:.4} deviates from Boltzmann reference {:.4} by > {:.2}", + p_small, p_other_boltzmann, TOL, + ); + assert!( + (p_half - p_other_boltzmann).abs() < TOL, + "Magnitude P(Half)={:.4} deviates from Boltzmann reference {:.4} by > {:.2}", + p_half, p_other_boltzmann, TOL, + ); + + let p_market = p(ord_hist[0]); + assert!( + (p_market - p_best_boltzmann).abs() < TOL, + "Order P(Market)={:.4} deviates from Boltzmann reference {:.4} by > {:.2}", + p_market, p_best_boltzmann, TOL, + ); + + let p_urg1 = p(urg_hist[1]); + assert!( + (p_urg1 - p_best_boltzmann).abs() < TOL, + "Urgency P(urg=1)={:.4} deviates from Boltzmann reference {:.4} by > {:.2}", + p_urg1, p_best_boltzmann, TOL, + ); +} + diff --git a/docs/dqn-wire-up-audit.md b/docs/dqn-wire-up-audit.md index 6d55fa08b..83f29cd58 100644 --- a/docs/dqn-wire-up-audit.md +++ b/docs/dqn-wire-up-audit.md @@ -2119,5 +2119,29 @@ A sanity assertion also verifies P(Long+Short) ≥ 0.20 in both histograms — without this a coincidental shared-mode collapse would produce KS ≈ 0 without actually exercising Thompson behaviour. +### Test 2.C — Mag/ord/urg branches unaffected (Task T8) + +`crates/ml/src/trainers/dqn/distributional_q_tests.rs::test_2c_mag_ord_urg_branches_unaffected` +asserts that the magnitude/order/urgency branches of the production +kernel still follow the unchanged Boltzmann softmax with adaptive tau. +The plan-prescribed pre-T2 snapshot approach was impossible (T2 had +already landed); replacement strategy (ii) — behavioral parity vs an +analytical Boltzmann reference — is used. + +Setup forces direction = Long (d=2) deterministically via peaked C51 +logits (Long peaked at v=+0.8 atom; other directions at v=-0.5), so +the kernel's Hold/Flat → mag_idx=0 short-circuit doesn't mask the +magnitude branch's Boltzmann sampling. q_values are crafted with each +branch (mag/ord/urg) peaked at a single bin with magnitude 1.0; with +q_range=1.0 in all three branches, tau collapses to 1.0 and the +analytical Boltzmann probabilities are P(best) = 1/(1 + 2/e) ≈ 0.5767 +and P(other) = 1/e/(1 + 2/e) ≈ 0.2117. + +At batch=8192 the 1-σ Bernoulli noise is ~0.0055 for p≈0.58; ±5% +absolute tolerance covers ~9σ — safely above any finite-sample noise. +Algorithmic divergence in mag/ord/urg branches (e.g. an inadvertent +strict-argmax substitution) would shift P(best) to 1.0, which the +±5% tolerance trivially detects. + Plan C T11 follow-up A.1 (2026-04-29): reset `prev_epoch_q_mean` and `adaptive_tau` in `DQNTrainer::reset_for_fold` (`crates/ml/src/trainers/dqn/trainer/mod.rs`). Q-drift kill criterion's prev-baseline carried across fold boundaries because `q_value_history` was cleared but its derived `prev_epoch_q_mean` was not — the ratio at fold-N epoch 0 was computed against fold-(N-1) epoch 5's q_mean. Companion `adaptive_tau` modulation state also reset to `hyperparams.tau` baseline. Independent bug fix; applies regardless of Plan C Thompson outcome. Identified by the Goal-1 q-drift research (researcher report 2026-04-29) when reconciling a52d99613's q_mean=5.003 fold-1 spike (no kill — criterion was added later in 1c917e3cb) against Plan C's fold-0 ep2 trigger.