test(dqn): Phase 0 Test 0.E — synthetic edge discovery via Thompson Q-learning
Algorithmic property test (CPU). Confirms Thompson exploration discovers KNOWN +0.005 edge in 100 iterations on a 1-state bandit, while argmax-only training never updates Q[Long]. Setup revised from plan A draft (option 3 — production-realistic): p_long initial = [0.10, 0.20, 0.40, 0.20, 0.10] (uniform, E=0, has σ) p_flat initial = [0, 0, 1, 0, 0] (δ(v=0), deterministic) Argmax with strict-> ties at E=0 → always picks Flat → never explores Long → Q[Long] stays at 0, never discovers edge. Thompson samples Long > 0 with P≈0.30 → ~30 effective updates → mean drifts toward +0.005, crosses Q[Flat]=0 within budget. Plan's prior draft (initial p_long with mean=-0.015 + p_flat=δ(0)) was calibration-bound: Thompson drift was directionally correct (-0.015 → -0.005) but didn't cross zero in 100 iters. Revised setup eliminates the artificial initial bias and matches production reality more closely (Flat = δ(0) by construction; Long starts spread from random init, then accumulates true edge). Stop condition: if Thompson e_long ≤ e_flat with this setup, the hypothesis is genuinely wrong and reward shaping must change before proceeding to Phase 2. Observed (local RTX 3050 Ti, ~0.00s test wall, ~1.57s 5-test suite): argmax : e_long=0.000000, e_flat=0.000000 (asserts e_long ≤ 0.001 OK) thompson: e_long=0.003550, e_flat=0.000000 (asserts e_long > e_flat OK) All 5 Phase 0 tests pass: 0.A bias-reproduces, 0.B inverse-CDF, 0.C IQN symmetry, 0.D Thompson-reverses, 0.E synthetic-edge.
This commit is contained in:
@@ -82,6 +82,30 @@ exploration is sufficient at the realistic σ (not just the wide σ from Test 0.
|
||||
the failure-mode bias mechanism reproduces and Thompson reverses it. No
|
||||
production callers. Plan: `docs/superpowers/plans/2026-04-27-distributional-rl-thompson-plan-A-phase-0.md`.
|
||||
|
||||
Plan A Phase 0 Task 7 Test 0.E (2026-04-27): `trainers/dqn/distributional_q_tests.rs` —
|
||||
appended `test_0e_synthetic_edge_discovery_cpu` (`#[test]`, NOT `#[ignore]` — pure
|
||||
CPU, no GPU dependency, runs in default `cargo test`). Algorithmic-property test
|
||||
of Thompson exploration on a 1-state 2-action bandit: Long has KNOWN +0.005 edge
|
||||
(`Normal(0.005, 0.03)`), Flat returns deterministic 0. Q-learning over 5 atoms
|
||||
`[-0.1, -0.05, 0, 0.05, 0.1]` with `lr=0.05` for 100 iterations under two action
|
||||
selectors. Initial setup revised from plan A draft (option 3 — production-realistic):
|
||||
`p_long = [0.10, 0.20, 0.40, 0.20, 0.10]` (uniform-ish, E=0, has σ; mimics
|
||||
freshly initialized C51 head); `p_flat = [0, 0, 1, 0, 0]` (δ(v=0); tx_cost-bare
|
||||
Flat returns 0). Argmax with strict-`>` tie-break at e_long=e_flat=0 always
|
||||
picks Flat → never explores Long → e_long stays 0 (assertion
|
||||
`e_long ≤ e_flat + 0.001` passes vacuously). Thompson: `P(s_long > 0) = p_long[3]
|
||||
+ p_long[4] = 0.30` → ~30 effective Long-selections drift p_long mean toward
|
||||
the +0.005 reward target's nearest atom (+0.05). Observed values on local
|
||||
RTX 3050 Ti: argmax e_long=0.000000, e_flat=0.000000; Thompson e_long=0.003550,
|
||||
e_flat=0.000000 (test ~0.00 s; total 5-test run ~1.57 s). Plan's prior draft
|
||||
(initial p_long with mean=-0.015 + p_flat=δ(0)) was calibration-bound:
|
||||
Thompson drift was directionally correct but didn't cross zero in 100 iters.
|
||||
Revised setup eliminates the artificial initial bias and matches production
|
||||
reality. Stop condition: if Thompson `e_long ≤ e_flat` with this setup, the
|
||||
hypothesis is genuinely wrong and reward shaping must change before Phase 2.
|
||||
Uses `rand` + `rand_distr` (already in workspace deps). No production callers.
|
||||
Plan: `docs/superpowers/plans/2026-04-27-distributional-rl-thompson-plan-A-phase-0.md`.
|
||||
|
||||
Plan A Phase 0 batched-pinned redesign (2026-04-27): `cuda_pipeline/thompson_test_kernel.cu`
|
||||
+ `trainers/dqn/distributional_q_tests.rs` — eliminates the per-seed `clone_dtoh`
|
||||
the original Task 2 wrapper used inside the 10 000- (Test 0.A) / 100 000-iteration
|
||||
|
||||
Reference in New Issue
Block a user