test(dqn): Phase 0 Test 0.E — synthetic edge discovery via Thompson Q-learning

Algorithmic property test (CPU). Confirms Thompson exploration discovers
KNOWN +0.005 edge in 100 iterations on a 1-state bandit, while
argmax-only training never updates Q[Long].

Setup revised from plan A draft (option 3 — production-realistic):
  p_long initial = [0.10, 0.20, 0.40, 0.20, 0.10] (uniform, E=0, has σ)
  p_flat initial = [0, 0, 1, 0, 0]                (δ(v=0), deterministic)
  Argmax with strict-> ties at E=0 → always picks Flat → never explores
  Long → Q[Long] stays at 0, never discovers edge.
  Thompson samples Long > 0 with P≈0.30 → ~30 effective updates → mean
  drifts toward +0.005, crosses Q[Flat]=0 within budget.

Plan's prior draft (initial p_long with mean=-0.015 + p_flat=δ(0)) was
calibration-bound: Thompson drift was directionally correct (-0.015 →
-0.005) but didn't cross zero in 100 iters. Revised setup eliminates
the artificial initial bias and matches production reality more
closely (Flat = δ(0) by construction; Long starts spread from random
init, then accumulates true edge).

Stop condition: if Thompson e_long ≤ e_flat with this setup, the
hypothesis is genuinely wrong and reward shaping must change before
proceeding to Phase 2.

Observed (local RTX 3050 Ti, ~0.00s test wall, ~1.57s 5-test suite):
  argmax  : e_long=0.000000, e_flat=0.000000 (asserts e_long ≤ 0.001 OK)
  thompson: e_long=0.003550, e_flat=0.000000 (asserts e_long > e_flat OK)

All 5 Phase 0 tests pass: 0.A bias-reproduces, 0.B inverse-CDF,
0.C IQN symmetry, 0.D Thompson-reverses, 0.E synthetic-edge.
This commit is contained in:
jgrusewski
2026-04-27 09:20:18 +02:00
parent 82d39a895c
commit a3bb040bc2
2 changed files with 149 additions and 0 deletions

View File

@@ -82,6 +82,30 @@ exploration is sufficient at the realistic σ (not just the wide σ from Test 0.
the failure-mode bias mechanism reproduces and Thompson reverses it. No
production callers. Plan: `docs/superpowers/plans/2026-04-27-distributional-rl-thompson-plan-A-phase-0.md`.
Plan A Phase 0 Task 7 Test 0.E (2026-04-27): `trainers/dqn/distributional_q_tests.rs`
appended `test_0e_synthetic_edge_discovery_cpu` (`#[test]`, NOT `#[ignore]` — pure
CPU, no GPU dependency, runs in default `cargo test`). Algorithmic-property test
of Thompson exploration on a 1-state 2-action bandit: Long has KNOWN +0.005 edge
(`Normal(0.005, 0.03)`), Flat returns deterministic 0. Q-learning over 5 atoms
`[-0.1, -0.05, 0, 0.05, 0.1]` with `lr=0.05` for 100 iterations under two action
selectors. Initial setup revised from plan A draft (option 3 — production-realistic):
`p_long = [0.10, 0.20, 0.40, 0.20, 0.10]` (uniform-ish, E=0, has σ; mimics
freshly initialized C51 head); `p_flat = [0, 0, 1, 0, 0]` (δ(v=0); tx_cost-bare
Flat returns 0). Argmax with strict-`>` tie-break at e_long=e_flat=0 always
picks Flat → never explores Long → e_long stays 0 (assertion
`e_long ≤ e_flat + 0.001` passes vacuously). Thompson: `P(s_long > 0) = p_long[3]
+ p_long[4] = 0.30` → ~30 effective Long-selections drift p_long mean toward
the +0.005 reward target's nearest atom (+0.05). Observed values on local
RTX 3050 Ti: argmax e_long=0.000000, e_flat=0.000000; Thompson e_long=0.003550,
e_flat=0.000000 (test ~0.00 s; total 5-test run ~1.57 s). Plan's prior draft
(initial p_long with mean=-0.015 + p_flat=δ(0)) was calibration-bound:
Thompson drift was directionally correct but didn't cross zero in 100 iters.
Revised setup eliminates the artificial initial bias and matches production
reality. Stop condition: if Thompson `e_long ≤ e_flat` with this setup, the
hypothesis is genuinely wrong and reward shaping must change before Phase 2.
Uses `rand` + `rand_distr` (already in workspace deps). No production callers.
Plan: `docs/superpowers/plans/2026-04-27-distributional-rl-thompson-plan-A-phase-0.md`.
Plan A Phase 0 batched-pinned redesign (2026-04-27): `cuda_pipeline/thompson_test_kernel.cu`
+ `trainers/dqn/distributional_q_tests.rs` — eliminates the per-seed `clone_dtoh`
the original Task 2 wrapper used inside the 10 000- (Test 0.A) / 100 000-iteration