diff --git a/docs/superpowers/specs/2026-05-03-sp10-unconditional-thompson-selector-isv-temperature.md b/docs/superpowers/specs/2026-05-03-sp10-unconditional-thompson-selector-isv-temperature.md new file mode 100644 index 000000000..a2daec7cf --- /dev/null +++ b/docs/superpowers/specs/2026-05-03-sp10-unconditional-thompson-selector-isv-temperature.md @@ -0,0 +1,234 @@ +# SP10 — Unconditional Thompson Selector + ISV-driven Temperature + +**Date:** 2026-05-03 +**Status:** Draft (user review pending) +**Authors:** session-1777830566225 +**Related pearls:** +- `pearl_thompson_for_distributional_action_selection.md` (existing — selector/target asymmetry) +- `pearl_controller_anchors_isv_driven.md` (anchors must be ISV-driven) +- `pearl_blend_formulas_must_have_permanent_floor.md` (permanent floor pattern) +- `pearl_cold_start_exit_signal_or.md` (OR'd cold-start signals) + +## 1. Overview + +Eliminate argmax-at-eval bias in the direction-branch action selector by +making the selector unconditional Thompson sampling with an ISV-driven +temperature parameter. The temperature is signal-driven from SP9's +existing `intent_eval_divergence` canary: high divergence (eval collapsed) +→ raise τ (more exploration); low divergence (healthy) → lower τ (closer +to argmax-like exploitation). + +This is the next link in the Fix 33→37 chain. Fix 33-37 addressed +training-side controller dynamics; SP10 addresses the eval-side selector +that was producing val-Flat-collapse regardless of controller state. + +## 2. Pathology + +T10 train-multi-seed-khr7c ep2 (commit `8a25b330f`, all SP9 fixes +applied): + +``` +val: trade_count=1 in 214,654 bars + active_frac=0.0 dir_entropy=0.0 sharpe=0.0 +sp9_kelly_warmup [floor=0.25 (engaged) + divergence=500000+ (eval collapsed) + conf [stat=1.0 bhv=0.0 tmp=0.2]] +``` + +Despite Kelly warmup floor=0.25 (allowing Half-mag exposure), Kelly +controllers all healthy, IQN warnings absent — the model picks ONE +direction (Hold) at every val bar. dir_entropy=0 means deterministic +single-action over the whole 214k window. + +Root cause: `experience_action_select` kernel uses `argmax(E[Q])` at +eval (line 1208). With Hold's E[Q] ≈ 0 (no position cost) and +Short/Long's E[Q] = ε (small edge minus tx costs), argmax wins Hold +deterministically every bar. Thompson sampling would let positive-Q +directions win proportional to their posterior overlap with Hold. + +The `pearl_thompson_for_distributional_action_selection` was about +selector/target asymmetry in the **Bellman target** (DDQN target Q +computation). It does NOT prohibit Thompson at the rollout selector; +only at the target. SP10 honors that distinction. + +## 3. Architecture + +### 3.1 Kernel change + +Replace the `if (eval_mode)` branch in `experience_action_select` with +unconditional Thompson sampling, modulated by an ISV-driven temperature: + +```cuda +// Read ISV-driven temperature +float temp = isv_signals[EVAL_THOMPSON_TEMP_INDEX]; + +// Per-direction Thompson sample with temperature blend +float best_q_eff = -1e30f; +int best_d = 0; +for (int d = 0; d < b0_size; d++) { + softmax_c51_inline(logits_i + d * n_atoms, n_atoms, probs_d); + compute_atom_values_inline(per_sample_support, atom_positions, + i, d, b0_size, n_atoms, atom_vals_d); + float u = philox_uniform(i, timestep, rng_ctr++); + float q_sample = sample_c51_inverse_cdf(probs_d, atom_vals_d, n_atoms, u); + + // Temperature blend: τ=0 → argmax of E[Q]; τ=1 → pure Thompson; + // τ>1 → exaggerated exploration (corrective during collapse). + float q_eff = e_dir[d] + temp * (q_sample - e_dir[d]); + + if (q_eff > best_q_eff) { best_q_eff = q_eff; best_d = d; } +} +dir_idx = best_d; +``` + +The `if (eval_mode)` branch and its argmax body are deleted entirely. +No conditional path remains — one selector for both training and eval. + +### 3.2 Temperature formula (ISV producer) + +Extended from SP9's existing `intent_eval_divergence_compute_kernel`. The +existing kernel already computes `divergence`. Add a second scratch write: + +```cuda +// Existing SP9 output: +scratch_buf[scratch_div_idx] = divergence; + +// SP10 addition: temperature output, using same divergence + target +float div_target = isv_signals[KELLY_DIVERGENCE_TARGET_INDEX]; // = 2.0 (Invariant-1) +float temp = clamp(divergence / max(div_target, EPS_DIV), + THOMPSON_TEMP_MIN, THOMPSON_TEMP_MAX); +scratch_buf[scratch_temp_idx] = temp; +``` + +Where: +- `THOMPSON_TEMP_MIN = 0.5f` (Invariant 1: permanent stochasticity floor per + `pearl_blend_formulas_must_have_permanent_floor`; never fully deterministic) +- `THOMPSON_TEMP_MAX = 2.0f` (Invariant 1: cap on exploration boost; prevents + pathological all-uniform sampling) +- `EPS_DIV = 1e-6f` (Invariant 1: divide-by-zero protection) + +Behavior: +- Healthy (`divergence ≈ target = 2.0`): `temp ≈ 1.0` → pure Thompson +- Collapsed (`divergence ≫ target`, e.g., 1e6 sentinel): `temp = 2.0` → exploration boost +- Healthy and converging (`divergence < target`): `temp = 0.5` → closer to argmax behavior, but never fully deterministic + +The MIN_TEMP=0.5 floor is the permanent-stochasticity guarantee per +`pearl_blend_formulas_must_have_permanent_floor`. Eval action selection +ALWAYS has at least 50% Thompson contribution. + +### 3.3 ISV slot allocation + +After SP9's last slot at 338: + +| Slot | Purpose | Reset semantics | +|---|---|---| +| `EVAL_THOMPSON_TEMP_INDEX = 339` | Action selector temperature | FoldReset (sentinel 0, Pearl A bootstrap to MIN_TEMP) | + +Bump `SP5_SLOT_END = 340` and `ISV_TOTAL_DIM = 340`. + +Add 1 new scratch slot for the temperature output. + +### 3.4 Producer kernel — extend or new? + +**Decision: extend existing `intent_eval_divergence_compute_kernel`** rather +than create a separate producer. Reasons: + +1. The temperature is a deterministic function of `divergence` (no new + inputs needed) +2. Both outputs come from the same compute (cheap to add one extra write) +3. Atomic commit per `feedback_no_partial_refactor` — slot + kernel + extension + consumer migration all together +4. Matches `pearl_engagement_rate_self_correction` pattern (one signal, + multiple consumers) + +The kernel signature gains 2 new parameters (scratch_temp_idx and +divergence_target_isv_index) but otherwise unchanged. + +## 4. Pearl amendments + +### 4.1 `pearl_thompson_for_distributional_action_selection.md` + +Update wording to clarify the selector/target distinction: +- BEFORE: "Thompson at training, argmax at eval" +- AFTER: "Thompson at all times for the rollout selector; argmax ONLY for the + Bellman target Q computation (DDQN target). Temperature on the Thompson + sample is ISV-driven from intent_eval_divergence per + pearl_controller_anchors_isv_driven." + +### 4.2 No new pearl + +The principle here is captured by combining existing pearls: +- `pearl_thompson_for_distributional_action_selection` (amended) — selector vs target +- `pearl_controller_anchors_isv_driven` — the temperature is ISV-driven +- `pearl_blend_formulas_must_have_permanent_floor` — MIN_TEMP=0.5 floor +- `pearl_engagement_rate_self_correction` — temperature self-corrects via divergence + +## 5. Test changes + +`crates/ml/src/cuda_pipeline/mod.rs:1242` — `test_eval_action_select_eval_argmax_picks_best`: + +- **Rename** to `test_eval_action_select_thompson_picks_proportionally` +- **Update** assertion: instead of "best direction always wins", assert + "best direction wins with probability ≥ posterior weight of best + direction" (allowing the temperature blend's stochasticity) +- **Histogram check**: with τ=1.0 and a clearly-best direction (large + Q gap), best wins ≥ 80% of samples; with smaller gap, ≥ 50% + +Existing comment at `mod.rs:1370` ("Thompson-eval histogram: short=... +long=... (expected: P(Long)=1.0)") suggests prior preparation for this +case — test framework already aware of Thompson-eval semantics. + +## 6. Implementation phases (atomic per feedback_no_partial_refactor) + +Single commit: + +1. Allocate 1 new ISV slot (EVAL_THOMPSON_TEMP_INDEX=339), bump + SP5_SLOT_END and ISV_TOTAL_DIM +2. Extend `intent_eval_divergence_compute_kernel.cu` with temperature + output (~10 LOC addition + signature change) +3. Update launcher in `gpu_dqn_trainer.rs` for new kernel signature +4. Modify `experience_action_select` kernel: delete `if (eval_mode)` argmax + branch, replace Thompson branch with temperature-blended version, add + `EVAL_THOMPSON_TEMP_INDEX` ISV read at top of direction-selection block +5. Update FoldReset registry: 1 new slot entry, 1 new dispatch arm +6. Update test `test_eval_action_select_eval_argmax_picks_best` → + `test_eval_action_select_thompson_picks_proportionally` +7. Update pearl `pearl_thompson_for_distributional_action_selection.md` + in MEMORY (selector/target wording) +8. Audit doc Fix 38 entry + +Total estimated: ~300-500 LOC across ~6 files. ~45-60 min focused work. + +## 7. Validation criteria + +After SP10 lands: +- Smoke (5-epoch L40S `multi_fold_convergence`): + - val_active_frac > 0.10 with τ ≥ 0.5 (always some action variety) + - dir_entropy at eval > 0.3 (multi-direction selection) + - HEALTH_DIAG sp10 line shows τ evolving with divergence +- T10 50-epoch: + - val_trade_count > 1000 in 214k-bar window (was 1) + - val_active_frac > 0.20 across folds + - sharpe positive net of fold transitions + +## 8. Risks and open questions + +- **Risk 1**: Pure Thompson eval introduces PnL variance — short windows + show more sharpe noise. Mitigation: τ floor at 0.5, not full Thompson; + diagnostic is multi-fold mean sharpe, not single-fold spot value. +- **Risk 2**: Production trading reproducibility — Philox RNG + state + + timestep makes Thompson deterministic given seed. Same model + same + seed = same actions. Reproducibility is preserved at the seed level. +- **Risk 3**: The pearl amendment may conflict with intent at the time it + was written. Per `feedback_trust_code_not_docs`, the smoke evidence + trumps the original wording; pearl gets updated to reflect reality. + +## 9. Followups (NOT in this commit) + +- Adaptive temperature beyond the divergence canary (e.g., per-state + uncertainty from Q variance, regime-dependent τ) +- Apply same Thompson selector logic to mag/ord/urg branches if their + argmax-at-eval pathways are found to bias similarly (currently mag + uses Boltzmann softmax, ord/urg also stochastic — likely fine) +- Bellman target Q computation review: confirm DDQN target argmax stays + argmax (no SP10 change there)