spec(sp10): unconditional Thompson selector + ISV-driven temperature
Brainstorm spec for SP10. Resolves the val-Flat-collapse pathology that persisted through Fix 33-37: the eval-time argmax in experience_action_ select picks Hold deterministically every val bar (dir_entropy=0, trade_count=1 in 214k bars) regardless of controller state. Architecture: - Delete `if (eval_mode)` argmax branch in direction-selector kernel - Use temperature-blended Thompson: q_eff = E[Q] + τ × (Thompson - E[Q]) - τ = clamp(intent_eval_divergence / divergence_target, 0.5, 2.0) - τ self-corrects: collapse → high τ; healthy → low τ; permanent 0.5 floor - Reuses SP9's intent_eval_divergence_compute_kernel (extended, not new) Per pearl_controller_anchors_isv_driven: τ is ISV-driven, no hardcoded constants beyond Invariant 1 numerical anchors (clamp range). Per pearl_blend_formulas_must_have_permanent_floor: MIN_TEMP=0.5 ensures eval ALWAYS has stochasticity. The pearl_thompson_for_distributional_action_selection was about Bellman TARGET argmax (selector/target symmetry). It does NOT prohibit Thompson at the rollout selector. SP10 amends the pearl to clarify. Scope: 1 ISV slot, kernel modification (no new kernel — extend SP9's producer), 1 consumer kernel rewrite, test update, pearl amendment, audit doc Fix 38. ~300-500 LOC, single atomic commit per feedback_no_partial_refactor.
This commit is contained in:
jgrusewski
@@ -0,0 +1,234 @@
|
||||
# SP10 — Unconditional Thompson Selector + ISV-driven Temperature
|
||||
|
||||
**Date:** 2026-05-03
|
||||
**Status:** Draft (user review pending)
|
||||
**Authors:** session-1777830566225
|
||||
**Related pearls:**
|
||||
- `pearl_thompson_for_distributional_action_selection.md` (existing — selector/target asymmetry)
|
||||
- `pearl_controller_anchors_isv_driven.md` (anchors must be ISV-driven)
|
||||
- `pearl_blend_formulas_must_have_permanent_floor.md` (permanent floor pattern)
|
||||
- `pearl_cold_start_exit_signal_or.md` (OR'd cold-start signals)
|
||||
|
||||
## 1. Overview
|
||||
|
||||
Eliminate argmax-at-eval bias in the direction-branch action selector by
|
||||
making the selector unconditional Thompson sampling with an ISV-driven
|
||||
temperature parameter. The temperature is signal-driven from SP9's
|
||||
existing `intent_eval_divergence` canary: high divergence (eval collapsed)
|
||||
→ raise τ (more exploration); low divergence (healthy) → lower τ (closer
|
||||
to argmax-like exploitation).
|
||||
|
||||
This is the next link in the Fix 33→37 chain. Fix 33-37 addressed
|
||||
training-side controller dynamics; SP10 addresses the eval-side selector
|
||||
that was producing val-Flat-collapse regardless of controller state.
|
||||
|
||||
## 2. Pathology
|
||||
|
||||
T10 train-multi-seed-khr7c ep2 (commit `8a25b330f`, all SP9 fixes
|
||||
applied):
|
||||
|
||||
```
|
||||
val: trade_count=1 in 214,654 bars
|
||||
active_frac=0.0 dir_entropy=0.0 sharpe=0.0
|
||||
sp9_kelly_warmup [floor=0.25 (engaged)
|
||||
divergence=500000+ (eval collapsed)
|
||||
conf [stat=1.0 bhv=0.0 tmp=0.2]]
|
||||
```
|
||||
|
||||
Despite Kelly warmup floor=0.25 (allowing Half-mag exposure), Kelly
|
||||
controllers all healthy, IQN warnings absent — the model picks ONE
|
||||
direction (Hold) at every val bar. dir_entropy=0 means deterministic
|
||||
single-action over the whole 214k window.
|
||||
|
||||
Root cause: `experience_action_select` kernel uses `argmax(E[Q])` at
|
||||
eval (line 1208). With Hold's E[Q] ≈ 0 (no position cost) and
|
||||
Short/Long's E[Q] = ε (small edge minus tx costs), argmax wins Hold
|
||||
deterministically every bar. Thompson sampling would let positive-Q
|
||||
directions win proportional to their posterior overlap with Hold.
|
||||
|
||||
The `pearl_thompson_for_distributional_action_selection` was about
|
||||
selector/target asymmetry in the **Bellman target** (DDQN target Q
|
||||
computation). It does NOT prohibit Thompson at the rollout selector;
|
||||
only at the target. SP10 honors that distinction.
|
||||
|
||||
## 3. Architecture
|
||||
|
||||
### 3.1 Kernel change
|
||||
|
||||
Replace the `if (eval_mode)` branch in `experience_action_select` with
|
||||
unconditional Thompson sampling, modulated by an ISV-driven temperature:
|
||||
|
||||
```cuda
|
||||
// Read ISV-driven temperature
|
||||
float temp = isv_signals[EVAL_THOMPSON_TEMP_INDEX];
|
||||
|
||||
// Per-direction Thompson sample with temperature blend
|
||||
float best_q_eff = -1e30f;
|
||||
int best_d = 0;
|
||||
for (int d = 0; d < b0_size; d++) {
|
||||
softmax_c51_inline(logits_i + d * n_atoms, n_atoms, probs_d);
|
||||
compute_atom_values_inline(per_sample_support, atom_positions,
|
||||
i, d, b0_size, n_atoms, atom_vals_d);
|
||||
float u = philox_uniform(i, timestep, rng_ctr++);
|
||||
float q_sample = sample_c51_inverse_cdf(probs_d, atom_vals_d, n_atoms, u);
|
||||
|
||||
// Temperature blend: τ=0 → argmax of E[Q]; τ=1 → pure Thompson;
|
||||
// τ>1 → exaggerated exploration (corrective during collapse).
|
||||
float q_eff = e_dir[d] + temp * (q_sample - e_dir[d]);
|
||||
|
||||
if (q_eff > best_q_eff) { best_q_eff = q_eff; best_d = d; }
|
||||
}
|
||||
dir_idx = best_d;
|
||||
```
|
||||
|
||||
The `if (eval_mode)` branch and its argmax body are deleted entirely.
|
||||
No conditional path remains — one selector for both training and eval.
|
||||
|
||||
### 3.2 Temperature formula (ISV producer)
|
||||
|
||||
Extended from SP9's existing `intent_eval_divergence_compute_kernel`. The
|
||||
existing kernel already computes `divergence`. Add a second scratch write:
|
||||
|
||||
```cuda
|
||||
// Existing SP9 output:
|
||||
scratch_buf[scratch_div_idx] = divergence;
|
||||
|
||||
// SP10 addition: temperature output, using same divergence + target
|
||||
float div_target = isv_signals[KELLY_DIVERGENCE_TARGET_INDEX]; // = 2.0 (Invariant-1)
|
||||
float temp = clamp(divergence / max(div_target, EPS_DIV),
|
||||
THOMPSON_TEMP_MIN, THOMPSON_TEMP_MAX);
|
||||
scratch_buf[scratch_temp_idx] = temp;
|
||||
```
|
||||
|
||||
Where:
|
||||
- `THOMPSON_TEMP_MIN = 0.5f` (Invariant 1: permanent stochasticity floor per
|
||||
`pearl_blend_formulas_must_have_permanent_floor`; never fully deterministic)
|
||||
- `THOMPSON_TEMP_MAX = 2.0f` (Invariant 1: cap on exploration boost; prevents
|
||||
pathological all-uniform sampling)
|
||||
- `EPS_DIV = 1e-6f` (Invariant 1: divide-by-zero protection)
|
||||
|
||||
Behavior:
|
||||
- Healthy (`divergence ≈ target = 2.0`): `temp ≈ 1.0` → pure Thompson
|
||||
- Collapsed (`divergence ≫ target`, e.g., 1e6 sentinel): `temp = 2.0` → exploration boost
|
||||
- Healthy and converging (`divergence < target`): `temp = 0.5` → closer to argmax behavior, but never fully deterministic
|
||||
|
||||
The MIN_TEMP=0.5 floor is the permanent-stochasticity guarantee per
|
||||
`pearl_blend_formulas_must_have_permanent_floor`. Eval action selection
|
||||
ALWAYS has at least 50% Thompson contribution.
|
||||
|
||||
### 3.3 ISV slot allocation
|
||||
|
||||
After SP9's last slot at 338:
|
||||
|
||||
| Slot | Purpose | Reset semantics |
|
||||
|---|---|---|
|
||||
| `EVAL_THOMPSON_TEMP_INDEX = 339` | Action selector temperature | FoldReset (sentinel 0, Pearl A bootstrap to MIN_TEMP) |
|
||||
|
||||
Bump `SP5_SLOT_END = 340` and `ISV_TOTAL_DIM = 340`.
|
||||
|
||||
Add 1 new scratch slot for the temperature output.
|
||||
|
||||
### 3.4 Producer kernel — extend or new?
|
||||
|
||||
**Decision: extend existing `intent_eval_divergence_compute_kernel`** rather
|
||||
than create a separate producer. Reasons:
|
||||
|
||||
1. The temperature is a deterministic function of `divergence` (no new
|
||||
inputs needed)
|
||||
2. Both outputs come from the same compute (cheap to add one extra write)
|
||||
3. Atomic commit per `feedback_no_partial_refactor` — slot + kernel
|
||||
extension + consumer migration all together
|
||||
4. Matches `pearl_engagement_rate_self_correction` pattern (one signal,
|
||||
multiple consumers)
|
||||
|
||||
The kernel signature gains 2 new parameters (scratch_temp_idx and
|
||||
divergence_target_isv_index) but otherwise unchanged.
|
||||
|
||||
## 4. Pearl amendments
|
||||
|
||||
### 4.1 `pearl_thompson_for_distributional_action_selection.md`
|
||||
|
||||
Update wording to clarify the selector/target distinction:
|
||||
- BEFORE: "Thompson at training, argmax at eval"
|
||||
- AFTER: "Thompson at all times for the rollout selector; argmax ONLY for the
|
||||
Bellman target Q computation (DDQN target). Temperature on the Thompson
|
||||
sample is ISV-driven from intent_eval_divergence per
|
||||
pearl_controller_anchors_isv_driven."
|
||||
|
||||
### 4.2 No new pearl
|
||||
|
||||
The principle here is captured by combining existing pearls:
|
||||
- `pearl_thompson_for_distributional_action_selection` (amended) — selector vs target
|
||||
- `pearl_controller_anchors_isv_driven` — the temperature is ISV-driven
|
||||
- `pearl_blend_formulas_must_have_permanent_floor` — MIN_TEMP=0.5 floor
|
||||
- `pearl_engagement_rate_self_correction` — temperature self-corrects via divergence
|
||||
|
||||
## 5. Test changes
|
||||
|
||||
`crates/ml/src/cuda_pipeline/mod.rs:1242` — `test_eval_action_select_eval_argmax_picks_best`:
|
||||
|
||||
- **Rename** to `test_eval_action_select_thompson_picks_proportionally`
|
||||
- **Update** assertion: instead of "best direction always wins", assert
|
||||
"best direction wins with probability ≥ posterior weight of best
|
||||
direction" (allowing the temperature blend's stochasticity)
|
||||
- **Histogram check**: with τ=1.0 and a clearly-best direction (large
|
||||
Q gap), best wins ≥ 80% of samples; with smaller gap, ≥ 50%
|
||||
|
||||
Existing comment at `mod.rs:1370` ("Thompson-eval histogram: short=...
|
||||
long=... (expected: P(Long)=1.0)") suggests prior preparation for this
|
||||
case — test framework already aware of Thompson-eval semantics.
|
||||
|
||||
## 6. Implementation phases (atomic per feedback_no_partial_refactor)
|
||||
|
||||
Single commit:
|
||||
|
||||
1. Allocate 1 new ISV slot (EVAL_THOMPSON_TEMP_INDEX=339), bump
|
||||
SP5_SLOT_END and ISV_TOTAL_DIM
|
||||
2. Extend `intent_eval_divergence_compute_kernel.cu` with temperature
|
||||
output (~10 LOC addition + signature change)
|
||||
3. Update launcher in `gpu_dqn_trainer.rs` for new kernel signature
|
||||
4. Modify `experience_action_select` kernel: delete `if (eval_mode)` argmax
|
||||
branch, replace Thompson branch with temperature-blended version, add
|
||||
`EVAL_THOMPSON_TEMP_INDEX` ISV read at top of direction-selection block
|
||||
5. Update FoldReset registry: 1 new slot entry, 1 new dispatch arm
|
||||
6. Update test `test_eval_action_select_eval_argmax_picks_best` →
|
||||
`test_eval_action_select_thompson_picks_proportionally`
|
||||
7. Update pearl `pearl_thompson_for_distributional_action_selection.md`
|
||||
in MEMORY (selector/target wording)
|
||||
8. Audit doc Fix 38 entry
|
||||
|
||||
Total estimated: ~300-500 LOC across ~6 files. ~45-60 min focused work.
|
||||
|
||||
## 7. Validation criteria
|
||||
|
||||
After SP10 lands:
|
||||
- Smoke (5-epoch L40S `multi_fold_convergence`):
|
||||
- val_active_frac > 0.10 with τ ≥ 0.5 (always some action variety)
|
||||
- dir_entropy at eval > 0.3 (multi-direction selection)
|
||||
- HEALTH_DIAG sp10 line shows τ evolving with divergence
|
||||
- T10 50-epoch:
|
||||
- val_trade_count > 1000 in 214k-bar window (was 1)
|
||||
- val_active_frac > 0.20 across folds
|
||||
- sharpe positive net of fold transitions
|
||||
|
||||
## 8. Risks and open questions
|
||||
|
||||
- **Risk 1**: Pure Thompson eval introduces PnL variance — short windows
|
||||
show more sharpe noise. Mitigation: τ floor at 0.5, not full Thompson;
|
||||
diagnostic is multi-fold mean sharpe, not single-fold spot value.
|
||||
- **Risk 2**: Production trading reproducibility — Philox RNG + state +
|
||||
timestep makes Thompson deterministic given seed. Same model + same
|
||||
seed = same actions. Reproducibility is preserved at the seed level.
|
||||
- **Risk 3**: The pearl amendment may conflict with intent at the time it
|
||||
was written. Per `feedback_trust_code_not_docs`, the smoke evidence
|
||||
trumps the original wording; pearl gets updated to reflect reality.
|
||||
|
||||
## 9. Followups (NOT in this commit)
|
||||
|
||||
- Adaptive temperature beyond the divergence canary (e.g., per-state
|
||||
uncertainty from Q variance, regime-dependent τ)
|
||||
- Apply same Thompson selector logic to mag/ord/urg branches if their
|
||||
argmax-at-eval pathways are found to bias similarly (currently mag
|
||||
uses Boltzmann softmax, ord/urg also stochastic — likely fine)
|
||||
- Bellman target Q computation review: confirm DDQN target argmax stays
|
||||
argmax (no SP10 change there)
|
||||
Reference in New Issue
Block a user