foxhunt/docs/superpowers/specs/2026-05-08-sp18-reward-shape-hold-attractor-design.md

SP18 — Combined Hold-Attractor Fix: Reward-Shape D-leg + TD(λ) Q(s') B-leg (Design Spec)

Status: v2 — UPGRADED to combined SP18+SP19 scope. DRAFT FOR HUMAN REVIEW. v1 deferred Approach B (TD(λ) Q(s') bootstrap) to SP19. That deferral is overridden per the "no deferrals for complementary fixes" call. Both legs land atomically in one branch / one plan. Spec must precede plan per the superpowers brainstorming pattern. Plan: docs/superpowers/plans/2026-05-08-sp18-reward-shape-hold-attractor.md.

---

Why combined scope (and why it is safe)

The vtj9r terminated 5-epoch L40S smoke (sha 0c57a5a31, terminated 2026-05-09 at HD4 across both folds with confirmed −14%/epoch decay, see Problem Statement) leaves three independently-acting causal mechanisms behind the persistent Q(Hold) attractor:

Mechanism	Fixed by	Refactor scope
1. Reactive control loop (controller bandwidth < policy convergence speed)	D-leg (Approach D)	reward kernels — experience_kernels.cu per-bar Hold sites + new compute_sp18_hold_opportunity_cost device fn + atomic delete of hold_cost_scale_update_kernel chain
2. Distributional shape mismatch (Hold tight, Long/Short wide)	D-leg (Approach D)	same as above (raises Hold's Q-target distribution width into Long/Short range)
3. TD(λ) self-bootstrap (q_next = rewards, never sees Q(s'))	B-leg (Approach B)	td_lambda_kernel consumer — gpu_experience_collector.rs:4143-4170 + new target-net forward pass on next-states during experience collection (target-net params already exist via target_params_buf)

Refactor scopes have empty intersection. D-leg writes to r_micro / r_opp_cost reward components inside experience_kernels.cu. B-leg writes to the q_next argument fed into td_lambda_kernel (one launch site at gpu_experience_collector.rs:4154). Per feedback_no_partial_refactor, atomicity is required within each leg's consumer set; between the legs there is no shared buffer or contract that breaks if they ship in separate commits inside the same branch.

Synergy hypothesis (testable): B-leg's correct credit assignment makes D-leg's gradient signal meaningful by construction. Under q_next = rewards self-bootstrap, the TD-error δ_t = r_t + γ × q_next_(rewards) − Q(s_t, a_t) collapses to δ_t ≈ r_t + γ × r_{t+1} − Q(s_t, a_t) — independent of which action was actually taken. With q_next = Q_target(s_{t+1}, a*_{online}) the TD-error becomes δ_t = r_t + γ × Q_target(s', argmax_a Q_online(s', a)) − Q(s_t, a_t) — the bootstrap term depends on the next-state action distribution, which depends on the previous action choice. This may also explain SP17's v_share elevation pattern (vtj9r showed v_share 0.33 → 0.83 across HD0–HD4): when the bootstrap is action-independent, V absorbs the bias because the per-action centred A has no signal to learn from. Combined SP18 tests this.

---

Problem Statement

Across SP11→SP17 the model develops a persistent Q(Hold) attractor: the directional Q-table prefers Hold to Long/Short despite training rewards that should favour active trading. SP17 (mean-zero advantage centering) eliminated V/A non-identifiability as a confounder, so the bias is now expressed cleanly through A_centered_dir[Hold] > A_centered_dir[Long] rather than V masquerading as action preference. Yet vtj9r v_share rose from 0.33 (HD0) to 0.83 (HD4) — V absorbing what A should have absorbed. Combined SP18 hypothesises this is the rewards-only TD(λ) bootstrap's signature.

Terminated smoke train-multi-seed-vtj9r (SP17 5-epoch L40S, sha 0c57a5a31, killed 2026-05-09):

Epoch	val Sharpe	Hold%	dir_entropy	A_var_dir EMA	v_share	Q(Hold)−Q(Long)
HD0	80.70	16.1%	(high)	0.0023	0.33	≈ 0.05
HD2	91.58	18.2%	1.07	(stable)	0.51	≈ 0.06
HD3	78.85	31.6%	0.996	0.0021	0.71	0.05–0.06
HD4	(term)	(term)	(term)	(term)	0.83	(term)

Trajectory: SP17 delays the attractor by 2 epochs and lifts the peak Sharpe by ~13% relative to HD0, but Hold% climbs +13.4 pts from HD2 to HD3 (peak to first decay) and v_share climbs monotonically across all 5 epochs. Slope ≈ −14% Sharpe / epoch matches train-multi-seed-b5gmp Fold 0 HD2→HD3 (pre-SP17 baseline). The architecture is healthy (A_var_dir stable, no V dominance at HD0–HD2), so the gradient-update process is correctly learning Hold-as-preferred under the current reward + the v_share elevation pattern is a symptom of the rewards-only TD(λ) bootstrap.

Conclusions:

• The reward function is asking the model to prefer Hold (D-leg fix).
• The TD(λ) bootstrap is structurally biased toward whichever action's reward distribution looks closest to its long-run accumulation, regardless of state-value, eroding V/A separation across epochs (B-leg fix).

---

D-leg Empirical Reward Audit (against current code)

Audit performed against worktree HEAD 0c57a5a31 (feat/sp17-dueling). All file:line citations live.

What Hold rewards today

Bar regime	Hold-action reward path	Magnitude (current ISV)
Voluntary trade close (segment_complete && !trail) on a Hold pick	r_popart = capped_pnl − shaping × min_hold_penalty (experience_kernels.cu:3315); plus per-bar Hold cost −ISV[380] × ISV[461] subtracted from base_reward (line 3166)	small, P&L-driven
Trail-fire close while picking Hold	r_trail = capped_pnl (line 3230); same Hold cost subtraction	rare
Per-bar mid-trade (positioned, no event) on Hold	r_micro = 0.0f − ISV[380]×ISV[461] (line 3617–3643)	0 − HOLD_COST_BASE×scale = −0.005×[1..25] = −0.005..−0.125
Per-bar Flat-Hold (no position)	r_opp_cost = 0.0f − ISV[380]×ISV[461] (line 3697–3716)	same as above, −0.005..−0.125

What Long/Short rewards on the same bars

• Per-bar mid-trade: r_micro = 0.0f (no Hold cost subtracted; line 3617). Identical to a Hold mid-trade in dollar terms when the per-bar Hold cost is at its floor (scale=1.0).
• Per-bar Flat-Long entering: not a different per-bar branch — opening a new position is entering_trade=true and runs through the segment_complete machinery on its eventual close, paying compute_lump_sum_opp_cost = −shaping × holding_cost_rate × |position| × hold_time (trade_physics.cuh:590) at exit.
• Trade close P&L: capped asymmetrically at [REWARD_NEG_CAP=−10, REWARD_POS_CAP=+5] (state_layout.cuh:337–338), with adaptive ISV[452]/ISV[453] producers tracking p99(winning_returns) × 1.5 (reward_cap_update_kernel).

Numerical reality check

Static base: HOLD_COST_BASE = 0.005 (sp13_isv_slots.rs:73), gain 5.0, floor 0.5×base, ceil 5.0×base ⇒ host controller emits cost in [0.0025, 0.025] per bar. SP16 P2 multiplier: ISV[461] ∈ [1, 25] driven by (observed_hold_rate − 0.20) / 0.20 overrun. With Wiener-α non-stationary floor 0.4 (post-T3, min_hold_temperature_update_kernel.cu:121).

Effective per-bar Hold penalty range: [0.0025, 0.625] (when scale=25).

Per-bar reward magnitudes during a typical trade (vol-normalized return, capped at +5/−10): O(0.01–0.1). So at the floor of the SP16 P2 ramp the Hold cost is ~5%–25% of a typical per-bar reward — competitive but small. At the ceiling it dominates. The ramp depends entirely on hold_rate_observed_ema exceeding 0.20.

Where the D-leg asymmetry actually lives

Three concrete asymmetries between Hold and Long/Short under the present reward shape:

1. Cost density. Hold pays a per-bar cost (line 3637–3643, line 3710–3716). Long/Short pay a lump-sum carrying cost only at trade close. With td_lambda=0.9 (config.rs:1427) and gamma≈0.95, the integrated and per-bar cost are mathematically equivalent only at γ=1; at γ<1 the per-bar Hold pricing discounts less per dollar than a deferred close-event cost.

2. Reward-cap clipping. Big winners get +5, big losers get −10 (2:1 Kahneman). Hold participates in the same clamp via compute_asymmetric_capped_pnl only if a position is open. Flat-Hold reward is dominated by −ISV[380]×ISV[461] only, never by a positive P&L term — and that subtraction is bounded by 0.625/bar. Long/Short collect occasional large bounded wins; Hold collects a steady stream of small bounded penalties. The Q-target distribution for Hold is a tight low-variance band; the Q-target distribution for Long/Short is a wide high-variance distribution centered near 0. CQL pessimism plus distributional collapse rewards low-variance branches automatically.

3. Bellman-target inflation under TD(λ) self-bootstrap. This is the B-leg's territory; carved out below.

---

B-leg TD(λ) Self-Bootstrap Audit (against current code)

Audit performed against worktree HEAD 0c57a5a31. All file:line citations live.

Where the bug lives — single launch site

gpu_experience_collector.rs:4135-4174 is the entire production TD(λ) wiring:

// Line 4135-4143:
let td_lambda = config.td_lambda;            // = 0.9 from config.rs:1427
let max_trace = config.max_trace_length;
if td_lambda > 0.01 && max_trace > 1 {
    let raw_rewards_td = dtod_clone_f32_native(&self.stream, &self.rewards_out, total, "raw_rewards_td")?;
    let raw_dones_td   = dtod_clone_f32_native(&self.stream, &self.done_out,    total, "raw_dones_td")?;
    let q_next         = dtod_clone_f32_native(&self.stream, &self.rewards_out, total, "q_next_bootstrap")?;
    //                                                            ^^^^^^^^^^^^^^^^ ← THE BUG
    // Line 4146-4170: launch td_lambda_kernel with (raw_rewards_td, raw_dones_td, q_next, rewards, dones, ...)
}

The kernel's own header explicitly licenses this:

nstep_kernel.cu:95-96
 * Requires bootstrapped Q(s') values. When not available, pass raw_rewards
 * as q_next for a self-bootstrap approximation.

Consumers of the kernel's output

td_lambda_kernel output writes to rewards (out_rewards, line 4158) and dones (out_dones, line 4159) — these are the SAME buffers passed into the TD(λ)-trained replay batch. So the consumers of rewards post-TD(λ) are:

1. Replay buffer push — rewards flows into the batch that is appended to the per-fold replay buffer. (One push per experience-collection cycle.)
2. PER initial priorities — NOT computed at experience-collection time. PER priorities are recomputed at training time in agent.update_priorities_from_td(...) from current TD-error (config.rs:341, fused_training.rs:2963). Initial sampling priority is max_priority heuristic on first-store. No buffer schema migration is needed for B-leg.
3. Loss-time TD-error — at training time, the C51 / IQN loss kernels read rewards from the sampled replay batch and combine with target-net Q(s') (already done correctly via submit_forward_ops_main's Pass 2 target-net forward on next_states + compute_expected_q projection through c51_loss_batched).

The bug is confined to the experience-collection-time TD(λ) accumulation. The C51 loss kernel's distributional Bellman target site (c51_loss_kernel.cu:188 block_bellman_project_f) is already centered correctly because the loss-time target_probs come from a target-net forward pass that already exists at loss time. Mirror the SP17 finding that c51_loss was already centered — same pattern here for the target-Q at loss time.

The defect is that the TD(λ) returns themselves (the R^λ written into rewards during experience collection) are computed with rewards-as-q_next, biasing intermediate λ-blended terms toward E[reward] rather than E[Q(s')]. The loss kernel's outermost Bellman bootstrap is already a target-net Q, but the inner λ-weighted return is rewards-only. This is the right surgical site to fix: replace q_next at line 4143 with a target-net compute_expected_q evaluation on next_states.

Inputs needed to compute correct Q(s') during experience collection

To wire q_next = Q_target(s', argmax_a Q_online(s', a)) (DDQN per B-DD2/B-DD3) at experience-collection time:

Input	Where it lives	Available at collection time?
target_params (target-net weights)	gpu_dqn_trainer.rs:5359 target_params_buf	YES — Polyak-tracked, populated since SP14 P4.T1B-iv
next_states (vol-normalized post-action market state)	built at gpu_experience_collector.rs:4182 build_next_states_f32	YES — built in same scope, line 4182, AFTER td_lambda runs today (must be moved BEFORE td_lambda for B-leg)
online_params (online-net weights for DDQN argmax)	exposed via params_buf_device_ptr()	YES
Forward-pass machinery (encoder + GRN + VSN + dueling head)	all kernels exist; orchestration lives in submit_forward_ops_main (Pass 2 target-net branch already runs target on next_states at loss time)	NEEDS new wrapper for collection-time invocation
compute_expected_q for the projection	experience_kernels.cu:4572	YES — already takes params, inputs, writes per-action E[Q]

Replay buffer schema check

Searched agents/, trainers/, cuda_pipeline/ for replay_buffer writes. The push site stores (state, next_state, reward, done, priority_initial=max_priority). q_next is NEVER stored in the buffer — it's a transient scratch in the experience-collection cycle that gets consumed by td_lambda_kernel and then discarded. Confirmed: no replay buffer schema migration needed.

Graph-capture check

exp_fwd_graph capture region (gpu_experience_collector.rs:4651–4671) covers the experience-collection forward pass for action selection. The TD(λ) launch at line 4154 is outside the capture region (it runs after the per-step inner loop completes, after next_states is built). Adding a target-net forward pass before line 4154 is graph-captureable as long as it lives in the same post-loop scope as today's TD(λ) launch — purely kernel launches, no host branches per pearl_no_host_branches_in_captured_graph.

PopArt interaction

PopArt slot 63 normalizes Q-targets at loss time (after the trained replay batch is sampled). With the B-leg fix:

• Pre-fix: trained R^λ ≈ rewards-weighted-cumsum; PopArt sees R^λ + γ^N × Q_target(s_{t+N}) where the inner R^λ is rewards-mean-anchored.
• Post-fix: trained R^λ ≈ target-Q-bootstrapped at every λ-step; PopArt sees the same outer expression but the inner R^λ is now Q-mean-anchored (which is the magnitude PopArt was designed to track).

Risk: at the moment of switching (HD0 of first SP18 run), PopArt's existing EMA state was tuned to the rewards-distributed R^λ. Switching to Q-distributed R^λ shifts the input distribution at one step. PopArt has its own α-EMA that adapts; the question is whether it adapts within HD0 or causes a transient. B-DD11: per-fold reset of PopArt slot 63 EMA at the SP18 deployment boundary. (Cheap insurance, mirrors SP14 fold-boundary reset patterns.)

---

Approach Considered

Approach A — Symmetric trade-cost compensation (REJECTED)

Charge Hold a structural position × half_spread × decay per bar. Rationale: Long/Short already pay an entry/exit half-spread; a Hold action that maintains an open position is also "consuming" the time-decay aspect of having committed capital.

Why rejected: SP16 P2 (hold_cost_scale_update_kernel) is exactly this in spirit — adaptive multiplier on the per-bar Hold cost driven by observed Hold-rate overrun. The mechanism failed because it's reactive (closed-loop control with Wiener-α latency), not because it lacked the right semantic concept. Layering another version of the same fix would be the canonical feedback_no_quickfixes violation.

Approach B — Eligibility traces / TD(λ) Q(s') bootstrap fix (ACCEPTED — combined as B-leg)

v1 deferred this to SP19. v2 accepts as the B-leg of combined SP18. See "Why combined scope" above and the Architecture Changes B-leg section below.

Switching to a real Q(s') bootstrap (using the target-net's expected-Q on the realized next-state) gives n-step / TD(λ) returns the correct credit-assignment shape. This is exactly what pearl_event_driven_reward_density_alignment recommends ("eligibility traces, n-step returns, HER… are RL-architectural credit-assignment tools, not per-step reward hacking").

Wiring real Q(s') into td_lambda_kernel requires:

• adding a target-network forward pass on next_states during experience collection (currently only happens in the loss kernel)
• piping the resulting expected-Q back into the q_next argument
• accepting ~30% collection-time slowdown (extra forward pass; B-DD6 — H100 has headroom; profile in Pre-Phase to confirm before committing)
• per-fold reset of PopArt EMA (B-DD11)

No replay buffer schema migration. No PER priority migration (PER recomputes from current TD-error at training time). No C51 loss kernel migration (already correctly using target-Q).

Approach C — Bellman target redesign (REJECTED)

Center per-step reward at running mean before it enters the Bellman target. Concrete: r_centered = r − reward_mean_ema; Q-target uses r_centered.

Why rejected: collides with PopArt (slot 63), the SP11 reward-subsystem controller (slots 340–346), the SP15 DD-asymmetric-reward producer, and the SP12 asymmetric +5/−10 cap. Each consumes the un-centered reward; centering would either need to flow through all of them or live as a post-hoc bias subtraction at the Bellman site (which breaks the SP15 reward-axis "second-pass composer" contract).

Approach D — Asymmetric Hold-vs-position reward bound (ACCEPTED — combined as D-leg)

Per pearl_audit_unboundedness_for_implicit_asymmetry, the SP12 +5/−10 cap restored loss aversion to position P&L. Hold has no equivalent. A natural per-bar Hold reward is symmetric around zero (steady tiny negatives only when the controller ramps). Make the Hold per-bar reward asymmetric by adding a bounded reinforcement for the realized opportunity-cost of forgone directional moves, scaled by a controller-driven signal that reflects whether the data currently supports trading.

Concretely (atomic):

1. Hold-anchored reward decomposition. Replace per-bar Hold cost subtraction at the 3 sites with a per-bar Hold reward r_hold that is the sign-flipped vol-normalized realized tgt[1] next-bar log-return — bounded by the same compute_asymmetric_capped_pnl machinery the segment-complete path uses. This lifts Hold's reward distribution into the same dynamic range as Long/Short.

2. Asymmetric Hold reward cap. New ISV pair [483..485) — HOLD_REWARD_POS_CAP_INDEX and HOLD_REWARD_NEG_CAP_INDEX — driven by a producer mirroring reward_cap_update_kernel but tracking p99(|step_ret|) of Long/Short trade closes only.

3. Anchored at vol-normalized log-return only. Per pearl_one_unbounded_signal_per_reward: the unbounded multiplicand is vol_normalized_return; everything else is bounded.

Why D over the others (D-leg only — see "Why combined scope" for the multi-leg argument): addresses mechanism not magnitude; structural not reactive; reuses compute_asymmetric_capped_pnl and the reward_cap_update_kernel pattern; minimal new code surface; does not touch PopArt, SP11 controller, SP15 composer, or PER.

---

Architecture Changes — D-leg (assuming DD1=combined, DD2=a, DD3=b, DD4=a, DD5=b, DD6=as-specified, DD7=c, DD8=Phase 3.1)

CUDA changes (D-leg)

crates/ml/src/cuda_pipeline/experience_kernels.cu

1. At the 3 Hold-cost subtraction sites:

   • line 3162–3166 (segment_complete branch): replace the 4-line if (dir_idx == DIR_HOLD && isv_signals_ptr != NULL) block with a single call into compute_sp18_hold_opportunity_cost(...) (see point 2). Output flows into r_micro (additive, returns 0 if not Hold).
   • line 3637–3643 (per-bar mid-trade Hold): same.
   • line 3710–3716 (per-bar flat-Hold): same.

2. compute_sp18_hold_opportunity_cost lives in trade_physics.cuh (after compute_lump_sum_opp_cost at line 590):

   • Inputs: dir_idx, tgt[1] next-bar log-return, vol_proxy, hold_segment_bars, isv_pos_cap = isv[483] or fallback 5.0, isv_neg_cap = isv[484] or fallback −10.0, shaping_scale.
   • Output: 0.0 if dir_idx != DIR_HOLD; else compute_asymmetric_capped_pnl(−2.0 × vol_normalized_log_return, isv_neg_cap, isv_pos_cap) × shaping_scale.

crates/ml/src/cuda_pipeline/hold_reward_cap_update_kernel.cu (NEW)

Mirrors reward_cap_update_kernel shape: producer reads p99 of |step_ret| over Long/Short trade closes (per DD3=b), multiplies by safety_factor=1.5, writes POS=that and NEG=−2 × POS (DD5=b). Wiener-optimal α via Welford accumulators in [487..493). Pearl-A first-observation bootstrap; sentinel SENTINEL_HOLD_REWARD_POS_CAP=5.0. Per-fold cadence (slow EMA, similar to position-side cap producer α=0.01).

crates/ml/src/cuda_pipeline/sp14_isv_slots.rs

D-leg ISV section [483..493): 10 slots per DD6. Sentinels, MIN/MAX bounds, EMA α, safety factor. Compile-time assertion SP18_SLOT_END <= ISV_TOTAL_DIM.

crates/ml/src/cuda_pipeline/state_layout.cuh

Mirror constants for SP18 slots (POS/NEG sentinels, MIN/MAX bounds). Bump ISV_TOTAL_DIM mirror.

Rust changes (D-leg)

crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs

• Bump ISV_TOTAL_DIM 483 → (B-leg also adds slots; final value pinned in Pre-Phase task PP.2).
• Update layout_fingerprint_seed per existing pattern.
• New launch_hold_reward_cap_update method.

crates/ml/src/trainers/dqn/state_reset_registry.rs

• Add fold-reset entries for D-leg slots [483..493) and B-leg slots (TBD in PP.2).

crates/ml/src/trainers/dqn/trainer/training_loop.rs

• Per-epoch HEALTH_DIAG emit reading [483..487) for D-leg.
• DELETE host-side Hold-cost controller block at lines 4171–4188 per DD7(c).
• DELETE launch_hold_cost_scale_update call at training_loop.rs:564 per DD7(c).

crates/ml/src/cuda_pipeline/hold_cost_scale_update_kernel.cu — DELETE per DD7(c).

---

Architecture Changes — B-leg

CUDA changes (B-leg)

No new kernels. Reuses compute_expected_q (experience_kernels.cu:4572) via a new Rust orchestration path that invokes it with target-net params on next-states during experience collection.

Rust changes (B-leg)

crates/ml/src/cuda_pipeline/gpu_experience_collector.rs

1. Move let next_states = build_next_states_f32(...) from line 4182 to before line 4137 (the if td_lambda > 0.01 && max_trace > 1 block). Required so target-net forward pass can consume next_states.

2. Add new method compute_q_next_target_bootstrap(&mut self, next_states: &CudaSlice<f32>, target_params_ptr: u64, online_params_ptr: u64, B: usize) -> Result<CudaSlice<f32>> that:

   • Runs the encoder + dueling forward pass with online params on next_states, identical to submit_forward_ops_main's Pass 3 (DDQN argmax mirror at submit_forward_ops_ddqn). Captures per-branch argmax actions.
   • Runs the encoder + dueling forward pass with target params on next_states, identical to Pass 2 inside submit_forward_ops_main.
   • Calls compute_expected_q on the target-net distributional outputs to produce per-action E[Q] of size [B, total_actions=13].
   • Per B-DD3 per-branch DDQN: gathers Q_target[a*_online_dir] for the dir branch and likewise per other 3 branches; sums the 4 selected expected-Q values into a scalar q_next_per_sample[B] (matches the SP17 per-branch identifiability claim — each branch's argmax is taken on its own factored Q-table).
   • Writes the result into a CudaSlice<f32> of length B (one Q-bootstrap value per sample).

3. Replace line 4143 let q_next = dtod_clone_f32_native(...) with let q_next = self.compute_q_next_target_bootstrap(&next_states, target_params_ptr, online_params_ptr, total)?;. Drop the "q_next_bootstrap" debug name (the buffer is no longer a clone; rename to "q_next_target").

4. Distributional flavour of B-DD8: under C51 the natural "target Q" is per-atom; replacing it with a scalar expected_Q collapses distributional information at the bootstrap step. Two options:

   • B-DD8.a (scalar): q_next[b] = scalar expected_Q from compute_expected_q on target-net at next_states. Simplest, matches today's td_lambda_kernel signature (which expects float* q_next, one per sample).
   • B-DD8.b (distributional): extend td_lambda_kernel to consume per-atom target probs and emit R^λ per-atom, then project at PopArt. Larger surface; defer.

   Recommendation: B-DD8.a (scalar) for atomic SP18 scope. The C51 distributional Bellman target at loss time is already correct (uses target-net atom positions via block_bellman_project_f). The TD(λ) self-bootstrap defect is about credit assignment along the trace, not about the outer atom projection. Scalar expected_Q for the trace-blending is sufficient to fix the credit-assignment bug. Distributional λ-blending would be a follow-up if SP18 5-epoch smoke shows residual atom-distribution drift.

crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs

• Expose params_buf_device_ptr() and target_params_buf_device_ptr() to experience collector context (already exposed at lines 8847, 11426).
• Surface a wrapper compute_expected_q_for_collection(target_pass: bool) if needed — likely already accessible via existing pointers.

crates/ml/src/trainers/dqn/trainer/training_loop.rs

• B-leg per-epoch HEALTH_DIAG: emit td_error_magnitude_ema, q_next_target_p99, q_next_minus_reward_p99 (all per-batch from the new producer kernel — see B-DD12 below).

B-leg ISV slots [493..505)

12 slots (B-DD13 below):

[493]  TD_ERROR_MAG_EMA              (HEALTH_DIAG observable: avg(|TD_error|) post-fix)
[494]  Q_NEXT_TARGET_P99             (p99 of target-Q bootstrap value)
[495]  Q_NEXT_MINUS_REWARD_P99       (sanity: p99 of |q_next_target − rewards| — should be O(1) post-fix)
[496]  V_SHARE_TREND_DIAG            (slope of V_SHARE EMA across last 4 epochs; B-leg synergy probe)
[497]  POPART_RESET_FLAG             (1.0 first epoch post-deployment; 0.0 thereafter — gates the per-fold PopArt EMA reset per B-DD11)
[498]  TDB_TARGET_MEAN               (Welford for q_next_target Wiener-α)
[499]  TDB_TARGET_M2
[500]  TDB_DIFF_MEAN
[501]  TDB_DIFF_M2
[502]  TDB_PREV_TARGET
[503]  TDB_SAMPLE_COUNT
[504]  RESERVED                      (reserve 1 slot for B-leg follow-up if needed)

Total combined: 22 slots, [483..505). ISV_TOTAL_DIM bumps 483 → 505.

---

DESIGN DECISION 1 — Combined scope (CONFIRMED)

Recommendation: D-leg + B-leg combined, atomic in feat/sp18-reward-shape. Per task input override of v1's "defer B to SP19".

Human reviewer: confirm combined, or override to D-only / B-only.

---

DESIGN DECISION 2 — Hold opportunity-cost source (CONFIRMED, recommendation a)

(a) tgt[1] preprocessed next-bar log-return — already in scope at line ~2380, vol-normalized via the same atr_norm chain the segment_complete path uses. Lookahead-clean.

Reviewer: confirm (a).

---

DESIGN DECISION 3 — Cap producer scope (CONFIRMED, recommendation b)

(b) p99(|step_ret|) over Long/Short trade closes. Aligns Hold's reward distribution with policy's current realized Long/Short distribution.

Reviewer: confirm (b).

---

DESIGN DECISION 4 — When to apply Hold opportunity-cost (CONFIRMED, recommendation a)

(a) Always (every bar where action_dir==Hold), both flat and positioned.

Reviewer: confirm (a).

---

DESIGN DECISION 5 — Cap asymmetry inversion direction (CONFIRMED, recommendation b)

(b) Mirrored — same as position side, Hold-side [+5, −10]. Says "missing a winner is the bigger pain." Drives the policy toward engagement.

Reviewer: confirm (b).

---

DESIGN DECISION 6 — D-leg ISV slot allocation (CONFIRMED, as-specified)

10 slots [483..493) per the v1 spec.

Reviewer: confirm.

---

DESIGN DECISION 7 — Hold cost retention vs. replacement (CONFIRMED, recommendation c)

(c) Replace, but keep the observation chain (slots 380, 381, 382, the hold_rate_observer_kernel per-step writes). Delete only the cost-application sites + the scale producer + the host controller.

Reviewer: confirm (c).

---

DESIGN DECISION 8 — Cubin manifest registration timing (CONFIRMED, Phase 3.1)

Register hold_reward_cap_update_kernel.cubin in Phase 3.1 alongside the kernel skeleton (avoids broken-build state in Pre-Phase).

Reviewer: confirm.

---

DESIGN DECISION B1 — Target-net infrastructure source (RECOMMENDATION)

Use existing target_params_buf (Polyak-tracked since SP14 P4.T1B-iv). No new target-net infra. Verified at gpu_dqn_trainer.rs:5359 + the target Polyak EMA Pass 2 at loss-time (which proves the buffer is up-to-date relative to online).

Reviewer: confirm.

---

DESIGN DECISION B2 — DDQN argmax at bootstrap (RECOMMENDATION)

q_next = Q_target(s', argmax_a Q_online(s', a)). Standard double-DQN pattern.

Reviewer: confirm.

---

DESIGN DECISION B3 — Per-branch DDQN argmax (RECOMMENDATION)

Take 4 separate argmaxes — one each for dir, mag, ord, urg factored Q-tables. Matches SP17 per-branch identifiability claim (each branch is its own Q-axis; combining via sum after argmax is the SP17 contract). Sum the 4 selected target-Q values into the scalar q_next per sample.

Reviewer: confirm. Alternative: single global argmax on the 13-action joint, which loses the SP17 factored-Q semantics — REJECTED.

---

DESIGN DECISION B4 — PER priority recomputation (CONFIRMED)

PER priorities are already recomputed at training time from current TD-error (config.rs:341); not stored in buffer or precomputed in experience collector. No PER migration needed. The new TD(λ)-corrected rewards flowing into the buffer feed naturally into the existing priority recomputation. B9 oracle test required to confirm the post-fix priority distribution is reasonable.

Reviewer: confirm.

---

DESIGN DECISION B5 — Target-net forward pass during experience collection (CONFIRMED)

Per the audit, td_lambda_kernel consumes q_next at collection time. To replace with a real Q(s'), we MUST run the target-net forward pass at collection time (not loss time). Adds one extra forward pass per experience-collection cycle.

Reviewer: confirm.

---

DESIGN DECISION B6 — Accept ~30% collection-time slowdown (RECOMMENDATION)

Profile in Pre-Phase task PP.5 (NEW) — measure end-to-end experience collection time on H100 before merging Phase 4 to confirm headroom. If >50% slowdown, fall back to B-DD8.a scalar-only and skip per-atom target-Q (already the recommendation). If still >50%, halt and convene human review.

Reviewer: confirm.

---

DESIGN DECISION B7 — Reuse target-net VSN params (CONFIRMED)

Existing target_params_buf includes VSN params (per SP14 P4.T1B-iv migration). The target-net forward pass on next_states uses the same VSN gates as today's loss-time Pass 2 — same vsn_gated_next_states_buf. No new VSN handling.

Reviewer: confirm.

---

DESIGN DECISION B8 — Per-atom vs scalar target-Q (RECOMMENDATION a)

B-DD8.a (scalar) for atomic SP18. Distributional flavour deferred to follow-up if 5-epoch smoke shows atom-distribution drift. See B-leg Architecture Changes section above for full reasoning.

Reviewer: confirm a.

---

DESIGN DECISION B9 — TD-error magnitude smoke gate (RECOMMENDATION)

avg(|TD_error|) ratio post-fix / pre-fix ∈ [0.5, 5.0] at HD0–HD1 (sanity check that bootstrap change didn't blow up loss magnitude). HD0 baseline measured in Phase 0 diagnostic pre-fix; HD0 measurement post-fix taken in Phase 7 smoke; ratio computed per-fold per-epoch.

Reviewer: confirm.

---

DESIGN DECISION B10 — Atomic refactor scope (CONFIRMED)

Atomic refactor includes: gpu_experience_collector.rs:4143-4170 (the line 4143 q_next replacement + the target-net forward pass before line 4137) + the new compute_q_next_target_bootstrap method + the next_states build-site relocation + B-leg ISV slots and HEALTH_DIAG emit. Audit script in Phase 1 surfaces additional consumers. PER priority computation and C51 loss kernel migration NOT needed (per B-DD4 and B-leg audit).

Reviewer: confirm.

---

DESIGN DECISION B11 — PopArt slot 63 EMA reset at deployment boundary (NEW, RECOMMENDATION)

Per audit, switching q_next source from rewards-distributed to Q-distributed shifts the input distribution to PopArt at the SP18 deployment. PopArt's α-EMA will adapt but may take 1+ epochs.

Recommendation: per-fold reset of PopArt slot 63 EMA at the SP18 deployment boundary, gated by new ISV slot 497 POPART_RESET_FLAG. Slot 497 is set to 1.0 by host at first SP18 epoch and 0.0 thereafter (one-shot). On reset, PopArt EMA reinitializes to identity normalization (μ=0, σ=1). Cheap insurance.

Reviewer: confirm, or skip if you'd rather rely on PopArt's natural adaptation.

---

DESIGN DECISION B12 — B-leg HEALTH_DIAG observability (NEW, RECOMMENDATION)

Emit per-epoch:

• td_error_mag_ema (slot 493): avg(|TD-error|) computed post-fix
• q_next_target_p99 (slot 494): bound check on the new bootstrap
• q_next_minus_reward_p99 (slot 495): sanity — should be O(1), not O(0) (proves the fix is doing something) and not O(100) (proves it isn't blowing up)
• v_share_trend_diag (slot 496): slope of V_SHARE across last 4 epochs (should flatten or reverse under the synergy hypothesis)

Reviewer: confirm.

---

DESIGN DECISION B13 — B-leg ISV slot allocation (NEW, RECOMMENDATION)

12 slots [493..505) per the table in "B-leg ISV slots" above. Combined SP18 total: 22 slots [483..505). ISV_TOTAL_DIM bumps 483 → 505.

Reviewer: confirm 12-slot allocation, or trim B-leg diagnostics to live on HEALTH_DIAG-only readout (leave only [493..498) — net 6 B-leg slots, total 16 [483..499)).

---

Behavioral Test Predictions

Tests live at crates/ml/tests/sp18_hold_reward_oracle_tests.rs (D-leg) and crates/ml/tests/sp18_td_lambda_q_next_oracle_tests.rs (B-leg). Each is a single TDD cycle.

D-leg Tests

T1 — CPU oracle for compute_sp18_hold_opportunity_cost

Given: next_log_return = +0.005, vol_proxy = 0.0001, pos_cap = 5.0, neg_cap = -10.0, shaping_scale = 1.0. Expected: returns negative scalar bounded in [-10, 0]. Negative log-return inputs return positive scalar bounded in [0, +5]. dir_idx != DIR_HOLD returns 0.0 exactly.

T2 — GPU oracle parity with CPU

Same inputs on MappedF32Buffer; bit-identical to ε=1e-6.

T3 — Hold reward distribution on synthetic uniform-edge market

Synthetic tgt[1] ~ Uniform[-0.005, +0.005]. After 1000 Hold picks: mean(r_hold_opp) ≈ 0, var(r_hold_opp) > 0.001.

T4 — Hold reward distribution on positive-edge synthetic market

tgt[1] ~ N(+0.001, 0.002). After 1000 Hold picks: mean(r_hold_opp) < 0, mean(r_long_pick - r_hold_opp) > 0 for Long picks on the same bars.

T5 — Cap producer drives slots [483, 484]

After 100 simulated Long/Short close events with step_ret ∈ N(0, 0.01): ISV[483] ≈ p99(|step_ret|) × 1.5, ISV[484] = -2 × ISV[483], both within bounds.

T6 — End-to-end on converged b5gmp checkpoint

Hold expected-Q variance over actions ≥ 0.5× Long expected-Q variance over actions; no NaN; cap producers populate within 1 launch.

T7 — Scaffold-level: SP13/SP16 deletion compiles + tests pass

Atomic delete leaves the build green and existing tests pass.

B-leg Tests

T8 — CPU oracle for DDQN per-branch argmax bootstrap

Pure math: given synthetic Q_online[B, 13] and Q_target[B, 13] (factored 4+3+3+3), compute q_next_per_sample[b] as Σ_branch Q_target[b, argmax_a_in_branch Q_online[b, a]]. Lock the contract: dir-branch argmax in [0..4), mag in [4..7), ord in [7..10), urg in [10..13).

T9 — GPU oracle parity with CPU (B-leg)

Same inputs on MappedF32Buffer; bit-identical to ε=1e-6.

T10 — TD-error magnitude before vs after fix

Synthetic 100-step rollout: pre-fix q_next = rewards; post-fix q_next = Q_target(s', argmax_online). Verify avg(|TD_error|) ratio ∈ [0.5, 5.0] (B-DD9). On a converged checkpoint replay buffer this guards against the bootstrap change blowing up (or zeroing out) the loss.

T11 — Target-net forward consistency

Run the new compute_q_next_target_bootstrap and compare its output to the target-net forward result that already exists at loss time on the same (next_states, target_params). Must be bit-identical (same kernel, same params, same input). Locks "no second source of truth".

T12 — PER priority distribution post-fix

After 1 training step on a converged-checkpoint replay buffer with the SP18 fix wired: PER priority distribution shape (mean, p50, p99) within ±2× of pre-fix. No degenerate "all priorities equal" or "single priority dominates" signature.

T13 — Graph-capture compatibility

Capture exp_fwd_graph with the new target-net forward pass on next_states. Replay 10× and verify per-replay determinism. Locks B-leg Architecture point 1 (move next_states build before TD(λ) launch) doesn't break capture.

---

Smoke Gates (anchored to vtj9r and b5gmp empirical baselines)

Phase 9 5-epoch L40S smoke. Gates anchored to train-multi-seed-vtj9r (terminated 2026-05-09, sha 0c57a5a31) and train-multi-seed-b5gmp Fold 1 final (sha 641aa0dfd).

D-leg gates

Gate	Threshold	Source
val Sharpe HD2 (peak)	≥ 91.58	vtj9r HD2 best Sharpe — SP18 must not regress
val Sharpe HD3 (post-peak)	≥ 80	vtj9r HD3 = 78.85; b5gmp Fold-1 = 55.55 (failure)
Hold% HD3	≤ 25%	vtj9r HD3 = 31.6% (climbing); SP18 must keep below
dir_entropy HD3	≥ 1.0	vtj9r HD3 = 0.996
Hold opp-cost fire-rate (HD3)	≥ 0.80	structural — D-leg reward must fire
r_hold_opp distribution width	mean(|r_hold_opp|) > 0.5 × mean(|r_popart|)	structural — distribution match
ISV[483] settled	∈ [0.01, 5.0] by HD2	adaptive producer alive by HD2
Q(Hold) − Q(Long) gap	≤ 0.10 throughout	b5gmp Fold-1 = 0.148 (pre-SP17 failure)
no SP13/SP16 dead-code refs	grep returns 0	DD7(c) atomic delete clean

B-leg gates

Gate	Threshold	Source
avg(|TD-error|) ratio post-fix / pre-fix HD0	∈ [0.5, 5.0]	B-DD9 sanity
v_share trend HD0→HD4	monotone decreasing OR flat	vtj9r showed monotone increasing 0.33→0.83 — SP18 must reverse or flatten
q_next_target_p99 (HD2)	≤ 50.0	bound check; reward cap is +5/-10 so target Q ≤ ~50 reasonable
q_next_minus_reward_p99 (HD2)	> 0.1 AND < 100.0	structural — fix is doing something but not exploding
PER priority p99 / p50 ratio (HD2)	≤ 100.0	distribution sanity
PopArt EMA stable (HD0+1 to HD3)	slot 63 not increasing >2× HD0 value	PopArt should re-converge after B-DD11 reset; large drift = adaptation failure

Combined gates

Gate	Threshold	Source
HEALTH_DIAG never NaN	always	always
no train hang	always	always
audit script returns clean	grep returns 0 dead refs	DD7(c) + B-leg consumer migration

Phase 10 30-epoch full validation uses same gates; thresholds tightened on val Sharpe Fold-1 final ≥ 80, Hold% Fold-1 final ≤ 25%, dir_entropy Fold-1 final ≥ 0.95, Q(Flat)−Q(Long) gap ≤ 0.10.

If smoke fails any bold gate, halt and convene human review before 30-epoch dispatch.

---

Hard rules (apply to every task in the plan)

• Pre-dispatch consumer audit script (Phase 1 first task) — surfaces missed consumers BEFORE migration. Promoted to a pre-commit hook (Open Q-B confirmed). Generalizable beyond SP18.
• Outcome-anchored smoke gates — every threshold cites an empirical baseline.
• DESIGN DECISION markers for every genuine choice (DD1–8 + B-DD1–13).
• TDD per task — failing test first.
• No partial refactor within each leg.
• ISV-driven adaptive bounds — pos/neg caps via producer.
• No CPU compute — opp-cost producer + B-leg target-net forward + reward computation pure GPU.
• Mapped-pinned only for diagnostic readback.
• Pearl candidate per phase.

---

Memory pearl candidates (drafts)

Pearl 1 (post-Phase 1 atomic delete): pearl_replace_reactive_controller_with_structural_signal.md

When a reactive control loop on an emergent symptom (per-bar Hold cost driven by observed Hold-rate overrun) fails to keep up with the policy's per-batch convergence, replace it with a structural per-step signal that is immediately present in the reward shape regardless of policy state. SP13/SP16 raised Hold cost reactively after the policy had already over-Held; SP18 D-leg makes Hold's reward distribution structurally comparable to Long/Short by paying the realized log-return as opportunity cost on every Hold bar — distributional-collapse advantage eliminated at source, no controller bandwidth required.

Pearl 2 (post-Phase 5 if synergy holds): pearl_target_net_q_during_experience_collection_for_correct_credit_assignment.md

When TD(λ) / n-step return accumulators run during experience collection, the bootstrap value must be a real Q(s') (target-net evaluated on next-states), not the rewards stream. Self-bootstrap (q_next = rewards) collapses TD-error to a state-independent reward-tracker which biases the learning signal toward whichever action's reward distribution best matches its long-run accumulation. Symptoms include monotone V-share elevation (V absorbing the action-independent bias because A has no signal to learn) and reward-shape attractors that resist per-step controller fixes. SP18 B-leg fix: run target-net forward on next_states during experience collection, gather DDQN per-branch argmax E[Q] from the target-net's distributional output, feed as q_next into td_lambda_kernel. Adds ~30% to collection time on H100 (acceptable headroom). No replay buffer schema change; no PER priority change (recomputed at training time anyway); no C51 loss kernel change (loss-time target-Q already correct).

Pearl 3 (post-Phase 9 if smoke passes): pearl_distributional_shape_match_for_action_neutrality.md

When one action's Q-distribution is structurally tighter than another's, CQL pessimism + C51 atom occupancy patterns favour the tight branch independent of expected value. To avoid the resulting attractor, design rewards so each action's Q-target distribution has comparable width and tails. SP18 D-leg example: Hold-action Q-targets had width O(0.005) (per-bar penalty band), Long/Short Q-targets had width O(5) (capped trade P&L). Adding bounded opportunity-cost reward to Hold lifted Hold's distribution width to comparable scale.

Pearl 4 (post-Phase 9 if combined fix lands): pearl_complementary_fixes_no_deferrals.md

When two distinct mechanisms drive the same observed pathology, do NOT defer the more-architecturally-honest fix in favour of the simpler one. Fix both in one branch, atomically, so that synergy effects are exposed within the same smoke run. SP18 example: D-leg (reward-shape) and B-leg (TD(λ) bootstrap) were initially scoped as SP18 + SP19. Refactor scopes had empty intersection (D-leg writes reward components, B-leg replaces a dtod_clone_f32_native argument); combined ship was safe per feedback_no_partial_refactor. Splitting would have required two L40S validation cycles; combined exposed the synergy hypothesis (B-leg fix reverses V-share elevation, which D-leg alone cannot do) in one run. Safe combination criterion: refactor scope intersection MUST be empty AND atomicity rules within each leg are honoured. DEPRECATES the v1 SP18 spec's "defer SP19" reasoning.

Pearl 5 (post-Phase 9 if smoke fails on D-leg only): pearl_reward_shape_alone_insufficient_when_bootstrap_biased.md

Reward-shape interventions (asymmetric caps, opportunity costs, etc.) cannot overcome a biased TD(λ) self-bootstrap because the bootstrap biases V-share independent of the reward distribution. SP18 D-leg-only would land Hold's Q-distribution width comparable to Long/Short's, but if v_share continues climbing the reward-shape fix doesn't reach the credit-assignment defect. The combined SP18 D+B is the right scope.

---

Open questions for human review

1. DD1: Combined SP18 (D+B), or roll back to D-only / B-only? Recommendation: combined per task input.
2. DD2–DD7: D-leg recommendations confirmed. Reviewer-confirm.
3. DD8: Cubin manifest in Phase 3.1. Reviewer-confirm.
4. B-DD1: Reuse target_params_buf. Reviewer-confirm.
5. B-DD2: DDQN argmax. Reviewer-confirm.
6. B-DD3: Per-branch DDQN argmax (4 separate argmaxes). Reviewer-confirm.
7. B-DD4: PER priority recomputation — no migration. Reviewer-confirm (verified against config.rs:341).
8. B-DD5: Target-net forward at experience collection. Reviewer-confirm.
9. B-DD6: Accept ~30% collection-time slowdown. Profile in PP.5. Reviewer-confirm.
10. B-DD7: Reuse target-net VSN params. Reviewer-confirm (verified against SP14 P4.T1B-iv).
11. B-DD8: Scalar (a) vs distributional (b) target-Q at bootstrap. Recommendation: a (scalar). Reviewer-confirm.
12. B-DD9: TD-error ratio gate [0.5, 5.0]. Reviewer-confirm.
13. B-DD10: Atomic refactor scope (one launch site replacement + new method). Reviewer-confirm.
14. B-DD11: PopArt slot 63 EMA reset at SP18 deployment boundary. Recommendation: yes (cheap insurance). Reviewer-confirm.
15. B-DD12: B-leg HEALTH_DIAG observability. Reviewer-confirm.
16. B-DD13: B-leg 12-slot allocation [493..505). Reviewer-confirm or trim to 6 [493..499).
17. Open Q-A: pearl_reward_shape_drives_quality_over_quantity is already DEPRECATED — confirm Approach D unwinds the recipe (the recipe was per-trade asymmetric cap + min-hold + zero per-bar; D-leg replaces the zero-per-bar leg of that recipe with structural per-bar Hold opp-cost). Audit confirms: Approach D unwinds the per-bar gap; per-trade cap + min-hold remain operative and unchanged.
18. Open Q-B: Promote consumer-audit script to pre-commit hook. CONFIRMED — included as Phase 1 sub-task.
19. Open Q-C: SP19 placeholder. MOOT — absorbed.

---

Self-review checklist

• Problem statement anchored in measured vtj9r data (HD0–HD4 numbers cited; v_share elevation pattern cited).
• D-leg reward audit cites file:line references in current code.
• B-leg TD(λ) audit cites file:line references in current code.
• Four approaches considered (A, B, C, D) with explicit accept/reject reasoning.
• B-leg consumer audit performed: only one production launch site (gpu_experience_collector.rs:4154); PER recomputed at training time (no buffer migration); replay buffer doesn't store q_next; C51 loss target-Q already correct at loss time.
• Graph-capture interaction documented: target-net forward at collection time is graph-captureable per pearl_no_host_branches_in_captured_graph.
• PopArt interaction surfaced (B-DD11): per-fold reset recommended at deployment boundary.
• Consumer audit script is part of Phase 1 (not deferred); promoted to pre-commit hook.
• Every smoke gate cites an empirical baseline (vtj9r HD2/HD3 or b5gmp Fold-1 final or pre/post-fix TD-error ratio).
• DESIGN DECISION markers cover every genuine choice (DD1–8 + B-DD1–13).
• No hardcoded constants for any signal that could derive from data.
• No partial-refactor risk: D-leg deletion + B-leg q_next replacement are atomic within their respective legs.
• Pearl candidates flagged for both pass and fail outcomes (D-only fail surfaces Pearl 5; combined pass surfaces Pearl 4).
• SP19 absorbed (no longer flagged as follow-up).