docs(sp18): copy spec + plan to feat/sp18-combined branch
Mirrors the SP17 PP.1 plan-copy pattern. Plan and spec source-of-truth live on the sister `feat/sp17-dueling-q-network` worktree; this commit copies them onto the SP18 branch so subsequent commits reference local paths. Spec: docs/superpowers/specs/2026-05-08-sp18-reward-shape-hold-attractor-design.md Plan: docs/superpowers/plans/2026-05-08-sp18-reward-shape-hold-attractor.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,668 @@
|
||||
# SP18 — Combined Hold-Attractor Fix: Reward-Shape D-leg + TD(λ) `Q(s')` B-leg (Design Spec)
|
||||
|
||||
> **Status: v2 — UPGRADED to combined SP18+SP19 scope. DRAFT FOR HUMAN REVIEW.**
|
||||
> v1 deferred Approach B (TD(λ) `Q(s')` bootstrap) to SP19. **That deferral is overridden** per the
|
||||
> "no deferrals for complementary fixes" call. Both legs land atomically in one branch / one plan.
|
||||
> Spec must precede plan per the superpowers brainstorming pattern.
|
||||
> Plan: `docs/superpowers/plans/2026-05-08-sp18-reward-shape-hold-attractor.md`.
|
||||
|
||||
---
|
||||
|
||||
## Why combined scope (and why it is safe)
|
||||
|
||||
The vtj9r terminated 5-epoch L40S smoke (sha `0c57a5a31`, terminated 2026-05-09 at HD4 across both folds with confirmed −14%/epoch decay, see Problem Statement) leaves three independently-acting causal mechanisms behind the persistent Q(Hold) attractor:
|
||||
|
||||
| Mechanism | Fixed by | Refactor scope |
|
||||
|---|---|---|
|
||||
| 1. Reactive control loop (controller bandwidth < policy convergence speed) | **D-leg** (Approach D) | reward kernels — `experience_kernels.cu` per-bar Hold sites + new `compute_sp18_hold_opportunity_cost` device fn + atomic delete of `hold_cost_scale_update_kernel` chain |
|
||||
| 2. Distributional shape mismatch (Hold tight, Long/Short wide) | **D-leg** (Approach D) | same as above (raises Hold's Q-target distribution width into Long/Short range) |
|
||||
| 3. TD(λ) self-bootstrap (`q_next = rewards`, never sees `Q(s')`) | **B-leg** (Approach B) | `td_lambda_kernel` consumer — `gpu_experience_collector.rs:4143-4170` + new target-net forward pass on next-states during experience collection (target-net params already exist via `target_params_buf`) |
|
||||
|
||||
**Refactor scopes have empty intersection.** D-leg writes to `r_micro` / `r_opp_cost` reward components inside `experience_kernels.cu`. B-leg writes to the `q_next` argument fed into `td_lambda_kernel` (one launch site at `gpu_experience_collector.rs:4154`). Per `feedback_no_partial_refactor`, atomicity is required *within* each leg's consumer set; *between* the legs there is no shared buffer or contract that breaks if they ship in separate commits inside the same branch.
|
||||
|
||||
**Synergy hypothesis (testable):** B-leg's correct credit assignment makes D-leg's gradient signal meaningful by construction. Under `q_next = rewards` self-bootstrap, the TD-error `δ_t = r_t + γ × q_next_(rewards) − Q(s_t, a_t)` collapses to `δ_t ≈ r_t + γ × r_{t+1} − Q(s_t, a_t)` — independent of which action was actually taken. With `q_next = Q_target(s_{t+1}, a*_{online})` the TD-error becomes `δ_t = r_t + γ × Q_target(s', argmax_a Q_online(s', a)) − Q(s_t, a_t)` — the bootstrap term depends on the next-state action distribution, which depends on the previous action choice. **This may also explain SP17's v_share elevation pattern** (vtj9r showed v_share 0.33 → 0.83 across HD0–HD4): when the bootstrap is action-independent, V absorbs the bias because the per-action centred A has no signal to learn from. Combined SP18 tests this.
|
||||
|
||||
---
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Across SP11→SP17 the model develops a persistent **Q(Hold) attractor**: the directional Q-table prefers Hold to Long/Short despite training rewards that should favour active trading. SP17 (mean-zero advantage centering) eliminated `V`/`A` non-identifiability as a confounder, so the bias is now expressed cleanly through `A_centered_dir[Hold] > A_centered_dir[Long]` rather than `V` masquerading as action preference. **Yet vtj9r `v_share` rose from 0.33 (HD0) to 0.83 (HD4)** — V absorbing what A should have absorbed. Combined SP18 hypothesises this is the rewards-only TD(λ) bootstrap's signature.
|
||||
|
||||
**Terminated smoke `train-multi-seed-vtj9r` (SP17 5-epoch L40S, sha `0c57a5a31`, killed 2026-05-09)**:
|
||||
|
||||
| Epoch | val Sharpe | Hold% | dir_entropy | A_var_dir EMA | v_share | Q(Hold)−Q(Long) |
|
||||
|---|---|---|---|---|---|---|
|
||||
| HD0 | 80.70 | 16.1% | (high) | 0.0023 | 0.33 | ≈ 0.05 |
|
||||
| HD2 | **91.58** | 18.2% | 1.07 | (stable) | 0.51 | ≈ 0.06 |
|
||||
| HD3 | 78.85 | 31.6% | 0.996 | 0.0021 | 0.71 | 0.05–0.06 |
|
||||
| HD4 | (term) | (term) | (term) | (term) | 0.83 | (term) |
|
||||
|
||||
Trajectory: SP17 delays the attractor by 2 epochs and lifts the peak Sharpe by ~13% relative to HD0, but **Hold% climbs +13.4 pts from HD2 to HD3 (peak to first decay)** and **v_share climbs monotonically across all 5 epochs**. Slope ≈ −14% Sharpe / epoch matches `train-multi-seed-b5gmp` Fold 0 HD2→HD3 (pre-SP17 baseline). The architecture is healthy (`A_var_dir` stable, no `V` dominance *at HD0–HD2*), so the gradient-update process is **correctly** learning Hold-as-preferred under the current reward + the v_share elevation pattern is a *symptom* of the rewards-only TD(λ) bootstrap.
|
||||
|
||||
**Conclusions:**
|
||||
- The reward function is asking the model to prefer Hold (D-leg fix).
|
||||
- The TD(λ) bootstrap is structurally biased toward whichever action's reward distribution looks closest to its long-run accumulation, regardless of state-value, eroding V/A separation across epochs (B-leg fix).
|
||||
|
||||
---
|
||||
|
||||
## D-leg Empirical Reward Audit (against current code)
|
||||
|
||||
Audit performed against worktree HEAD `0c57a5a31` (`feat/sp17-dueling`). All file:line citations live.
|
||||
|
||||
### What Hold rewards today
|
||||
|
||||
| Bar regime | Hold-action reward path | Magnitude (current ISV) |
|
||||
|---|---|---|
|
||||
| Voluntary trade close (segment_complete && !trail) on a Hold pick | `r_popart = capped_pnl − shaping × min_hold_penalty` (`experience_kernels.cu:3315`); plus per-bar Hold cost `−ISV[380] × ISV[461]` subtracted from `base_reward` (line 3166) | small, P&L-driven |
|
||||
| Trail-fire close while picking Hold | `r_trail = capped_pnl` (line 3230); same Hold cost subtraction | rare |
|
||||
| Per-bar mid-trade (positioned, no event) on Hold | `r_micro = 0.0f − ISV[380]×ISV[461]` (line 3617–3643) | `0 − HOLD_COST_BASE×scale = −0.005×[1..25] = −0.005..−0.125` |
|
||||
| Per-bar Flat-Hold (no position) | `r_opp_cost = 0.0f − ISV[380]×ISV[461]` (line 3697–3716) | same as above, `−0.005..−0.125` |
|
||||
|
||||
### What Long/Short rewards on the same bars
|
||||
|
||||
- Per-bar mid-trade: `r_micro = 0.0f` (no Hold cost subtracted; line 3617). **Identical to a Hold mid-trade *in dollar terms*** when the per-bar Hold cost is at its floor (`scale=1.0`).
|
||||
- Per-bar Flat-Long entering: not a different per-bar branch — opening a new position is `entering_trade=true` and runs through the segment_complete machinery on its eventual close, paying `compute_lump_sum_opp_cost = −shaping × holding_cost_rate × |position| × hold_time` (`trade_physics.cuh:590`) **at exit**.
|
||||
- Trade close P&L: capped asymmetrically at `[REWARD_NEG_CAP=−10, REWARD_POS_CAP=+5]` (`state_layout.cuh:337–338`), with adaptive ISV[452]/ISV[453] producers tracking p99(winning_returns) × 1.5 (`reward_cap_update_kernel`).
|
||||
|
||||
### Numerical reality check
|
||||
|
||||
Static base: `HOLD_COST_BASE = 0.005` (`sp13_isv_slots.rs:73`), gain `5.0`, floor `0.5×base`, ceil `5.0×base` ⇒ host controller emits cost in `[0.0025, 0.025]` per bar.
|
||||
SP16 P2 multiplier: `ISV[461] ∈ [1, 25]` driven by `(observed_hold_rate − 0.20) / 0.20` overrun. With Wiener-α non-stationary floor 0.4 (post-T3, `min_hold_temperature_update_kernel.cu:121`).
|
||||
|
||||
**Effective per-bar Hold penalty range: `[0.0025, 0.625]`** (when `scale=25`).
|
||||
|
||||
Per-bar reward magnitudes during a typical trade (vol-normalized return, capped at +5/−10): `O(0.01–0.1)`. So at the **floor** of the SP16 P2 ramp the Hold cost is `~5%–25%` of a typical per-bar reward — competitive but small. At the **ceiling** it dominates. The ramp depends entirely on `hold_rate_observed_ema` exceeding 0.20.
|
||||
|
||||
### Where the D-leg asymmetry actually lives
|
||||
|
||||
Three concrete asymmetries between Hold and Long/Short under the present reward shape:
|
||||
|
||||
1. **Cost density.** Hold pays a per-bar cost (line 3637–3643, line 3710–3716). Long/Short pay a lump-sum carrying cost only at trade close. With `td_lambda=0.9` (`config.rs:1427`) and `gamma≈0.95`, the integrated and per-bar cost are mathematically equivalent only at γ=1; at γ<1 the per-bar Hold pricing **discounts less per dollar** than a deferred close-event cost.
|
||||
|
||||
2. **Reward-cap clipping.** Big winners get `+5`, big losers get `−10` (2:1 Kahneman). Hold *participates in the same clamp* via `compute_asymmetric_capped_pnl` only if a position is open. **Flat-Hold reward is dominated by `−ISV[380]×ISV[461]` only**, never by a positive P&L term — and that subtraction is bounded by 0.625/bar. Long/Short collect occasional large bounded wins; Hold collects a steady stream of small bounded penalties. **The Q-target distribution for Hold is a tight low-variance band; the Q-target distribution for Long/Short is a wide high-variance distribution centered near 0**. CQL pessimism plus distributional collapse rewards low-variance branches automatically.
|
||||
|
||||
3. **Bellman-target inflation under TD(λ) self-bootstrap.** This is the B-leg's territory; carved out below.
|
||||
|
||||
---
|
||||
|
||||
## B-leg TD(λ) Self-Bootstrap Audit (against current code)
|
||||
|
||||
Audit performed against worktree HEAD `0c57a5a31`. All file:line citations live.
|
||||
|
||||
### Where the bug lives — single launch site
|
||||
|
||||
`gpu_experience_collector.rs:4135-4174` is the entire production TD(λ) wiring:
|
||||
|
||||
```rust
|
||||
// Line 4135-4143:
|
||||
let td_lambda = config.td_lambda; // = 0.9 from config.rs:1427
|
||||
let max_trace = config.max_trace_length;
|
||||
if td_lambda > 0.01 && max_trace > 1 {
|
||||
let raw_rewards_td = dtod_clone_f32_native(&self.stream, &self.rewards_out, total, "raw_rewards_td")?;
|
||||
let raw_dones_td = dtod_clone_f32_native(&self.stream, &self.done_out, total, "raw_dones_td")?;
|
||||
let q_next = dtod_clone_f32_native(&self.stream, &self.rewards_out, total, "q_next_bootstrap")?;
|
||||
// ^^^^^^^^^^^^^^^^ ← THE BUG
|
||||
// Line 4146-4170: launch td_lambda_kernel with (raw_rewards_td, raw_dones_td, q_next, rewards, dones, ...)
|
||||
}
|
||||
```
|
||||
|
||||
The kernel's own header explicitly licenses this:
|
||||
|
||||
```
|
||||
nstep_kernel.cu:95-96
|
||||
* Requires bootstrapped Q(s') values. When not available, pass raw_rewards
|
||||
* as q_next for a self-bootstrap approximation.
|
||||
```
|
||||
|
||||
### Consumers of the kernel's output
|
||||
|
||||
`td_lambda_kernel` *output* writes to `rewards` (`out_rewards`, line 4158) and `dones` (`out_dones`, line 4159) — these are the SAME buffers passed into the TD(λ)-trained replay batch. So the consumers of `rewards` post-TD(λ) are:
|
||||
|
||||
1. **Replay buffer push** — `rewards` flows into the batch that is appended to the per-fold replay buffer. (One push per experience-collection cycle.)
|
||||
2. **PER initial priorities** — NOT computed at experience-collection time. PER priorities are recomputed at *training* time in `agent.update_priorities_from_td(...)` from current TD-error (`config.rs:341`, `fused_training.rs:2963`). Initial sampling priority is `max_priority` heuristic on first-store. **No buffer schema migration is needed for B-leg.**
|
||||
3. **Loss-time TD-error** — at training time, the C51 / IQN loss kernels read `rewards` from the sampled replay batch and combine with target-net `Q(s')` (already done correctly via `submit_forward_ops_main`'s Pass 2 target-net forward on next_states + `compute_expected_q` projection through `c51_loss_batched`).
|
||||
|
||||
**The bug is confined to the experience-collection-time TD(λ) accumulation.** The C51 loss kernel's distributional Bellman target site (`c51_loss_kernel.cu:188 block_bellman_project_f`) is **already centered correctly** because the loss-time `target_probs` come from a target-net forward pass that already exists at loss time. Mirror the SP17 finding that `c51_loss` was already centered — same pattern here for the *target-Q at loss time*.
|
||||
|
||||
The defect is that the *TD(λ) returns themselves* (the `R^λ` written into `rewards` during experience collection) are computed with rewards-as-q_next, biasing intermediate λ-blended terms toward `E[reward]` rather than `E[Q(s')]`. The loss kernel's outermost Bellman bootstrap is already a target-net Q, but the inner λ-weighted return is rewards-only. **This is the right surgical site to fix**: replace `q_next` at line 4143 with a target-net `compute_expected_q` evaluation on `next_states`.
|
||||
|
||||
### Inputs needed to compute correct `Q(s')` during experience collection
|
||||
|
||||
To wire `q_next = Q_target(s', argmax_a Q_online(s', a))` (DDQN per **B-DD2/B-DD3**) at experience-collection time:
|
||||
|
||||
| Input | Where it lives | Available at collection time? |
|
||||
|---|---|---|
|
||||
| `target_params` (target-net weights) | `gpu_dqn_trainer.rs:5359 target_params_buf` | YES — Polyak-tracked, populated since SP14 P4.T1B-iv |
|
||||
| `next_states` (vol-normalized post-action market state) | built at `gpu_experience_collector.rs:4182 build_next_states_f32` | YES — built in same scope, line 4182, AFTER td_lambda runs today (must be moved BEFORE td_lambda for B-leg) |
|
||||
| `online_params` (online-net weights for DDQN argmax) | exposed via `params_buf_device_ptr()` | YES |
|
||||
| Forward-pass machinery (encoder + GRN + VSN + dueling head) | all kernels exist; orchestration lives in `submit_forward_ops_main` (Pass 2 target-net branch already runs target on next_states at loss time) | NEEDS new wrapper for collection-time invocation |
|
||||
| `compute_expected_q` for the projection | `experience_kernels.cu:4572` | YES — already takes params, inputs, writes per-action E[Q] |
|
||||
|
||||
### Replay buffer schema check
|
||||
|
||||
Searched `agents/`, `trainers/`, `cuda_pipeline/` for `replay_buffer` writes. The push site stores `(state, next_state, reward, done, priority_initial=max_priority)`. **`q_next` is NEVER stored in the buffer** — it's a transient scratch in the experience-collection cycle that gets consumed by `td_lambda_kernel` and then discarded. **Confirmed: no replay buffer schema migration needed.**
|
||||
|
||||
### Graph-capture check
|
||||
|
||||
`exp_fwd_graph` capture region (`gpu_experience_collector.rs:4651–4671`) covers the experience-collection forward pass for action selection. The TD(λ) launch at line 4154 is *outside* the capture region (it runs after the per-step inner loop completes, after `next_states` is built). Adding a target-net forward pass before line 4154 is graph-captureable as long as it lives in the same post-loop scope as today's TD(λ) launch — purely kernel launches, no host branches per `pearl_no_host_branches_in_captured_graph`.
|
||||
|
||||
### PopArt interaction
|
||||
|
||||
PopArt slot 63 normalizes Q-targets at *loss time* (after the trained replay batch is sampled). With the B-leg fix:
|
||||
|
||||
- Pre-fix: trained `R^λ` ≈ rewards-weighted-cumsum; PopArt sees `R^λ + γ^N × Q_target(s_{t+N})` where the inner `R^λ` is rewards-mean-anchored.
|
||||
- Post-fix: trained `R^λ` ≈ target-Q-bootstrapped at every λ-step; PopArt sees the same outer expression but the inner `R^λ` is now Q-mean-anchored (which is the magnitude PopArt was *designed* to track).
|
||||
|
||||
**Risk**: at the moment of switching (HD0 of first SP18 run), PopArt's existing EMA state was tuned to the rewards-distributed `R^λ`. Switching to Q-distributed `R^λ` shifts the input distribution at one step. PopArt has its own α-EMA that adapts; the question is whether it adapts within HD0 or causes a transient. **B-DD11**: per-fold reset of PopArt slot 63 EMA at the SP18 deployment boundary. (Cheap insurance, mirrors SP14 fold-boundary reset patterns.)
|
||||
|
||||
---
|
||||
|
||||
## Approach Considered
|
||||
|
||||
### Approach A — Symmetric trade-cost compensation (REJECTED)
|
||||
|
||||
Charge Hold a structural `position × half_spread × decay` per bar. Rationale: Long/Short already pay an entry/exit half-spread; a Hold action that maintains an open position is also "consuming" the time-decay aspect of having committed capital.
|
||||
|
||||
**Why rejected**: SP16 P2 (`hold_cost_scale_update_kernel`) is exactly this in spirit — adaptive multiplier on the per-bar Hold cost driven by observed Hold-rate overrun. The mechanism failed because it's reactive (closed-loop control with Wiener-α latency), not because it lacked the right semantic concept. Layering another version of the same fix would be the canonical `feedback_no_quickfixes` violation.
|
||||
|
||||
### Approach B — Eligibility traces / TD(λ) `Q(s')` bootstrap fix (ACCEPTED — combined as B-leg)
|
||||
|
||||
**v1 deferred this to SP19. v2 accepts as the B-leg of combined SP18.** See "Why combined scope" above and the Architecture Changes B-leg section below.
|
||||
|
||||
Switching to a real `Q(s')` bootstrap (using the target-net's expected-Q on the realized next-state) gives n-step / TD(λ) returns the correct credit-assignment shape. This is exactly what `pearl_event_driven_reward_density_alignment` recommends ("eligibility traces, n-step returns, HER… are RL-architectural credit-assignment tools, not per-step reward hacking").
|
||||
|
||||
Wiring real `Q(s')` into `td_lambda_kernel` requires:
|
||||
- adding a target-network forward pass on `next_states` *during experience collection* (currently only happens in the loss kernel)
|
||||
- piping the resulting expected-Q back into the `q_next` argument
|
||||
- accepting ~30% collection-time slowdown (extra forward pass; **B-DD6** — H100 has headroom; profile in Pre-Phase to confirm before committing)
|
||||
- per-fold reset of PopArt EMA (**B-DD11**)
|
||||
|
||||
**No replay buffer schema migration.** **No PER priority migration** (PER recomputes from current TD-error at training time). **No C51 loss kernel migration** (already correctly using target-Q).
|
||||
|
||||
### Approach C — Bellman target redesign (REJECTED)
|
||||
|
||||
Center per-step reward at running mean before it enters the Bellman target. Concrete: `r_centered = r − reward_mean_ema; Q-target uses r_centered`.
|
||||
|
||||
**Why rejected**: collides with PopArt (slot 63), the SP11 reward-subsystem controller (slots 340–346), the SP15 DD-asymmetric-reward producer, and the SP12 asymmetric `+5/−10` cap. Each consumes the un-centered reward; centering would either need to flow through *all* of them or live as a post-hoc bias subtraction at the Bellman site (which breaks the SP15 reward-axis "second-pass composer" contract).
|
||||
|
||||
### Approach D — Asymmetric Hold-vs-position reward bound (ACCEPTED — combined as D-leg)
|
||||
|
||||
Per `pearl_audit_unboundedness_for_implicit_asymmetry`, the SP12 `+5/−10` cap restored loss aversion to *position* P&L. **Hold has no equivalent.** A natural per-bar Hold reward is *symmetric around zero* (steady tiny negatives only when the controller ramps). Make the Hold per-bar reward asymmetric by adding a **bounded reinforcement** for the realized opportunity-cost of forgone directional moves, scaled by a controller-driven signal that reflects whether the data *currently* supports trading.
|
||||
|
||||
Concretely (atomic):
|
||||
|
||||
1. **Hold-anchored reward decomposition**. Replace per-bar Hold cost subtraction at the 3 sites with a per-bar Hold reward `r_hold` that is the **sign-flipped** vol-normalized realized `tgt[1]` next-bar log-return — bounded by the same `compute_asymmetric_capped_pnl` machinery the segment-complete path uses. This **lifts Hold's reward distribution into the same dynamic range as Long/Short**.
|
||||
|
||||
2. **Asymmetric Hold reward cap**. New ISV pair `[483..485)` — `HOLD_REWARD_POS_CAP_INDEX` and `HOLD_REWARD_NEG_CAP_INDEX` — driven by a producer mirroring `reward_cap_update_kernel` but tracking `p99(|step_ret|)` of *Long/Short* trade closes only.
|
||||
|
||||
3. **Anchored at vol-normalized log-return only**. Per `pearl_one_unbounded_signal_per_reward`: the unbounded multiplicand is `vol_normalized_return`; everything else is bounded.
|
||||
|
||||
**Why D over the others** (D-leg only — see "Why combined scope" for the multi-leg argument): addresses mechanism not magnitude; structural not reactive; reuses `compute_asymmetric_capped_pnl` and the `reward_cap_update_kernel` pattern; minimal new code surface; does not touch PopArt, SP11 controller, SP15 composer, or PER.
|
||||
|
||||
---
|
||||
|
||||
## Architecture Changes — D-leg (assuming DD1=combined, DD2=a, DD3=b, DD4=a, DD5=b, DD6=as-specified, DD7=c, DD8=Phase 3.1)
|
||||
|
||||
### CUDA changes (D-leg)
|
||||
|
||||
**`crates/ml/src/cuda_pipeline/experience_kernels.cu`**
|
||||
|
||||
1. At the 3 Hold-cost subtraction sites:
|
||||
- line 3162–3166 (segment_complete branch): replace the 4-line `if (dir_idx == DIR_HOLD && isv_signals_ptr != NULL)` block with a single call into `compute_sp18_hold_opportunity_cost(...)` (see point 2). Output flows into `r_micro` (additive, returns 0 if not Hold).
|
||||
- line 3637–3643 (per-bar mid-trade Hold): same.
|
||||
- line 3710–3716 (per-bar flat-Hold): same.
|
||||
|
||||
2. `compute_sp18_hold_opportunity_cost` lives in `trade_physics.cuh` (after `compute_lump_sum_opp_cost` at line 590):
|
||||
- Inputs: `dir_idx`, `tgt[1]` next-bar log-return, `vol_proxy`, `hold_segment_bars`, `isv_pos_cap = isv[483] or fallback 5.0`, `isv_neg_cap = isv[484] or fallback −10.0`, `shaping_scale`.
|
||||
- Output: 0.0 if `dir_idx != DIR_HOLD`; else `compute_asymmetric_capped_pnl(−2.0 × vol_normalized_log_return, isv_neg_cap, isv_pos_cap)` × shaping_scale.
|
||||
|
||||
**`crates/ml/src/cuda_pipeline/hold_reward_cap_update_kernel.cu`** (NEW)
|
||||
|
||||
Mirrors `reward_cap_update_kernel` shape: producer reads p99 of `|step_ret|` over Long/Short trade closes (per DD3=b), multiplies by safety_factor=1.5, writes POS=that and NEG=`−2 × POS` (DD5=b). Wiener-optimal α via Welford accumulators in `[487..493)`. Pearl-A first-observation bootstrap; sentinel `SENTINEL_HOLD_REWARD_POS_CAP=5.0`. Per-fold cadence (slow EMA, similar to position-side cap producer α=0.01).
|
||||
|
||||
**`crates/ml/src/cuda_pipeline/sp14_isv_slots.rs`**
|
||||
|
||||
D-leg ISV section [483..493): 10 slots per DD6. Sentinels, MIN/MAX bounds, EMA α, safety factor. Compile-time assertion `SP18_SLOT_END <= ISV_TOTAL_DIM`.
|
||||
|
||||
**`crates/ml/src/cuda_pipeline/state_layout.cuh`**
|
||||
|
||||
Mirror constants for SP18 slots (POS/NEG sentinels, MIN/MAX bounds). Bump `ISV_TOTAL_DIM` mirror.
|
||||
|
||||
### Rust changes (D-leg)
|
||||
|
||||
**`crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs`**
|
||||
|
||||
- Bump `ISV_TOTAL_DIM` 483 → (B-leg also adds slots; final value pinned in Pre-Phase task PP.2).
|
||||
- Update `layout_fingerprint_seed` per existing pattern.
|
||||
- New `launch_hold_reward_cap_update` method.
|
||||
|
||||
**`crates/ml/src/trainers/dqn/state_reset_registry.rs`**
|
||||
|
||||
- Add fold-reset entries for D-leg slots `[483..493)` and B-leg slots (TBD in PP.2).
|
||||
|
||||
**`crates/ml/src/trainers/dqn/trainer/training_loop.rs`**
|
||||
|
||||
- Per-epoch HEALTH_DIAG emit reading `[483..487)` for D-leg.
|
||||
- DELETE host-side Hold-cost controller block at lines 4171–4188 per DD7(c).
|
||||
- DELETE `launch_hold_cost_scale_update` call at `training_loop.rs:564` per DD7(c).
|
||||
|
||||
**`crates/ml/src/cuda_pipeline/hold_cost_scale_update_kernel.cu`** — DELETE per DD7(c).
|
||||
|
||||
---
|
||||
|
||||
## Architecture Changes — B-leg
|
||||
|
||||
### CUDA changes (B-leg)
|
||||
|
||||
No new kernels. **Reuses `compute_expected_q`** (`experience_kernels.cu:4572`) via a new Rust orchestration path that invokes it with target-net params on next-states during experience collection.
|
||||
|
||||
### Rust changes (B-leg)
|
||||
|
||||
**`crates/ml/src/cuda_pipeline/gpu_experience_collector.rs`**
|
||||
|
||||
1. **Move** `let next_states = build_next_states_f32(...)` from line 4182 to *before* line 4137 (the `if td_lambda > 0.01 && max_trace > 1` block). Required so target-net forward pass can consume `next_states`.
|
||||
|
||||
2. **Add** new method `compute_q_next_target_bootstrap(&mut self, next_states: &CudaSlice<f32>, target_params_ptr: u64, online_params_ptr: u64, B: usize) -> Result<CudaSlice<f32>>` that:
|
||||
- Runs the encoder + dueling forward pass with **online** params on `next_states`, identical to `submit_forward_ops_main`'s Pass 3 (DDQN argmax mirror at `submit_forward_ops_ddqn`). Captures per-branch argmax actions.
|
||||
- Runs the encoder + dueling forward pass with **target** params on `next_states`, identical to Pass 2 inside `submit_forward_ops_main`.
|
||||
- Calls `compute_expected_q` on the target-net distributional outputs to produce per-action E[Q] of size `[B, total_actions=13]`.
|
||||
- Per **B-DD3** per-branch DDQN: gathers `Q_target[a*_online_dir]` for the dir branch and likewise per other 3 branches; sums the 4 selected expected-Q values into a scalar `q_next_per_sample[B]` (matches the SP17 per-branch identifiability claim — each branch's argmax is taken on its own factored Q-table).
|
||||
- Writes the result into a `CudaSlice<f32>` of length `B` (one Q-bootstrap value per sample).
|
||||
|
||||
3. **Replace** line 4143 `let q_next = dtod_clone_f32_native(...)` with `let q_next = self.compute_q_next_target_bootstrap(&next_states, target_params_ptr, online_params_ptr, total)?;`. Drop the `"q_next_bootstrap"` debug name (the buffer is no longer a clone; rename to `"q_next_target"`).
|
||||
|
||||
4. **Distributional flavour of B-DD8**: under C51 the natural "target Q" is per-atom; replacing it with a *scalar* `expected_Q` collapses distributional information at the bootstrap step. Two options:
|
||||
- **B-DD8.a (scalar)**: `q_next[b] = scalar expected_Q from compute_expected_q on target-net at next_states`. Simplest, matches today's `td_lambda_kernel` signature (which expects `float* q_next`, one per sample).
|
||||
- **B-DD8.b (distributional)**: extend `td_lambda_kernel` to consume per-atom target probs and emit `R^λ` per-atom, then project at PopArt. Larger surface; defer.
|
||||
|
||||
**Recommendation: B-DD8.a (scalar) for atomic SP18 scope.** The C51 distributional Bellman target at *loss time* is already correct (uses target-net atom positions via `block_bellman_project_f`). The TD(λ) self-bootstrap defect is about credit assignment along the trace, not about the outer atom projection. Scalar `expected_Q` for the trace-blending is sufficient to fix the credit-assignment bug. Distributional λ-blending would be a follow-up if SP18 5-epoch smoke shows residual atom-distribution drift.
|
||||
|
||||
**`crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs`**
|
||||
|
||||
- Expose `params_buf_device_ptr()` and `target_params_buf_device_ptr()` to experience collector context (already exposed at lines 8847, 11426).
|
||||
- Surface a wrapper `compute_expected_q_for_collection(target_pass: bool)` if needed — likely already accessible via existing pointers.
|
||||
|
||||
**`crates/ml/src/trainers/dqn/trainer/training_loop.rs`**
|
||||
|
||||
- B-leg per-epoch HEALTH_DIAG: emit `td_error_magnitude_ema`, `q_next_target_p99`, `q_next_minus_reward_p99` (all per-batch from the new producer kernel — see B-DD12 below).
|
||||
|
||||
### B-leg ISV slots [493..505)
|
||||
|
||||
12 slots (B-DD13 below):
|
||||
|
||||
```
|
||||
[493] TD_ERROR_MAG_EMA (HEALTH_DIAG observable: avg(|TD_error|) post-fix)
|
||||
[494] Q_NEXT_TARGET_P99 (p99 of target-Q bootstrap value)
|
||||
[495] Q_NEXT_MINUS_REWARD_P99 (sanity: p99 of |q_next_target − rewards| — should be O(1) post-fix)
|
||||
[496] V_SHARE_TREND_DIAG (slope of V_SHARE EMA across last 4 epochs; B-leg synergy probe)
|
||||
[497] POPART_RESET_FLAG (1.0 first epoch post-deployment; 0.0 thereafter — gates the per-fold PopArt EMA reset per B-DD11)
|
||||
[498] TDB_TARGET_MEAN (Welford for q_next_target Wiener-α)
|
||||
[499] TDB_TARGET_M2
|
||||
[500] TDB_DIFF_MEAN
|
||||
[501] TDB_DIFF_M2
|
||||
[502] TDB_PREV_TARGET
|
||||
[503] TDB_SAMPLE_COUNT
|
||||
[504] RESERVED (reserve 1 slot for B-leg follow-up if needed)
|
||||
```
|
||||
|
||||
**Total combined: 22 slots, [483..505). ISV_TOTAL_DIM bumps 483 → 505.**
|
||||
|
||||
---
|
||||
|
||||
## DESIGN DECISION 1 — Combined scope (CONFIRMED)
|
||||
|
||||
**Recommendation: D-leg + B-leg combined, atomic in feat/sp18-reward-shape.** Per task input override of v1's "defer B to SP19".
|
||||
|
||||
> _Human reviewer: confirm combined, or override to D-only / B-only._
|
||||
|
||||
---
|
||||
|
||||
## DESIGN DECISION 2 — Hold opportunity-cost source (CONFIRMED, recommendation a)
|
||||
|
||||
(a) **`tgt[1]` preprocessed next-bar log-return** — already in scope at line ~2380, vol-normalized via the same `atr_norm` chain the segment_complete path uses. Lookahead-clean.
|
||||
|
||||
> _Reviewer: confirm (a)._
|
||||
|
||||
---
|
||||
|
||||
## DESIGN DECISION 3 — Cap producer scope (CONFIRMED, recommendation b)
|
||||
|
||||
(b) **`p99(|step_ret|)` over Long/Short trade closes**. Aligns Hold's reward distribution with policy's *current realized* Long/Short distribution.
|
||||
|
||||
> _Reviewer: confirm (b)._
|
||||
|
||||
---
|
||||
|
||||
## DESIGN DECISION 4 — When to apply Hold opportunity-cost (CONFIRMED, recommendation a)
|
||||
|
||||
(a) **Always (every bar where action_dir==Hold)**, both flat and positioned.
|
||||
|
||||
> _Reviewer: confirm (a)._
|
||||
|
||||
---
|
||||
|
||||
## DESIGN DECISION 5 — Cap asymmetry inversion direction (CONFIRMED, recommendation b)
|
||||
|
||||
(b) **Mirrored** — same as position side, Hold-side `[+5, −10]`. Says "missing a winner is the bigger pain." Drives the policy *toward* engagement.
|
||||
|
||||
> _Reviewer: confirm (b)._
|
||||
|
||||
---
|
||||
|
||||
## DESIGN DECISION 6 — D-leg ISV slot allocation (CONFIRMED, as-specified)
|
||||
|
||||
10 slots [483..493) per the v1 spec.
|
||||
|
||||
> _Reviewer: confirm._
|
||||
|
||||
---
|
||||
|
||||
## DESIGN DECISION 7 — Hold cost retention vs. replacement (CONFIRMED, recommendation c)
|
||||
|
||||
(c) **Replace, but keep the observation chain** (slots 380, 381, 382, the `hold_rate_observer_kernel` per-step writes). Delete only the cost-application sites + the scale producer + the host controller.
|
||||
|
||||
> _Reviewer: confirm (c)._
|
||||
|
||||
---
|
||||
|
||||
## DESIGN DECISION 8 — Cubin manifest registration timing (CONFIRMED, Phase 3.1)
|
||||
|
||||
Register `hold_reward_cap_update_kernel.cubin` in Phase 3.1 alongside the kernel skeleton (avoids broken-build state in Pre-Phase).
|
||||
|
||||
> _Reviewer: confirm._
|
||||
|
||||
---
|
||||
|
||||
## DESIGN DECISION B1 — Target-net infrastructure source (RECOMMENDATION)
|
||||
|
||||
**Use existing `target_params_buf` (Polyak-tracked since SP14 P4.T1B-iv).** No new target-net infra. Verified at `gpu_dqn_trainer.rs:5359` + the target Polyak EMA Pass 2 at loss-time (which proves the buffer is up-to-date relative to online).
|
||||
|
||||
> _Reviewer: confirm._
|
||||
|
||||
---
|
||||
|
||||
## DESIGN DECISION B2 — DDQN argmax at bootstrap (RECOMMENDATION)
|
||||
|
||||
`q_next = Q_target(s', argmax_a Q_online(s', a))`. Standard double-DQN pattern.
|
||||
|
||||
> _Reviewer: confirm._
|
||||
|
||||
---
|
||||
|
||||
## DESIGN DECISION B3 — Per-branch DDQN argmax (RECOMMENDATION)
|
||||
|
||||
Take 4 separate argmaxes — one each for `dir`, `mag`, `ord`, `urg` factored Q-tables. Matches SP17 per-branch identifiability claim (each branch is its own Q-axis; combining via sum after argmax is the SP17 contract). Sum the 4 selected target-Q values into the scalar `q_next` per sample.
|
||||
|
||||
> _Reviewer: confirm. Alternative: single global argmax on the 13-action joint, which loses the SP17 factored-Q semantics — REJECTED._
|
||||
|
||||
---
|
||||
|
||||
## DESIGN DECISION B4 — PER priority recomputation (CONFIRMED)
|
||||
|
||||
PER priorities are **already** recomputed at training time from current TD-error (`config.rs:341`); not stored in buffer or precomputed in experience collector. **No PER migration needed.** The new TD(λ)-corrected `rewards` flowing into the buffer feed naturally into the existing priority recomputation. **B9 oracle test required** to confirm the post-fix priority *distribution* is reasonable.
|
||||
|
||||
> _Reviewer: confirm._
|
||||
|
||||
---
|
||||
|
||||
## DESIGN DECISION B5 — Target-net forward pass during experience collection (CONFIRMED)
|
||||
|
||||
Per the audit, `td_lambda_kernel` consumes `q_next` at collection time. To replace with a real `Q(s')`, we MUST run the target-net forward pass at collection time (not loss time). Adds one extra forward pass per experience-collection cycle.
|
||||
|
||||
> _Reviewer: confirm._
|
||||
|
||||
---
|
||||
|
||||
## DESIGN DECISION B6 — Accept ~30% collection-time slowdown (RECOMMENDATION)
|
||||
|
||||
Profile in Pre-Phase task PP.5 (NEW) — measure end-to-end experience collection time on H100 before merging Phase 4 to confirm headroom. If >50% slowdown, fall back to **B-DD8.a** scalar-only and skip per-atom target-Q (already the recommendation). If still >50%, halt and convene human review.
|
||||
|
||||
> _Reviewer: confirm._
|
||||
|
||||
---
|
||||
|
||||
## DESIGN DECISION B7 — Reuse target-net VSN params (CONFIRMED)
|
||||
|
||||
Existing `target_params_buf` includes VSN params (per SP14 P4.T1B-iv migration). The target-net forward pass on next_states uses the same VSN gates as today's loss-time Pass 2 — same `vsn_gated_next_states_buf`. No new VSN handling.
|
||||
|
||||
> _Reviewer: confirm._
|
||||
|
||||
---
|
||||
|
||||
## DESIGN DECISION B8 — Per-atom vs scalar target-Q (RECOMMENDATION a)
|
||||
|
||||
**B-DD8.a (scalar) for atomic SP18.** Distributional flavour deferred to follow-up if 5-epoch smoke shows atom-distribution drift. See B-leg Architecture Changes section above for full reasoning.
|
||||
|
||||
> _Reviewer: confirm a._
|
||||
|
||||
---
|
||||
|
||||
## DESIGN DECISION B9 — TD-error magnitude smoke gate (RECOMMENDATION)
|
||||
|
||||
`avg(|TD_error|) ratio post-fix / pre-fix ∈ [0.5, 5.0]` at HD0–HD1 (sanity check that bootstrap change didn't blow up loss magnitude). HD0 baseline measured in Phase 0 diagnostic pre-fix; HD0 measurement post-fix taken in Phase 7 smoke; ratio computed per-fold per-epoch.
|
||||
|
||||
> _Reviewer: confirm._
|
||||
|
||||
---
|
||||
|
||||
## DESIGN DECISION B10 — Atomic refactor scope (CONFIRMED)
|
||||
|
||||
Atomic refactor includes: `gpu_experience_collector.rs:4143-4170` (the line 4143 q_next replacement + the target-net forward pass before line 4137) + the new `compute_q_next_target_bootstrap` method + the `next_states` build-site relocation + B-leg ISV slots and HEALTH_DIAG emit. **Audit script in Phase 1 surfaces additional consumers.** PER priority computation and C51 loss kernel migration NOT needed (per B-DD4 and B-leg audit).
|
||||
|
||||
> _Reviewer: confirm._
|
||||
|
||||
---
|
||||
|
||||
## DESIGN DECISION B11 — PopArt slot 63 EMA reset at deployment boundary (NEW, RECOMMENDATION)
|
||||
|
||||
Per audit, switching `q_next` source from rewards-distributed to Q-distributed shifts the input distribution to PopArt at the SP18 deployment. PopArt's α-EMA will adapt but may take 1+ epochs.
|
||||
|
||||
**Recommendation**: per-fold reset of PopArt slot 63 EMA at the SP18 deployment boundary, gated by new ISV slot 497 `POPART_RESET_FLAG`. Slot 497 is set to 1.0 by host at first SP18 epoch and 0.0 thereafter (one-shot). On reset, PopArt EMA reinitializes to identity normalization (μ=0, σ=1). Cheap insurance.
|
||||
|
||||
> _Reviewer: confirm, or skip if you'd rather rely on PopArt's natural adaptation._
|
||||
|
||||
---
|
||||
|
||||
## DESIGN DECISION B12 — B-leg HEALTH_DIAG observability (NEW, RECOMMENDATION)
|
||||
|
||||
Emit per-epoch:
|
||||
- `td_error_mag_ema` (slot 493): avg(|TD-error|) computed post-fix
|
||||
- `q_next_target_p99` (slot 494): bound check on the new bootstrap
|
||||
- `q_next_minus_reward_p99` (slot 495): sanity — should be O(1), not O(0) (proves the fix is doing something) and not O(100) (proves it isn't blowing up)
|
||||
- `v_share_trend_diag` (slot 496): slope of V_SHARE across last 4 epochs (should *flatten* or *reverse* under the synergy hypothesis)
|
||||
|
||||
> _Reviewer: confirm._
|
||||
|
||||
---
|
||||
|
||||
## DESIGN DECISION B13 — B-leg ISV slot allocation (NEW, RECOMMENDATION)
|
||||
|
||||
12 slots `[493..505)` per the table in "B-leg ISV slots" above. Combined SP18 total: 22 slots `[483..505)`. ISV_TOTAL_DIM bumps 483 → 505.
|
||||
|
||||
> _Reviewer: confirm 12-slot allocation, or trim B-leg diagnostics to live on HEALTH_DIAG-only readout (leave only [493..498) — net 6 B-leg slots, total 16 [483..499))._
|
||||
|
||||
---
|
||||
|
||||
## Behavioral Test Predictions
|
||||
|
||||
Tests live at `crates/ml/tests/sp18_hold_reward_oracle_tests.rs` (D-leg) and `crates/ml/tests/sp18_td_lambda_q_next_oracle_tests.rs` (B-leg). Each is a single TDD cycle.
|
||||
|
||||
### D-leg Tests
|
||||
|
||||
#### T1 — CPU oracle for `compute_sp18_hold_opportunity_cost`
|
||||
|
||||
Given: `next_log_return = +0.005`, `vol_proxy = 0.0001`, `pos_cap = 5.0`, `neg_cap = -10.0`, `shaping_scale = 1.0`. Expected: returns negative scalar bounded in `[-10, 0]`. Negative log-return inputs return positive scalar bounded in `[0, +5]`. `dir_idx != DIR_HOLD` returns 0.0 exactly.
|
||||
|
||||
#### T2 — GPU oracle parity with CPU
|
||||
|
||||
Same inputs on `MappedF32Buffer`; bit-identical to ε=1e-6.
|
||||
|
||||
#### T3 — Hold reward distribution on synthetic uniform-edge market
|
||||
|
||||
Synthetic `tgt[1] ~ Uniform[-0.005, +0.005]`. After 1000 Hold picks: `mean(r_hold_opp) ≈ 0`, `var(r_hold_opp) > 0.001`.
|
||||
|
||||
#### T4 — Hold reward distribution on positive-edge synthetic market
|
||||
|
||||
`tgt[1] ~ N(+0.001, 0.002)`. After 1000 Hold picks: `mean(r_hold_opp) < 0`, `mean(r_long_pick - r_hold_opp) > 0` for Long picks on the same bars.
|
||||
|
||||
#### T5 — Cap producer drives slots [483, 484]
|
||||
|
||||
After 100 simulated Long/Short close events with `step_ret ∈ N(0, 0.01)`: `ISV[483] ≈ p99(|step_ret|) × 1.5`, `ISV[484] = -2 × ISV[483]`, both within bounds.
|
||||
|
||||
#### T6 — End-to-end on converged b5gmp checkpoint
|
||||
|
||||
Hold expected-Q variance over actions ≥ 0.5× Long expected-Q variance over actions; no NaN; cap producers populate within 1 launch.
|
||||
|
||||
#### T7 — Scaffold-level: SP13/SP16 deletion compiles + tests pass
|
||||
|
||||
Atomic delete leaves the build green and existing tests pass.
|
||||
|
||||
### B-leg Tests
|
||||
|
||||
#### T8 — CPU oracle for DDQN per-branch argmax bootstrap
|
||||
|
||||
Pure math: given synthetic `Q_online[B, 13]` and `Q_target[B, 13]` (factored 4+3+3+3), compute `q_next_per_sample[b]` as `Σ_branch Q_target[b, argmax_a_in_branch Q_online[b, a]]`. Lock the contract: dir-branch argmax in `[0..4)`, mag in `[4..7)`, ord in `[7..10)`, urg in `[10..13)`.
|
||||
|
||||
#### T9 — GPU oracle parity with CPU (B-leg)
|
||||
|
||||
Same inputs on `MappedF32Buffer`; bit-identical to ε=1e-6.
|
||||
|
||||
#### T10 — TD-error magnitude before vs after fix
|
||||
|
||||
Synthetic 100-step rollout: pre-fix `q_next = rewards`; post-fix `q_next = Q_target(s', argmax_online)`. Verify `avg(|TD_error|) ratio ∈ [0.5, 5.0]` (B-DD9). On a converged checkpoint replay buffer this guards against the bootstrap change blowing up (or zeroing out) the loss.
|
||||
|
||||
#### T11 — Target-net forward consistency
|
||||
|
||||
Run the new `compute_q_next_target_bootstrap` and compare its output to the target-net forward result that already exists at loss time on the same `(next_states, target_params)`. Must be bit-identical (same kernel, same params, same input). Locks "no second source of truth".
|
||||
|
||||
#### T12 — PER priority distribution post-fix
|
||||
|
||||
After 1 training step on a converged-checkpoint replay buffer with the SP18 fix wired: PER priority distribution shape (mean, p50, p99) within ±2× of pre-fix. No degenerate "all priorities equal" or "single priority dominates" signature.
|
||||
|
||||
#### T13 — Graph-capture compatibility
|
||||
|
||||
Capture `exp_fwd_graph` with the new target-net forward pass on `next_states`. Replay 10× and verify per-replay determinism. Locks B-leg Architecture point 1 (move `next_states` build before TD(λ) launch) doesn't break capture.
|
||||
|
||||
---
|
||||
|
||||
## Smoke Gates (anchored to vtj9r and b5gmp empirical baselines)
|
||||
|
||||
Phase 9 5-epoch L40S smoke. Gates anchored to `train-multi-seed-vtj9r` (terminated 2026-05-09, sha 0c57a5a31) and `train-multi-seed-b5gmp` Fold 1 final (sha 641aa0dfd).
|
||||
|
||||
### D-leg gates
|
||||
|
||||
| Gate | Threshold | Source |
|
||||
|---|---|---|
|
||||
| **val Sharpe HD2 (peak)** | **≥ 91.58** | vtj9r HD2 best Sharpe — SP18 must not regress |
|
||||
| **val Sharpe HD3 (post-peak)** | **≥ 80** | vtj9r HD3 = 78.85; b5gmp Fold-1 = 55.55 (failure) |
|
||||
| **Hold% HD3** | **≤ 25%** | vtj9r HD3 = 31.6% (climbing); SP18 must keep below |
|
||||
| **dir_entropy HD3** | **≥ 1.0** | vtj9r HD3 = 0.996 |
|
||||
| Hold opp-cost fire-rate (HD3) | ≥ 0.80 | structural — D-leg reward must fire |
|
||||
| `r_hold_opp` distribution width | mean(\|r_hold_opp\|) > 0.5 × mean(\|r_popart\|) | structural — distribution match |
|
||||
| ISV[483] settled | ∈ [0.01, 5.0] by HD2 | adaptive producer alive by HD2 |
|
||||
| Q(Hold) − Q(Long) gap | ≤ 0.10 throughout | b5gmp Fold-1 = 0.148 (pre-SP17 failure) |
|
||||
| no SP13/SP16 dead-code refs | grep returns 0 | DD7(c) atomic delete clean |
|
||||
|
||||
### B-leg gates
|
||||
|
||||
| Gate | Threshold | Source |
|
||||
|---|---|---|
|
||||
| **avg(\|TD-error\|) ratio post-fix / pre-fix HD0** | **∈ [0.5, 5.0]** | B-DD9 sanity |
|
||||
| **v_share trend HD0→HD4** | **monotone decreasing OR flat** | vtj9r showed monotone increasing 0.33→0.83 — SP18 must reverse or flatten |
|
||||
| q_next_target_p99 (HD2) | ≤ 50.0 | bound check; reward cap is `+5/-10` so target Q ≤ ~50 reasonable |
|
||||
| q_next_minus_reward_p99 (HD2) | > 0.1 AND < 100.0 | structural — fix is doing something but not exploding |
|
||||
| PER priority p99 / p50 ratio (HD2) | ≤ 100.0 | distribution sanity |
|
||||
| **PopArt EMA stable (HD0+1 to HD3)** | **slot 63 not increasing >2× HD0 value** | PopArt should re-converge after B-DD11 reset; large drift = adaptation failure |
|
||||
|
||||
### Combined gates
|
||||
|
||||
| Gate | Threshold | Source |
|
||||
|---|---|---|
|
||||
| HEALTH_DIAG never NaN | always | always |
|
||||
| no train hang | always | always |
|
||||
| audit script returns clean | grep returns 0 dead refs | DD7(c) + B-leg consumer migration |
|
||||
|
||||
**Phase 10 30-epoch full validation** uses same gates; thresholds tightened on `val Sharpe Fold-1 final ≥ 80`, `Hold% Fold-1 final ≤ 25%`, `dir_entropy Fold-1 final ≥ 0.95`, `Q(Flat)−Q(Long) gap ≤ 0.10`.
|
||||
|
||||
If smoke fails any **bold** gate, halt and convene human review before 30-epoch dispatch.
|
||||
|
||||
---
|
||||
|
||||
## Hard rules (apply to every task in the plan)
|
||||
|
||||
- **Pre-dispatch consumer audit script** (Phase 1 first task) — surfaces missed consumers BEFORE migration. Promoted to a **pre-commit hook** (Open Q-B confirmed). Generalizable beyond SP18.
|
||||
- **Outcome-anchored smoke gates** — every threshold cites an empirical baseline.
|
||||
- **DESIGN DECISION markers** for every genuine choice (DD1–8 + B-DD1–13).
|
||||
- **TDD** per task — failing test first.
|
||||
- **No partial refactor** within each leg.
|
||||
- **ISV-driven adaptive bounds** — pos/neg caps via producer.
|
||||
- **No CPU compute** — opp-cost producer + B-leg target-net forward + reward computation pure GPU.
|
||||
- **Mapped-pinned only** for diagnostic readback.
|
||||
- **Pearl candidate per phase**.
|
||||
|
||||
---
|
||||
|
||||
## Memory pearl candidates (drafts)
|
||||
|
||||
### Pearl 1 (post-Phase 1 atomic delete): `pearl_replace_reactive_controller_with_structural_signal.md`
|
||||
|
||||
> When a reactive control loop on an emergent symptom (per-bar Hold cost driven by observed Hold-rate overrun) fails to keep up with the policy's per-batch convergence, replace it with a *structural* per-step signal that is immediately present in the reward shape regardless of policy state. SP13/SP16 raised Hold cost reactively after the policy had already over-Held; SP18 D-leg makes Hold's reward distribution structurally comparable to Long/Short by paying the realized log-return as opportunity cost on every Hold bar — distributional-collapse advantage eliminated at source, no controller bandwidth required.
|
||||
|
||||
### Pearl 2 (post-Phase 5 if synergy holds): `pearl_target_net_q_during_experience_collection_for_correct_credit_assignment.md`
|
||||
|
||||
> When TD(λ) / n-step return accumulators run during experience collection, the bootstrap value must be a real `Q(s')` (target-net evaluated on next-states), not the rewards stream. Self-bootstrap (`q_next = rewards`) collapses TD-error to a state-independent reward-tracker which biases the learning signal toward whichever action's reward distribution best matches its long-run accumulation. Symptoms include monotone V-share elevation (V absorbing the action-independent bias because A has no signal to learn) and reward-shape attractors that resist per-step controller fixes. SP18 B-leg fix: run target-net forward on `next_states` during experience collection, gather DDQN per-branch argmax E[Q] from the target-net's distributional output, feed as `q_next` into `td_lambda_kernel`. Adds ~30% to collection time on H100 (acceptable headroom). No replay buffer schema change; no PER priority change (recomputed at training time anyway); no C51 loss kernel change (loss-time target-Q already correct).
|
||||
|
||||
### Pearl 3 (post-Phase 9 if smoke passes): `pearl_distributional_shape_match_for_action_neutrality.md`
|
||||
|
||||
> When one action's Q-distribution is structurally tighter than another's, CQL pessimism + C51 atom occupancy patterns favour the tight branch independent of expected value. To avoid the resulting attractor, design rewards so each action's Q-target distribution has comparable width and tails. SP18 D-leg example: Hold-action Q-targets had width O(0.005) (per-bar penalty band), Long/Short Q-targets had width O(5) (capped trade P&L). Adding bounded opportunity-cost reward to Hold lifted Hold's distribution width to comparable scale.
|
||||
|
||||
### Pearl 4 (post-Phase 9 if combined fix lands): `pearl_complementary_fixes_no_deferrals.md`
|
||||
|
||||
> When two distinct mechanisms drive the same observed pathology, do NOT defer the more-architecturally-honest fix in favour of the simpler one. Fix both in one branch, atomically, so that synergy effects are exposed within the same smoke run. SP18 example: D-leg (reward-shape) and B-leg (TD(λ) bootstrap) were initially scoped as SP18 + SP19. Refactor scopes had empty intersection (D-leg writes reward components, B-leg replaces a `dtod_clone_f32_native` argument); combined ship was safe per `feedback_no_partial_refactor`. Splitting would have required two L40S validation cycles; combined exposed the synergy hypothesis (B-leg fix reverses V-share elevation, which D-leg alone cannot do) in one run. Safe combination criterion: refactor scope intersection MUST be empty AND atomicity rules within each leg are honoured. **DEPRECATES** the v1 SP18 spec's "defer SP19" reasoning.
|
||||
|
||||
### Pearl 5 (post-Phase 9 if smoke fails on D-leg only): `pearl_reward_shape_alone_insufficient_when_bootstrap_biased.md`
|
||||
|
||||
> Reward-shape interventions (asymmetric caps, opportunity costs, etc.) cannot overcome a biased TD(λ) self-bootstrap because the bootstrap biases V-share independent of the reward distribution. SP18 D-leg-only would land Hold's Q-distribution width comparable to Long/Short's, but if v_share continues climbing the reward-shape fix doesn't reach the credit-assignment defect. The combined SP18 D+B is the right scope.
|
||||
|
||||
---
|
||||
|
||||
## Open questions for human review
|
||||
|
||||
1. **DD1**: Combined SP18 (D+B), or roll back to D-only / B-only? **Recommendation: combined per task input.**
|
||||
2. **DD2–DD7**: D-leg recommendations confirmed. Reviewer-confirm.
|
||||
3. **DD8**: Cubin manifest in Phase 3.1. Reviewer-confirm.
|
||||
4. **B-DD1**: Reuse `target_params_buf`. Reviewer-confirm.
|
||||
5. **B-DD2**: DDQN argmax. Reviewer-confirm.
|
||||
6. **B-DD3**: Per-branch DDQN argmax (4 separate argmaxes). Reviewer-confirm.
|
||||
7. **B-DD4**: PER priority recomputation — no migration. Reviewer-confirm (verified against `config.rs:341`).
|
||||
8. **B-DD5**: Target-net forward at experience collection. Reviewer-confirm.
|
||||
9. **B-DD6**: Accept ~30% collection-time slowdown. Profile in PP.5. Reviewer-confirm.
|
||||
10. **B-DD7**: Reuse target-net VSN params. Reviewer-confirm (verified against SP14 P4.T1B-iv).
|
||||
11. **B-DD8**: Scalar (a) vs distributional (b) target-Q at bootstrap. **Recommendation: a (scalar).** Reviewer-confirm.
|
||||
12. **B-DD9**: TD-error ratio gate `[0.5, 5.0]`. Reviewer-confirm.
|
||||
13. **B-DD10**: Atomic refactor scope (one launch site replacement + new method). Reviewer-confirm.
|
||||
14. **B-DD11**: PopArt slot 63 EMA reset at SP18 deployment boundary. **Recommendation: yes (cheap insurance).** Reviewer-confirm.
|
||||
15. **B-DD12**: B-leg HEALTH_DIAG observability. Reviewer-confirm.
|
||||
16. **B-DD13**: B-leg 12-slot allocation [493..505). Reviewer-confirm or trim to 6 [493..499).
|
||||
17. **Open Q-A**: `pearl_reward_shape_drives_quality_over_quantity` is already DEPRECATED — confirm Approach D unwinds the recipe (the recipe was per-trade asymmetric cap + min-hold + zero per-bar; D-leg replaces the zero-per-bar leg of that recipe with structural per-bar Hold opp-cost). **Audit confirms: Approach D unwinds the per-bar gap; per-trade cap + min-hold remain operative and unchanged.**
|
||||
18. **Open Q-B**: Promote consumer-audit script to pre-commit hook. **CONFIRMED — included as Phase 1 sub-task.**
|
||||
19. **Open Q-C**: SP19 placeholder. **MOOT — absorbed.**
|
||||
|
||||
---
|
||||
|
||||
## Self-review checklist
|
||||
|
||||
- [x] Problem statement anchored in measured vtj9r data (HD0–HD4 numbers cited; v_share elevation pattern cited).
|
||||
- [x] D-leg reward audit cites file:line references in current code.
|
||||
- [x] B-leg TD(λ) audit cites file:line references in current code.
|
||||
- [x] Four approaches considered (A, B, C, D) with explicit accept/reject reasoning.
|
||||
- [x] B-leg consumer audit performed: only one production launch site (`gpu_experience_collector.rs:4154`); PER recomputed at training time (no buffer migration); replay buffer doesn't store `q_next`; C51 loss target-Q already correct at loss time.
|
||||
- [x] Graph-capture interaction documented: target-net forward at collection time is graph-captureable per `pearl_no_host_branches_in_captured_graph`.
|
||||
- [x] PopArt interaction surfaced (B-DD11): per-fold reset recommended at deployment boundary.
|
||||
- [x] Consumer audit script is part of Phase 1 (not deferred); promoted to pre-commit hook.
|
||||
- [x] Every smoke gate cites an empirical baseline (vtj9r HD2/HD3 or b5gmp Fold-1 final or pre/post-fix TD-error ratio).
|
||||
- [x] DESIGN DECISION markers cover every genuine choice (DD1–8 + B-DD1–13).
|
||||
- [x] No hardcoded constants for any signal that could derive from data.
|
||||
- [x] No partial-refactor risk: D-leg deletion + B-leg `q_next` replacement are atomic within their respective legs.
|
||||
- [x] Pearl candidates flagged for both pass and fail outcomes (D-only fail surfaces Pearl 5; combined pass surfaces Pearl 4).
|
||||
- [x] SP19 absorbed (no longer flagged as follow-up).
|
||||
Reference in New Issue
Block a user