feat(sp20): Phase 3 Task 3.2 — Hold opportunity-cost dual emission (Component 2)
Lands the per-bar Hold opportunity-cost dual emission at
experience_env_step's per-bar branches (positioned-non-event ~3650 +
flat-non-event ~3737), atomically with the trade-close consumer
wiring at the segment_complete branch (~3289 — replaces the Phase 2
Task 2.2 forward-reference placeholder `hold_baseline = 0.0f`) per
`feedback_no_partial_refactor`.
Spec §4.2 dual-emission contract:
per_bar_opp_cost = -aux_conf × cost_scale
Path 1 (Hold-only): r_micro/r_opp_cost += per_bar_opp_cost
- ISV[HOLD_REWARD_EMA]
(centered for Q-target stability)
Path 2 (always): hold_baseline_buffer[env, t % 30] = per_bar_opp_cost
(uncentered, consumed at trade close)
At trade close:
hold_baseline = sp20_sum_hold_baseline_over_trade(buf, env, 30,
current_t,
segment_hold_time)
alpha = R_event - hold_baseline # replaces Phase 2 placeholder
The summation walks backwards `min(30, segment_hold_time)` slots from
`(current_t - 1) % 30` in env i's row of the per-env circular buffer.
The trade-close bar's segment_complete branch does NOT write to the
buffer, so the most recent per-bar write is at current_t - 1.
Layout decision (plan errata Gap 9): per-env stride
`[N_envs × HOLD_BASELINE_BUFFER_SIZE = 30]` row-major. The kernel is
per-env-parallel; a single global ring would interleave bars across
envs and break trade-attribution semantic.
Trade-open tracking decision (plan errata Gap 10): NO new state slot
needed. The existing `segment_hold_time` (= saved_hold_time captured
before PS_HOLD_TIME reset at experience_kernels.cu:2618) plus
current_t suffice — walk backwards in the buffer.
Replaces SP18 D-leg `compute_sp18_hold_opportunity_cost` calls at
experience_kernels.cu:3672 and :3783. The device function in
trade_physics.cuh:655 is RETAINED — still called by
sp18_hold_opp_test_kernel.cu (oracle test surface). Per
`feedback_no_stubs`: only production callers migrate; the helper
stays for the test surface.
New device helpers (sp20_hold_baseline.cuh):
- sp20_compute_per_bar_aux_conf_k2(logits) — bit-identical to the
formula in sp20_stats_compute_kernel.cu Pass A.
- sp20_sum_hold_baseline_over_trade(buf, env, size, current_t,
segment_hold_time) — walks backwards with modulo wrap-around.
New buffer + reset infrastructure:
- GpuExperienceCollector.hold_baseline_buffer field
- HOLD_BASELINE_BUFFER_SIZE = 30 constant (re-exported)
- StateResetRegistry FoldReset entry + invariant test
- reset_named_state dispatch arm
Six new kernel args appended to experience_env_step signature:
aux_logits_per_env, hold_baseline_buffer, hold_buffer_size, aux_k,
hold_cost_scale_idx, hold_reward_ema_idx. All NULL-tolerant.
HOLD_REWARD_EMA forward-reference: ISV[516] is updated by
sp20_emas_compute_kernel reading per_bar_hold_reward from
sp20_aggregate_inputs_kernel which currently emits 0.0f as a Phase
3.2 forward-ref placeholder (line 292). The parallel `sp20-phase-2-fix`
agent owns wiring the real producer; Task 3.2's centering math
references the ISV slot (not a hardcoded 0), so when the parallel
branch lands, EMA starts updating and centering becomes load-bearing
automatically. No additional Task 3.2 work needed post-merge.
Tests (sp20_hold_baseline_test.rs, 4 GPU oracle tests):
1. aux_conf_k2_bounds_and_fixed_points — bounds, uniform/saturated/
spec-warmup/spec-confident/symmetry fixed points.
2. sum_hold_baseline_over_trade_indexing — basic indexing,
hold_time>size clamp, defensive guards, modulo wrap-around.
3. sum_hold_baseline_per_env_stride — env-stride correctness across
row-major buffer.
4. dual_emission_hold_vs_non_hold — full Path 1 + Path 2 contract,
mirrors plan §3.2 reference fixture.
Verification:
SQLX_OFFLINE=true cargo check -p ml --tests --features cuda # green
SQLX_OFFLINE=true cargo test -p ml --lib sp20 # 21/21 pass
SQLX_OFFLINE=true cargo build -p ml # cubin compile
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
@@ -2,7 +2,227 @@
|
||||
|
||||
**Status:** Populated during Plan 1 Task 6 (A.5 orphan audit). Updated on every commit per Invariant 7.
|
||||
|
||||
## 2026-05-10 — SP20 Phase 2 Task 2.2-fix: post-fix runtime diagnostic infrastructure (read-back + producer test scaffold)
|
||||
## 2026-05-10 — SP20 Phase 3 Task 3.2: Hold opportunity-cost dual emission (Component 2, atomic)
|
||||
|
||||
Lands the per-bar Hold opportunity-cost dual emission at the
|
||||
`experience_env_step` per-bar branches (positioned-non-event ~line
|
||||
3650 + flat-non-event ~line 3737), atomically with the trade-close
|
||||
consumer wiring at the segment_complete branch (~line 3289 — replaces
|
||||
the Phase 2 Task 2.2 forward-reference placeholder
|
||||
`hold_baseline = 0.0f`) per `feedback_no_partial_refactor`.
|
||||
|
||||
### Spec §4.2 dual-emission contract
|
||||
|
||||
```
|
||||
Per bar i:
|
||||
aux_conf_i = max(softmax(aux_logits_i)) - 0.5 # K=2
|
||||
cost_scale = ISV[HOLD_COST_SCALE_INDEX = 513] # adaptive [0.01, 0.5]
|
||||
|
||||
per_bar_opp_cost_i = -aux_conf_i × cost_scale
|
||||
|
||||
# Path 1 — real reward, fires only on Hold action:
|
||||
if dir_idx == DIR_HOLD:
|
||||
r_micro_or_opp_cost += per_bar_opp_cost_i - ISV[HOLD_REWARD_EMA_INDEX = 516]
|
||||
(centered for Q-target stability)
|
||||
|
||||
# Path 2 — counterfactual buffer, ALWAYS recorded:
|
||||
hold_baseline_buffer[i, current_t % 30] = per_bar_opp_cost_i
|
||||
(uncentered, consumed by Component 1 trade-close)
|
||||
|
||||
At trade close (segment_complete):
|
||||
hold_baseline = sp20_sum_hold_baseline_over_trade(
|
||||
hold_baseline_buffer, env=i, hold_buffer_size=30,
|
||||
current_t, segment_hold_time
|
||||
) # walks backwards
|
||||
# min(30, hold_time) bars
|
||||
# in env i's row
|
||||
alpha = R_event - hold_baseline # replaces Phase 2 Task 2.2
|
||||
# placeholder = 0.0f
|
||||
```
|
||||
|
||||
### Replaced sites
|
||||
|
||||
1. **Trade-close site** (experience_kernels.cu line ~3289) —
|
||||
`const float hold_baseline = 0.0f;` Phase 2 Task 2.2 placeholder
|
||||
replaced with the real `sp20_sum_hold_baseline_over_trade(...)`
|
||||
call. NULL-tolerant: when `hold_baseline_buffer == NULL` (test
|
||||
scaffolds), falls back to 0.0f (preserves Phase 2 placeholder
|
||||
behaviour bit-for-bit).
|
||||
|
||||
2. **Per-bar positioned-non-event branch** (line ~3650) — SP18 D-leg
|
||||
`compute_sp18_hold_opportunity_cost(...)` call replaced with the
|
||||
dual emission writing `r_micro` (rc[3] component) and
|
||||
`hold_baseline_buffer[i, t % 30]`.
|
||||
|
||||
3. **Per-bar flat-non-event branch** (line ~3737) — SP18 D-leg call
|
||||
replaced with the dual emission writing `r_opp_cost` (rc[4]
|
||||
component) and `hold_baseline_buffer[i, t % 30]`.
|
||||
|
||||
The SP18 D-leg device function (`compute_sp18_hold_opportunity_cost`
|
||||
at `trade_physics.cuh:655`) is RETAINED — still called by
|
||||
`sp18_hold_opp_test_kernel.cu` (the SP18 oracle test surface). Per
|
||||
`feedback_no_stubs`, only the production callers are migrated; the
|
||||
helper itself stays for the test surface.
|
||||
|
||||
### New device helpers (`sp20_hold_baseline.cuh`)
|
||||
|
||||
- `sp20_compute_per_bar_aux_conf_k2(logits)`:
|
||||
Per-row K=2 peak-confidence above the K=2 uniform mean.
|
||||
`aux_conf = max(softmax(logits)) - 0.5` ∈ [0, 0.5]. Numerically
|
||||
stable via row-max subtraction — bit-identical to the formula in
|
||||
`sp20_stats_compute_kernel.cu` Pass A.
|
||||
|
||||
- `sp20_sum_hold_baseline_over_trade(buf, env, size, current_t,
|
||||
segment_hold_time)`:
|
||||
Sum the per-env circular buffer over the most recent `min(size,
|
||||
segment_hold_time)` slots BEFORE `current_t`. Walks backwards from
|
||||
`(current_t - 1) % size` (the most recent per-bar write — the
|
||||
trade-close bar's segment_complete branch does NOT write to the
|
||||
buffer). Modulo arithmetic handles wrap-around correctly.
|
||||
|
||||
Both are `__device__ __forceinline__` so the test wrapper kernel can
|
||||
share them bit-for-bit per `feedback_no_cpu_test_fallbacks`.
|
||||
|
||||
### New buffer + reset infrastructure
|
||||
|
||||
- `GpuExperienceCollector.hold_baseline_buffer: CudaSlice<f32>` —
|
||||
`[alloc_episodes × HOLD_BASELINE_BUFFER_SIZE = 30]` row-major
|
||||
f32 device-resident scratch. Allocated next to `alpha_per_env`.
|
||||
|
||||
- `pub const HOLD_BASELINE_BUFFER_SIZE: usize = 30` — top-of-file
|
||||
constant matching spec §4.2 (LOOKAHEAD_HORIZON_MAX). Re-exported
|
||||
so external tests can reference the same value.
|
||||
|
||||
- Registered in `StateResetRegistry` with `FoldReset` category;
|
||||
reset path mirrors `alpha_per_env`'s `stream.memset_zeros` async
|
||||
GPU memset (no host buffer to fill).
|
||||
|
||||
- Dispatch arm in `training_loop.rs::reset_named_state`.
|
||||
|
||||
- Pin-test `sp20_hold_baseline_buffer_registered_fold_reset` —
|
||||
verifies registry entry + category + description anchors.
|
||||
|
||||
### New experience_env_step kernel args
|
||||
|
||||
Six new args appended to the kernel signature (after the Task 2.2
|
||||
`alpha_per_env` / `loss_cap_idx` / `alpha_ema_idx` block):
|
||||
|
||||
1. `aux_logits_per_env: const float*` — `[N, K=2]` reuses
|
||||
`exp_aux_nb_logits_buf` (the same buffer the SP20 stats kernel
|
||||
reads). Stream-implicit producer→consumer ordering.
|
||||
2. `hold_baseline_buffer: float*` — `[N × hold_buffer_size]`.
|
||||
3. `hold_buffer_size: int` — = 30 (HOLD_BASELINE_BUFFER_SIZE).
|
||||
4. `aux_k: int` — = 2 (AUX_NEXT_BAR_K). Reserved for future K>2
|
||||
support; the helper currently hardcodes K=2 per spec §4.2 K=2
|
||||
AMENDED note.
|
||||
5. `hold_cost_scale_idx: int` — ISV slot 513. Mirrors the
|
||||
`loss_cap_idx` / `alpha_ema_idx` pattern (caller passes slot
|
||||
index so the kernel stays decoupled from SP14 slot-number drift).
|
||||
6. `hold_reward_ema_idx: int` — ISV slot 516.
|
||||
|
||||
All NULL-tolerant: `aux_logits_per_env == NULL` ⇒ both paths skip
|
||||
(per-bar emission collapses to zero, preserving existing behavior
|
||||
bit-identically). `hold_baseline_buffer == NULL` ⇒ Path 2 skipped;
|
||||
Path 1 still fires (centered Hold reward independent of buffer).
|
||||
Both NULL ⇒ trade-close `hold_baseline` falls back to 0.0f (Phase 2
|
||||
placeholder behavior).
|
||||
|
||||
### HOLD_REWARD_EMA forward-reference (concurrent-agent collision avoidance)
|
||||
|
||||
`ISV[HOLD_REWARD_EMA_INDEX]` is updated by `sp20_emas_compute_kernel.cu`
|
||||
which reads `ema_inputs->per_bar_hold_reward` from
|
||||
`sp20_aggregate_inputs_kernel.cu`. The aggregate kernel currently
|
||||
emits `per_bar_hold_reward = 0.0f` (Phase 3.2 forward-reference
|
||||
placeholder at line 292) — the parallel agent on
|
||||
`sp20-phase-2-fix` branch is wiring the real producer. Until that
|
||||
branch lands:
|
||||
|
||||
- `ISV[HOLD_REWARD_EMA] = 0.0` (sentinel).
|
||||
- Path 1 centered = `per_bar_opp_cost - 0.0` = `per_bar_opp_cost`
|
||||
(centering is a no-op pre-merge).
|
||||
|
||||
The Task 3.2 contract is **forward-compatible**: the centering math
|
||||
references the ISV slot (not a hardcoded 0); when the parallel
|
||||
branch's aggregate-kernel update lands, EMA starts updating, and the
|
||||
centering becomes load-bearing automatically. No additional Phase 3.2
|
||||
work needed post-merge.
|
||||
|
||||
### Plan accuracy errata
|
||||
|
||||
See `docs/superpowers/plans/2026-05-09-sp19-20-wr-first.md` Phase 3
|
||||
Errata Gaps 9 (per-env layout), 10 (existing `PS_HOLD_TIME` /
|
||||
`segment_hold_time` suffices, no new state slot) for the
|
||||
plan-vs-reality decisions. Spec §4.2 phrasing was implicit
|
||||
single-thread; reality is per-env-parallel with per-env stride.
|
||||
|
||||
### New device kernel: `sp20_hold_baseline_test_kernel.cu`
|
||||
|
||||
Test wrapper kernels (3 entry points) for the GPU oracle tests:
|
||||
|
||||
- `sp20_per_bar_aux_conf_k2_test_kernel(logits, out)` — drives
|
||||
`sp20_compute_per_bar_aux_conf_k2` once.
|
||||
- `sp20_sum_hold_baseline_over_trade_test_kernel(buf, env, size,
|
||||
current_t, hold_time, out)` — drives the buffer summation helper.
|
||||
- `sp20_dual_emission_test_kernel(...)` — fixture mirroring the
|
||||
production per-bar emission site for both Hold-action AND
|
||||
non-Hold-action paths.
|
||||
|
||||
No production callers; cubin built by `crates/ml/build.rs`.
|
||||
|
||||
### Tests
|
||||
|
||||
`crates/ml/tests/sp20_hold_baseline_test.rs` — 4 GPU oracle tests:
|
||||
|
||||
1. `aux_conf_k2_bounds_and_fixed_points` — bounds [0, 0.5],
|
||||
uniform/saturated/spec-warmup/spec-confident/symmetry fixed
|
||||
points. Six sub-assertions across one test.
|
||||
2. `sum_hold_baseline_over_trade_indexing` — basic indexing,
|
||||
`segment_hold_time > size` clamp, empty-trade defensive guard,
|
||||
empty-buffer defensive guard, wrap-around modulo.
|
||||
3. `sum_hold_baseline_per_env_stride` — multiple envs in row-major
|
||||
buffer; verifies the env stride lands in the right row
|
||||
(a missing stride would fail the assertion clearly).
|
||||
4. `dual_emission_hold_vs_non_hold` — full dual-emission contract:
|
||||
Path 1 fires Hold-only AND Path 2 fires always; non-target
|
||||
buffer slots remain untouched. Mirrors plan §3.2 reference
|
||||
fixture (aux_conf=0.3, cost_scale=0.1, hold_reward_ema=-0.02).
|
||||
|
||||
### Files modified
|
||||
|
||||
| File | Status | Purpose |
|
||||
|------|--------|---------|
|
||||
| `crates/ml/src/cuda_pipeline/sp20_hold_baseline.cuh` | NEW | 2 device helpers (Component 2 dual-emission + summation) |
|
||||
| `crates/ml/src/cuda_pipeline/sp20_hold_baseline_test_kernel.cu` | NEW | 3 test-kernel entry points |
|
||||
| `crates/ml/src/cuda_pipeline/experience_kernels.cu` | +6 kernel args, +helper +include, +trade-close consumer wiring, ~70 LoC delete + ~70 LoC insert at 2 per-bar branches | Production wiring (Tasks 3.2 dual emission + trade-close consumer) |
|
||||
| `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs` | +HOLD_BASELINE_BUFFER_SIZE const, +field, +alloc, +6 kernel-arg passes | Per-env buffer infrastructure + production launch wiring |
|
||||
| `crates/ml/src/trainers/dqn/state_reset_registry.rs` | +entry + invariant test | FoldReset for `hold_baseline_buffer` |
|
||||
| `crates/ml/src/trainers/dqn/trainer/training_loop.rs` | +match arm | `reset_named_state` dispatch |
|
||||
| `crates/ml/build.rs` | +1 entry | sp20_hold_baseline_test_kernel cubin |
|
||||
| `crates/ml/tests/sp20_hold_baseline_test.rs` | NEW | 4 GPU oracle tests + 1 helper module |
|
||||
| `docs/dqn-wire-up-audit.md` | This entry | Audit log |
|
||||
|
||||
### Verification
|
||||
|
||||
```
|
||||
SQLX_OFFLINE=true cargo build -p ml # cubin compile
|
||||
SQLX_OFFLINE=true cargo check -p ml --tests --features cuda # tests compile
|
||||
SQLX_OFFLINE=true cargo test -p ml --lib sp20 # 21/21 lib tests
|
||||
SQLX_OFFLINE=true CUDA_COMPUTE_CAP=86 cargo test -p ml \
|
||||
--test sp20_hold_baseline_test --features cuda \
|
||||
-- --ignored --nocapture # 4 GPU oracle tests
|
||||
```
|
||||
|
||||
### Phase 3 → Phase 4 forward references
|
||||
|
||||
- Phase 3.4: replay buffer schema — per-bar `aux_conf` field for
|
||||
the Phase 5 Aux→Q gate (lands separately as the next atomic
|
||||
commit on this branch).
|
||||
- Phase 4: n-step credit distributor consumes `R_used = alpha -
|
||||
alpha_ema` (Component 1 output) — the `alpha` produced HERE in
|
||||
Phase 3.2 (= `R_event - sum_hold_baseline_over_trade`) becomes
|
||||
Phase 4's input.
|
||||
|
||||
## 2026-05-10 — SP20 Phase 3 Task 3.3: target_hold_pct behavioral spot-checks (test-only)
|
||||
|
||||
Adds infrastructure for verifying the SP20 Phase 2 Task 2.2-fix
|
||||
(`is_win_per_env`) producer side at runtime. The consumer-side
|
||||
|
||||
Reference in New Issue
Block a user