gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(sp21): T3.3 ISV defrost — hold_reward_ema (atomic Phase 3.2 wireup)

Browse Source 
Closes the Phase 3.2 forward-reference loop in the SP20 aggregator.
Previously `out_inputs->per_bar_hold_reward = 0.0f` was hardcoded; the
per-bar Hold opportunity-cost producer existed
(experience_kernels.cu:3823 — `per_bar_opp_cost = -aux_conf × cost_scale`)
and wrote to `hold_baseline_buffer` and `r_micro` directly, but never
reached HOLD_REWARD_EMA. Result: HOLD_REWARD_EMA frozen at sentinel 0.0
across all observed epochs; the SP20 reward centering loop
(`r_micro += per_bar_opp_cost - HOLD_REWARD_EMA`) stayed uncentered,
biasing the policy's reward signal away from zero-mean.

T3.3 wireup mirrors the T2.2 alpha refactor: a per-env scratch buffer
that the producer writes at every step (alongside the existing
`hold_baseline_buffer` write), and the aggregator reads at the same
step. Sums opp_cost over Hold-direction envs only; emits the mean as
`per_bar_hold_reward`. The HOLD_REWARD_EMA's gate (`hold_fraction > 0.5f`
from T3.2) preserves the strict-majority semantic from before.

Atomic across producer site, kernel signature, aggregator, launcher,
collector, and tests (per feedback_no_partial_refactor):

  - experience_kernels.cu — new `float* per_bar_opp_cost_per_env`
    kernel arg, NULL-tolerant write at line ~3823.
  - sp20_aggregate_inputs_kernel.cu — new arg, 6th shmem stripe
    (`sh_opp_cost_sum`), per-thread Hold-gated accumulation, tree
    reduction extended to 6 stripes, output `per_bar_hold_reward =
    opp_cost_sum / hold_count` when hold_count > 0 else 0.
  - sp20_aggregate_inputs.rs — launcher signature, dynamic_shmem_bytes
    5→6 stripes, doc table, internal test.
  - gpu_experience_collector.rs — new `per_bar_opp_cost_per_env:
    CudaSlice<f32>` field, alloc, struct construction, kernel-arg
    pass at both experience_env_step and sp20_aggregate_inputs
    launches (raw_ptr).
  - sp20_aggregate_inputs_test.rs — helper renamed
    `run_kernel_with_is_win_and_opp_cost`, 4 existing sites pass
    NULL, 2 NEW oracle tests verifying real producer + NULL fallback.
  - sp20_phase1_4_wireup_test.rs — NULL fallback at the wireup site;
    HOLD_REWARD_EMA-stays-at-zero assertion remains valid.

Verification:
  - cargo check -p ml --tests: passes (warnings only)
  - cargo test -p ml --test sp20_aggregate_inputs_test --features cuda
    -- --ignored: 12/12 GPU oracle tests pass on RTX 3050 Ti, including
    both new T3.3 tests (per_bar_hold_reward_means_over_hold_envs_only,
    null_per_bar_opp_cost_emits_zero).
  - cargo test -p ml --test sp20_phase1_4_wireup_test --features cuda
    -- --ignored: 2/2 pass under the new NULL-tolerant contract.

Plan reference: docs/plans/2026-05-10-sp21-train-eval-coherence-isv-defrost.md
Tier 3 status: T3.1 ✓, T3.2 ✓, T3.3 ✓ (this commit), T3.4 withdrawn,
T3.5 cascade-pending, T3.6 withdrawn.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-10 20:35:39 +02:00
parent 1790a31b66
commit 4d4cd996db
8 changed files with 334 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files

79
docs/dqn-wire-up-audit.md
View File

@@ -2,6 +2,85 @@
**Status:** Populated during Plan 1 Task 6 (A.5 orphan audit). Updated on every commit per Invariant 7.
## 2026-05-10 — SP21 T3.3: hold_reward_ema defrost (atomic Phase 3.2 producer wireup)
**Branch:** `sp20-aux-h-fixed` (off `1790a31b6`).
**Commits:** 1 atomic (this commit).
Closes the Phase 3.2 forward-reference loop in the SP20 aggregator.
Previously `out_inputs->per_bar_hold_reward = 0.0f` was hardcoded as a
deferred Phase 3.2 placeholder; the per-bar Hold opportunity-cost
producer existed (`experience_kernels.cu:3823` — `per_bar_opp_cost =
-aux_conf × cost_scale`) and wrote to `hold_baseline_buffer` and
`r_micro` directly, but never reached HOLD_REWARD_EMA. Result:
HOLD_REWARD_EMA frozen at sentinel 0.0 across all observed epochs,
the SP20 reward centering loop (`r_micro += per_bar_opp_cost -
HOLD_REWARD_EMA`) stayed uncentered, biasing the policy's reward
signal away from zero-mean.
**T3.3 wireup mirrors the T2.2 alpha refactor**: a per-env scratch
buffer that the producer writes at every step (alongside the existing
`hold_baseline_buffer` write), and the aggregator reads at the same
step. Sums opp_cost over Hold-direction envs only; emits the mean as
`per_bar_hold_reward`. The HOLD_REWARD_EMA's gate (`hold_fraction >
0.5f` from T3.2) preserves the strict-majority semantic from before
the binary→fractional refactor.
**Affected files (atomic per `feedback_no_partial_refactor`):**
- `crates/ml/src/cuda_pipeline/experience_kernels.cu` — new
`float* per_bar_opp_cost_per_env` kernel arg; per-env producer
write at the existing per-bar opp-cost computation site (line
~3823) alongside the `hold_baseline_buffer` write.
- `crates/ml/src/cuda_pipeline/sp20_aggregate_inputs_kernel.cu` —
new kernel arg, 6th shmem stripe (`sh_opp_cost_sum`), per-thread
Hold-gated accumulation (`if (dir == hold_action_id)
local_opp_cost_sum += per_bar_opp_cost_per_env[env]`), tree
reduction extended to 6 stripes, output computation
(`per_bar_hold_reward = opp_cost_sum / hold_count`).
- `crates/ml/src/cuda_pipeline/sp20_aggregate_inputs.rs` — launcher
signature accepts `per_bar_opp_cost_per_env_dev: u64` (NULL = `0`),
`dynamic_shmem_bytes()` 5→6 stripes, doc table, internal test.
- `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs` — new
`pub(crate) per_bar_opp_cost_per_env: CudaSlice<f32>` field, alloc
in constructor, struct-construction line, kernel-arg pass at both
the experience_env_step launch and the sp20_aggregate_inputs
launch (raw_ptr).
- `crates/ml/tests/sp20_aggregate_inputs_test.rs` — `run_kernel_with_
is_win` renamed `run_kernel_with_is_win_and_opp_cost`; 4 existing
call sites pass NULL; **2 new GPU oracle tests** verifying
(a) `per_bar_hold_reward = mean over Hold-dir envs` with synthetic
opp_costs and a Long env whose slot must be excluded; (b) NULL
producer fallback emits 0.0 preserving the pre-T3.3 contract.
- `crates/ml/tests/sp20_phase1_4_wireup_test.rs` — NULL fallback at
the wireup test's `launch_sp20_aggregate_inputs` site (the
`alpha_and_hold_reward_emas_stay_at_zero_with_null_producers`
assertion remains valid).
**Verification:**
- `cargo check -p ml --tests` — passes (warnings only).
- `cargo test -p ml --test sp20_aggregate_inputs_test --features cuda
-- --ignored` — **12/12 GPU oracle tests pass on RTX 3050 Ti**,
including both new T3.3 tests (`per_bar_hold_reward_means_over_
hold_envs_only`, `null_per_bar_opp_cost_emits_zero`).
- `cargo test -p ml --test sp20_phase1_4_wireup_test --features cuda
-- --ignored` — 2/2 pass; HOLD_REWARD_EMA assertion holds under
the new NULL-tolerant contract.
**Pearls applied:**
- `pearl_first_observation_bootstrap` — HOLD_REWARD_EMA's existing
bootstrap path (sentinel 0.0 → first observation) still applies;
T3.3 only changes WHAT the observation is (real opp_cost mean
instead of hardcoded 0).
- `feedback_no_partial_refactor` — kernel signature, launcher,
collector, tests all migrated atomically in this single commit.
- `pearl_blend_formulas_must_have_permanent_floor` — the consumer
side at `experience_kernels.cu:3838` (`r_micro += per_bar_opp_cost
- HOLD_REWARD_EMA`) is now correctly centered; previously the
EMA=0 made the centering a no-op.
## 2026-05-10 — SP21 Tier-3 foundation: T3.1+T3.2 ISV defrost (wr_ema, hold_pct_ema)
**Branch:** `sp20-aux-h-fixed` (off `6df44e4c6`).

4
docs/plans/2026-05-10-sp21-train-eval-coherence-isv-defrost.md
View File

@@ -132,9 +132,9 @@ Each ISV slot below stayed constant across all 3 observed epochs of `d7bj7`. The
| T3.1 | `wr_ema` | 0.0000 (frozen) | EMA of segment-close win rate ∈ [0, 1] | `pearl_per_bar_vs_segment_pnl_signal_mismatch` patch added `is_win_per_env` buffer; verify producer is reading it AND writing to `wr_ema_idx` |
| T3.2 | `hold_pct_ema` | 0.0000 (frozen) | EMA of fraction of bars where policy holds an active position | Likely an EMA producer reading from action_counts or position_active_mask |
| T3.3 | `hold_reward_ema` | 0.0000 (frozen) | EMA of opportunity-cost reward magnitude per bar | Likely populated alongside hold_pct |
| T3.4 | `loss_cap` | -1.0000 (sentinel) | Should be replaced with adaptive value after first observation per Pearl-A | First-observation bootstrap kernel — sentinel-detection guard may have wrong condition |
| ~~T3.4~~ | ~~`loss_cap`~~ | ~~-1.0000 (sentinel)~~ | **WITHDRAWN 2026-05-10 (not a bug)**: investigation shows the value `-1.0` is NOT a sentinel — it's the LOSS_CAP controller's correct output for `WR_EMA < 0.50` (`-1.0 - clamp((wr_ema-0.50)/0.05, 0, 1)`). The actual sentinel per `state_reset_registry` is `0.0` (FoldReset bootstrap). After T3.1 fix, WR_EMA moves to actual val WR ≈ 0.46, but LOSS_CAP stays at -1.0 because WR is still below the 0.50 ramp threshold. **Will move to -2.0 only when WR_EMA crosses 0.55** (linear ramp in [0.50, 0.55]). GPU test `loss_cap_ramp_boundaries` verified the controller logic is correct (passed in our run). | **No fix needed at this slot.** Same shape as T3.6 — downstream of upstream WR signal being below the controller's design threshold. Will defrost automatically when WR climbs above 0.50. |
| T3.5 | `hold_cost_scale` | 0.0100 (at min floor) | Controller-driven within [min, max] bounds | Controller reads `hold_pct_ema` (which is also frozen at 0) — chain failure: T3.2 must fix first |
| T3.6 | `aux_conf_threshold` | 0.0100 (at min floor) | Controller drives based on aux confidence distribution | Producer kernel reads aux confidence histogram; check if histogram producer is firing |
| ~~T3.6~~ | ~~`aux_conf_threshold`~~ | ~~0.0100 (at min floor)~~ | **WITHDRAWN 2026-05-10 (not a bug)**: investigation shows the controller at `sp20_controllers_compute_kernel.cu:152-157` (`clamp(aux_dir_acc_ema - 0.50, 0.01, 0.20)`) is *working correctly*. With observed `aux_dir_acc ≈ 0.46` (below random in d7bj7 logs), raw = -0.04, controller clamps at floor 0.01. Chain `aux_dir_acc_reduce_kernel → aggregator → EMA blend → controller` is intact and the EMA tracks the actual ~0.46 value. The slot will defrost on its own when aux improves above 0.51. **Downstream symptom of the upstream aux signal problem, NOT an independent producer-chain bug.** | **No fix needed at this slot.** Upstream problem (aux_dir_acc < 0.51) addressed by upstream feature/architecture work — separate from SP21 scope. |
**Strategy**: T3.1 fixed first as canonical example (it has a recent pearl + recent commit `64bbbe418` claiming the fix). Verify the patch actually propagated to the ISV slot, not just to `is_win_per_env` buffer. Once T3.1 is verified working, use the same pattern (find producer → check input availability → check write-to-isv → check sentinel guard) for T3.2-T3.6 in parallel.