gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(sp21): T3.3 ISV defrost — hold_reward_ema (atomic Phase 3.2 wireup)

Browse Source 
Closes the Phase 3.2 forward-reference loop in the SP20 aggregator.
Previously `out_inputs->per_bar_hold_reward = 0.0f` was hardcoded; the
per-bar Hold opportunity-cost producer existed
(experience_kernels.cu:3823 — `per_bar_opp_cost = -aux_conf × cost_scale`)
and wrote to `hold_baseline_buffer` and `r_micro` directly, but never
reached HOLD_REWARD_EMA. Result: HOLD_REWARD_EMA frozen at sentinel 0.0
across all observed epochs; the SP20 reward centering loop
(`r_micro += per_bar_opp_cost - HOLD_REWARD_EMA`) stayed uncentered,
biasing the policy's reward signal away from zero-mean.

T3.3 wireup mirrors the T2.2 alpha refactor: a per-env scratch buffer
that the producer writes at every step (alongside the existing
`hold_baseline_buffer` write), and the aggregator reads at the same
step. Sums opp_cost over Hold-direction envs only; emits the mean as
`per_bar_hold_reward`. The HOLD_REWARD_EMA's gate (`hold_fraction > 0.5f`
from T3.2) preserves the strict-majority semantic from before.

Atomic across producer site, kernel signature, aggregator, launcher,
collector, and tests (per feedback_no_partial_refactor):

  - experience_kernels.cu — new `float* per_bar_opp_cost_per_env`
    kernel arg, NULL-tolerant write at line ~3823.
  - sp20_aggregate_inputs_kernel.cu — new arg, 6th shmem stripe
    (`sh_opp_cost_sum`), per-thread Hold-gated accumulation, tree
    reduction extended to 6 stripes, output `per_bar_hold_reward =
    opp_cost_sum / hold_count` when hold_count > 0 else 0.
  - sp20_aggregate_inputs.rs — launcher signature, dynamic_shmem_bytes
    5→6 stripes, doc table, internal test.
  - gpu_experience_collector.rs — new `per_bar_opp_cost_per_env:
    CudaSlice<f32>` field, alloc, struct construction, kernel-arg
    pass at both experience_env_step and sp20_aggregate_inputs
    launches (raw_ptr).
  - sp20_aggregate_inputs_test.rs — helper renamed
    `run_kernel_with_is_win_and_opp_cost`, 4 existing sites pass
    NULL, 2 NEW oracle tests verifying real producer + NULL fallback.
  - sp20_phase1_4_wireup_test.rs — NULL fallback at the wireup site;
    HOLD_REWARD_EMA-stays-at-zero assertion remains valid.

Verification:
  - cargo check -p ml --tests: passes (warnings only)
  - cargo test -p ml --test sp20_aggregate_inputs_test --features cuda
    -- --ignored: 12/12 GPU oracle tests pass on RTX 3050 Ti, including
    both new T3.3 tests (per_bar_hold_reward_means_over_hold_envs_only,
    null_per_bar_opp_cost_emits_zero).
  - cargo test -p ml --test sp20_phase1_4_wireup_test --features cuda
    -- --ignored: 2/2 pass under the new NULL-tolerant contract.

Plan reference: docs/plans/2026-05-10-sp21-train-eval-coherence-isv-defrost.md
Tier 3 status: T3.1 ✓, T3.2 ✓, T3.3 ✓ (this commit), T3.4 withdrawn,
T3.5 cascade-pending, T3.6 withdrawn.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-10 20:35:39 +02:00
parent 1790a31b66
commit 4d4cd996db
8 changed files with 334 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
docs/plans/2026-05-10-sp21-train-eval-coherence-isv-defrost.md
View File

@@ -132,9 +132,9 @@ Each ISV slot below stayed constant across all 3 observed epochs of `d7bj7`. The
| T3.1 | `wr_ema` | 0.0000 (frozen) | EMA of segment-close win rate ∈ [0, 1] | `pearl_per_bar_vs_segment_pnl_signal_mismatch` patch added `is_win_per_env` buffer; verify producer is reading it AND writing to `wr_ema_idx` |
| T3.2 | `hold_pct_ema` | 0.0000 (frozen) | EMA of fraction of bars where policy holds an active position | Likely an EMA producer reading from action_counts or position_active_mask |
| T3.3 | `hold_reward_ema` | 0.0000 (frozen) | EMA of opportunity-cost reward magnitude per bar | Likely populated alongside hold_pct |
| T3.4 | `loss_cap` | -1.0000 (sentinel) | Should be replaced with adaptive value after first observation per Pearl-A | First-observation bootstrap kernel — sentinel-detection guard may have wrong condition |
| ~~T3.4~~ | ~~`loss_cap`~~ | ~~-1.0000 (sentinel)~~ | **WITHDRAWN 2026-05-10 (not a bug)**: investigation shows the value `-1.0` is NOT a sentinel — it's the LOSS_CAP controller's correct output for `WR_EMA < 0.50` (`-1.0 - clamp((wr_ema-0.50)/0.05, 0, 1)`). The actual sentinel per `state_reset_registry` is `0.0` (FoldReset bootstrap). After T3.1 fix, WR_EMA moves to actual val WR ≈ 0.46, but LOSS_CAP stays at -1.0 because WR is still below the 0.50 ramp threshold. **Will move to -2.0 only when WR_EMA crosses 0.55** (linear ramp in [0.50, 0.55]). GPU test `loss_cap_ramp_boundaries` verified the controller logic is correct (passed in our run). | **No fix needed at this slot.** Same shape as T3.6 — downstream of upstream WR signal being below the controller's design threshold. Will defrost automatically when WR climbs above 0.50. |
| T3.5 | `hold_cost_scale` | 0.0100 (at min floor) | Controller-driven within [min, max] bounds | Controller reads `hold_pct_ema` (which is also frozen at 0) — chain failure: T3.2 must fix first |
| T3.6 | `aux_conf_threshold` | 0.0100 (at min floor) | Controller drives based on aux confidence distribution | Producer kernel reads aux confidence histogram; check if histogram producer is firing |
| ~~T3.6~~ | ~~`aux_conf_threshold`~~ | ~~0.0100 (at min floor)~~ | **WITHDRAWN 2026-05-10 (not a bug)**: investigation shows the controller at `sp20_controllers_compute_kernel.cu:152-157` (`clamp(aux_dir_acc_ema - 0.50, 0.01, 0.20)`) is *working correctly*. With observed `aux_dir_acc ≈ 0.46` (below random in d7bj7 logs), raw = -0.04, controller clamps at floor 0.01. Chain `aux_dir_acc_reduce_kernel → aggregator → EMA blend → controller` is intact and the EMA tracks the actual ~0.46 value. The slot will defrost on its own when aux improves above 0.51. **Downstream symptom of the upstream aux signal problem, NOT an independent producer-chain bug.** | **No fix needed at this slot.** Upstream problem (aux_dir_acc < 0.51) addressed by upstream feature/architecture work — separate from SP21 scope. |
**Strategy**: T3.1 fixed first as canonical example (it has a recent pearl + recent commit `64bbbe418` claiming the fix). Verify the patch actually propagated to the ISV slot, not just to `is_win_per_env` buffer. Once T3.1 is verified working, use the same pattern (find producer → check input availability → check write-to-isv → check sentinel guard) for T3.2-T3.6 in parallel.