4bed8f2dbfb46387e263e06ef40a4c1948de35bf
122 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
jgrusewski
|
1434ecc421 |
test(rl): GPU oracle tests for IQN + NoisyNet + PRNG advance fix
Six GPU-oracle tests validating Phase 1 device-side PRNG kernels and the IQN/NoisyNet heads (mapped-pinned data transfer throughout): IQN: tau U(0,1) range + PRNG advance, forward finite, expected_q mean NoisyNet: factored transform invariant, zero-noise = mu, resample δ Fix: PRNG advance used prng_state[b] (zero on first call) instead of the self-seeded `seed` — xorshift32(0)=0 made second call identical. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
jgrusewski
|
a7c1d763d8 |
perf(rl): device-resident step counter + fused controllers (-9 launches)
Two CUDA Graph prerequisites implemented:
1. Device-resident step counter (ISV[548]):
- New rl_increment_step.cu kernel (single thread, ISV += 1)
- All kernels that took current_step as scalar now read from ISV
- Updated: confidence_gate, frd_gate, unit_state_update,
trade_context_update, gate_threshold_controller
- Enables CUDA Graph capture (no scalar arg changes between replays)
2. Fused controllers (rl_fused_controllers.cu):
- Combines 10 single-thread controllers into 1 kernel launch:
gamma, tau, ppo_clip, entropy, rollout_steps, per_alpha,
reward_scale (with ±2% clamp), ppo_ratio_clamp,
gate_threshold, q_distill_lambda
- Saves 9 kernel launches per step (~40-80μs)
- Individual .cu files retained for testing/documentation
ISV slot 548 (step counter). Local smoke: 100 steps, no crash.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
82481db06d |
feat(rl): gate warmup — confidence + FRD gates inactive for first 10k steps
Both gates now read RL_GATE_WARMUP_STEPS_INDEX (slot 524, default 10000) and return early when current_step < warmup. During warmup, the agent opens positions freely, collects reward signal, and calibrates Q. After warmup, gates activate and filter low-quality entries. Without warmup, gates blocked 100% of opening actions from step 0 (uniform Q → zero confidence → permanent Hold attractor → zero trades → zero reward → Q never learns). Confirmed by 50k-step L40S run with zero dones across 800k action decisions. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
jgrusewski
|
233894a4bf |
feat(rl): trade-context + multires features + P14 validation tests
P1: New rl_trade_context_update.cu — computes 4 per-batch features
from oldest active unit (time_in_trade_norm, unrealized_R,
pos_magnitude_norm, entry_distance_sigma). Output in
trade_context_d[B×4], updated after unit_state_update each step.
P0: New rl_multires_features_update.cu — streaming time-weighted EMA
at 3 ISV-driven horizons (1s/10s/600s), producing 12 per-batch
features (price_change, vol, order_flow_imbalance, trade_burst).
O(1) state per feature vs circular buffer — same time-constant
semantics.
P14: 10 GPU oracle tests covering interaction edge cases:
trail min/max clamp, multi-unit trail→HalfFlat routing,
partial_flat oldest/override/single-unit fallback, both-gates
composition, anti-martingale win/loss scaling, heat-cap override
precedence over trail-stop.
ISV slots: 521-523 (multires horizons). RL_SLOTS_END → 524.
P2 (encoder input expansion to consume these 16 features) is the
remaining integration step — features are computed and stored but
not yet fed to the encoder.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
3b23a0de5a |
feat(rl): per-batch outcome EMA, vol-adjusted trail, ISV-driven P_MIN
P10: New rl_recent_outcome_update.cu — per-batch signed outcome EMA
(sign(reward) on done steps) feeds per-batch anti-martingale
sizing in actions_to_market_targets. Replaces the single ISV
scalar with a per-batch buffer for multi-batch granularity.
P11: Trail bootstrap switched from vwap × 1e-3 × k_init to
k_init × MEAN_ABS_PNL_EMA (slot 423). Vol-derived trail
distance adapts to realized trade magnitude as the EMA updates.
P12: P_MIN in rl_pi_action_kernel now ISV-driven (slot 519,
default 0.015). At N=11, max single-action prob = 0.85
(uplift vs prior 0.80 at hardcoded P_MIN=0.02).
ISV slots: 519 P_MIN, 520 OUTCOME_ALPHA. RL_SLOTS_END -> 521.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
e9ecacbdfa |
feat(rl): FRD gate — override entries when forward-return is unfavorable
New rl_frd_gate.cu kernel reads the FRD head's horizon-2 (medium,
~300 ticks) categorical distribution. For long openings, sums
probability mass in the positive tail (atoms > +0.5σ); for short
openings, sums the negative tail (atoms < -0.5σ). Overrides to Hold
when favorable mass < threshold.
Fires after confidence gate, before trail/heat/market pipeline.
Same preconditions: only gates flat positions with opening actions.
ISV slots: 516 THR_LONG (0.35), 517 THR_SHORT (0.35),
518 fired_count (diag). RL_SLOTS_END → 519.
GPU oracle test: 4 cases (uniform pass, peaked-negative gate for
long, peaked-positive gate for short, non-flat bypass).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
e132d59a48 |
feat(rl): confidence gate — override low-certainty openings to Hold
New rl_confidence_gate.cu kernel computes C51 distributional Lower
Confidence Bound (μ - λσ) / σ_norm for the chosen action. When
position is flat and the selected action is an opening (a0/a1/a5/a6),
overrides to Hold if conf < threshold.
Fires after π action selection, before trail/heat/market pipeline.
Only gates on flat positions — existing positions pass through
unconditionally regardless of Q uncertainty.
ISV slots: 512 threshold (0.10), 513 λ (1.0), 514 σ_norm (1.0),
515 fired_count (diag). RL_SLOTS_END → 516.
GPU oracle test: 4 cases (low-conf gate, high-conf pass, non-flat
bypass, non-opening bypass).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
a583bb508c |
feat(rl): pyramiding semantics — add/partial-flat/anti-martingale sizing
Implements the full pyramid trade-management suite:
P7.a: actions_to_market_targets gates pyramid adds on ISV-driven
threshold (slot 506); rl_unit_state_update allocates sequential
unit slots on position growth, deactivates oldest on shrink.
P7.b: HalfFlat (a9/a10) closes oldest unit's lots when pyramid>1;
trail-stop routes breaches through HalfFlat + close_unit_index
override instead of nuclear full-flat.
P7.c: Anti-martingale sizing on opening actions via signed outcome EMA
(slot 508) — size_eff = base × clamp(1 + κ × ema, MIN, MAX).
Diag: pyramid { units_count, add_count, outcome_ema } in step JSONL.
ISV slots: 506 threshold, 507 add_count, 508 outcome_ema,
509 κ, 510 MIN, 511 MAX. RL_SLOTS_END → 512.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
45a2041db4 |
feat(rl): SP20 P6 position heat cap — force-flat on over-leverage
Last-defense guard: if |position_lots| exceeds the ISV-driven
RL_HEAT_CAP_MAX_LOTS (slot 504, default 8 = MAX_UNITS × max_order_size),
the kernel overrides actions[b] to FlatFromLong (a3) or FlatFromShort
(a4) — full flatten, no partial. Catches runaway pyramid accumulation
before it reaches actions_to_market_targets.
Override stack ordering (step_with_lobsim):
1. rl_trail_mutate (a7/a8)
2. rl_trail_stop_check → may override to FlatFromLong/Short
3. rl_position_heat_check (THIS) → may override to FlatFromLong/Short
4. actions_to_market_targets → reads final actions[b]
Kernel `cuda/rl_position_heat_check.cu`:
* 1 block, b_size threads (grid-stride for b_size > 256)
* Reads position_lots from pos_state at offset 0 (PosFlat layout)
* Cap read from ISV[504]; if cap ≤ 0 → no-op (guard disabled)
* Per feedback_no_atomicadd: fired-count diagnostic uses shared-mem
flag array + thread-0 serial count (b_size ≤ 256 in practice)
* Writes fired-count to ISV[505] for diag
ISV slots:
* 504: RL_HEAT_CAP_MAX_LOTS_INDEX (seed 8.0)
* 505: RL_HEAT_CAP_FIRED_COUNT_INDEX (diagnostic, written per step)
* RL_SLOTS_END bumped 505 → 506
Diag (alpha_rl_train):
* "heat_cap": { "fired_count": N, "max_lots": 8 }
GPU oracle test (trade_management_kernels.rs):
* position_heat_cap_overrides_on_breach — long 5 > cap 4 → a3;
short -5 < -cap -4 → a4; long 3 ≤ cap 4 → untouched (Hold)
Verification (RTX 3050 Ti):
* cargo check -p ml-alpha --examples → clean
* integrated_trainer_smoke 1/1 → ok
* trade_management_kernels 6/6 (was 5/5, +1 heat cap) → ok
* audit-rust-consts → 0 flags
|
||
|
jgrusewski
|
7df7c81d37 |
refactor(rl): FRD horizons + range_σ are ISV-driven, not literals
Closes the literal/const drift gap that F.5 introduced. Per
feedback_isv_for_adaptive_bounds + feedback_single_source_of_truth_no_duplicates:
adaptive bounds belong in ISV (or in a single canonical const that
ISV references), never duplicated as literals across modules.
Single canonical source: `crate::rl::common::FRD_HORIZON_TICKS` +
`FRD_BUCKET_RANGE_SIGMA` (already declared in F.5).
Producer-side fixes:
* Trainer ISV bootstrap (integrated.rs): the seed values for slots
500-503 now dereference the canonical consts instead of hardcoded
60.0/300.0/1800.0/3.0 literals. Future tuning of the consts
automatically propagates to both ISV seeds and loader-side
labels — no manual sync required, no drift possible.
* compute_frd_labels (loader.rs): takes `horizon_ticks` and
`range_sigma` as parameters instead of reading consts directly.
Caller (the file-load closure) sources them from the new
MultiHorizonLoaderConfig fields.
Consumer-side fixes — 8 MultiHorizonLoaderConfig literal sites now
provide the two new fields, all defaulting to the canonical consts:
* crates/ml-alpha/src/data/loader.rs (2 internal test-fixture sites)
* crates/ml-alpha/tests/multi_horizon_loader.rs (2 sites)
* crates/ml-alpha/examples/alpha_train.rs (2 sites)
* crates/ml-alpha/examples/alpha_rl_train.rs (2 sites)
* crates/ml-backtesting/src/harness.rs (1 site)
* crates/ml-backtesting/tests/{trainer_parity,ring3_replay}.rs (2 sites)
The "optimal by default" property is preserved: every caller that
doesn't explicitly override gets the spec-recommended 60/300/1800
ticks + ±3σ. Callers that need to retune set the config fields, and
the trainer's ISV slots provide a runtime knob for the same numerics.
Verification (RTX 3050 Ti):
* cargo check -p ml-alpha -p ml-backtesting --examples --tests → clean
* cargo test --lib (6/6 unit tests for FRD label gen + loss_balance) → pass
* frd_head 10/10 + integrated_trainer_smoke 1/1 + trade_mgmt 5/5 → pass
* audit-rust-consts → 0 flags
The two new MultiHorizonLoaderConfig fields are required (no Default
impl) — callers MUST opt in to the FRD label-generation contract by
naming the fields. This is the same discipline applied across other
config consumers; making them Option<...> would silently default to
"no FRD labels" and break F.4's expected supervised signal.
|
||
|
jgrusewski
|
0f75d6bb7b |
feat(rl): FRD layer-1 backward (dW1, db1, dh_t with ReLU mask) — F.3c
Third and final FRD backward stage. Closes the chain from
softmax+CE loss back to the encoder's hidden state h_t.
Kernel `cuda/rl_frd_layer1_bwd.cu`:
* grid_dim = (B, 1, 1), block_dim = (HIDDEN_DIM=128, 1, 1)
* Phase 0: threads 0..63 stage dL/dpre_hidden = grad_hidden ×
1{hidden > 0} into shared mem (the cached post-ReLU `hidden`
buffer encodes the mask — hidden == 0 ⇔ pre-activation was
≤ 0 → ReLU killed it). Same thread also writes db1_per_batch.
* Phase 1: each thread k (k < 128) writes one row of
grad_W1_per_batch[b, k, 0..64] (64 writes per thread, no atomics)
* Phase 2: same thread computes grad_h_t[b, k] =
Σ_i W1[k, i] × dL/dpre_hidden[b, i]
* Per-(b, k, i) sole-writer per feedback_no_atomicadd
Rust wiring `FrdHead::layer1_bwd` — takes h_t, hidden (forward cache),
grad_hidden (from layer2_bwd), self.w1_d; writes grad_w1_per_batch,
grad_b1_per_batch, grad_h_t. The grad_h_t buffer becomes the encoder-
upstream gradient that the trainer's grad_h_accumulate kernel folds
into the encoder's gradient with λ_frd scaling (same pattern as Q/π/V
heads — wiring lives in F.4).
Tests (2 new, 10/10 file total):
* frd_layer1_bwd_finite_diff_w1 — perturbs the W1 slot with MAX
|analytical gradient| (instead of an arbitrary fixed slot — fp32
finite-diff is rounding-error-limited so a tiny gradient gives
misleading rel_err). At max-magnitude slot (k=84, i=55): analytical
= -0.0451, numerical = -0.0448, rel_err = 5.6e-3 — well within
1e-2 tolerance (slightly looser than dW2's 5e-3 because dW1
crosses an extra matmul + the ReLU mask boundary).
* frd_layer1_bwd_relu_mask_zeros_grad — fixture with h_t = all -1
produces ~half the hidden slots ReLU-masked (cached hidden = 0).
For every masked slot i, asserts:
* db1_per_batch[b, i] == 0 (exact equality — mask is hard 0)
* dW1_per_batch[b, k, i] == 0 for every k (~32 × 128 = 4096
slots checked)
Empirically 32/64 masked, 32/64 active — confirms ReLU mask
is wired through the chain correctly without leaking gradient
through dead branches.
F.3 backward chain is now complete end-to-end:
rl_frd_softmax_ce_grad (F.3a) → rl_frd_layer2_bwd (F.3b) →
rl_frd_layer1_bwd (F.3c) → grad_h_t (consumed by F.4 wiring)
F.4 wires Adam optimizers for W1/b1/W2/b2 + grad_h_accumulate into
the encoder gradient + loader-side label generation + λ_frd × CE
into stats.l_total.
|
||
|
jgrusewski
|
91e2c5dc8a |
feat(rl): FRD layer-2 backward (dW2, db2, dhidden) — F.3b
Second of three FRD backward stages. Given dL/dlogits from F.3a's
softmax_ce_grad and the cached hidden activation from F.2's forward,
computes the layer-2 weight gradients via the standard chain rule
and emits the upstream gradient for layer-1 backward (F.3c).
Kernel `cuda/rl_frd_layer2_bwd.cu`:
* grid_dim = (B, 1, 1), block_dim = (FRD_HIDDEN_DIM=64, 1, 1)
* Phase 0: stage 63-slot grad_logits into shared (thread 63 idle)
* Phase 1: each thread i (i < 64) computes one row of per-batch
dW2 scratch: grad_w2_per_batch[b, i, 0..63] = h_bi × grad_logits[0..63]
(63 writes per thread, no atomics)
* Phase 2: each thread i computes dL/dhidden[b, i] = Σ_j W2[i, j] × grad_logits[j]
* Phase 3: thread i (i < 63) writes grad_b2_per_batch[b, i] = grad_logits[b, i]
* Per-batch scratch shape [B, FRD_HIDDEN_DIM, FRD_OUT_DIM] reduces
across batch via existing reduce_axis0 infra (caller's job, same
pattern as v_head_bwd / aux_heads_bwd)
Rust wiring `FrdHead::layer2_bwd`:
* Takes hidden (forward cache), grad_logits (from softmax_ce_grad),
self.w2_d
* Writes grad_w2_per_batch, grad_b2_per_batch, grad_hidden — all
sized to caller-allocated buffers
* Sole &self method (Adam step is the caller's responsibility)
Tests (2 new, 8/8 file total):
* frd_layer2_bwd_finite_diff_w2 — perturb W2[10, 5] by ±ε=1e-3,
compare (L(+) - L(-))/(2ε) to per-batch grad scratch. rel_err
= 6.27e-5 (better than F.3a's softmax-CE finite-diff because
the gradient magnitude here is larger so rounding error is
relatively smaller). Helper `ce_total_loss` re-uses
`softmax_ce_grad` to compute total CE for the perturbed forward
pass — pure GPU-oracle, no CPU softmax/CE reference impl.
* frd_layer2_bwd_db2_equals_grad_logits — analytical invariant:
db2_per_batch[b, j] must equal grad_logits[b, j] exactly (the
bias gradient is the identity passthrough at this layer). Cheap
structural check that catches dimension-shuffle bugs in the
kernel before they corrupt the reduce_axis0 step.
The kernel restores W2 to its original values after the perturbation
to keep test isolation clean — `&mut head` access pattern (proper
Rust borrowing, no UB const→mut casts).
F.3c (layer-1 backward: dW1, db1, dh_t with ReLU mask via the
cached hidden activation) is next.
|
||
|
jgrusewski
|
6cfd7e6691 |
feat(rl): FRD softmax + CE + dL/dlogits backward stage 1 (F.3a)
Per-(batch, horizon) softmax + cross-entropy loss + gradient w.r.t.
the 21 atom logits. First of three backward stages — F.3b adds layer-2
weight grads (dW2, db2, dhidden), F.3c adds layer-1 weight grads
(dW1, db1, dh_t with ReLU mask).
Kernel `cuda/rl_frd_softmax_ce_grad.cu`:
* grid_dim = (B, FRD_N_HORIZONS, 1), block_dim = (FRD_N_ATOMS=21, 1, 1)
— one block per (batch, horizon) pair, threads cooperate over the
21 atoms via shared mem
* Standard numerically-stable softmax: shift by row_max, exponentiate,
normalize by row_sum (thread 0 does the serial reductions — 21
atoms is small enough warp-shuffle overhead isn't worth it)
* Gradient: (p[a] - 1{a==label}) / B at the source per v_head_bwd
convention (mean-reduce over batch)
* Loss: -log(p[label]) with 1e-30 floor against log(0)
* Sentinel label (-1) zeros both gradient row and loss — for the
missing-horizon case at the rightmost edge of the snapshot stream
(forward returns at h=300 ticks aren't realized for the last
300 snapshots; loader marks those labels with -1)
* Per feedback_no_atomicadd: per-(b, h, a) sole-writer pattern
Rust wiring `src/rl/frd.rs::FrdHead::softmax_ce_grad`:
* Second cubin loaded alongside fwd (separate module per the
aux_heads pattern; small handle, no impact on init time)
* Caller provides labels_d [B, FRD_N_HORIZONS] of i32 and gets back
grad_logits + per-(b, h) raw CE; sum + λ_frd scaling left to the
caller (F.4 will hook this into stats.l_total + Adam step)
Tests `tests/frd_head.rs` — 3 new GPU-oracle tests (6/6 file total),
all PASS on RTX 3050 Ti:
1. frd_softmax_ce_grad_uniform_logits_match_log_n_atoms — for any
label, uniform logits → CE = ln(FRD_N_ATOMS) = ln(21) ≈ 3.0445.
Also asserts per-row Σ grad_logits = 0 (softmax-CE invariant).
2. frd_softmax_ce_grad_sentinel_label_zeros_row — label=-1 with
non-trivial random logits produces exactly zero loss + grad
for every row (no leak through the sentinel path).
3. frd_softmax_ce_grad_finite_diff_matches_analytical — perturbs
one logit slot by ±ε=1e-3, compares (L(+ε) - L(-ε))/(2ε) to
the kernel's analytical gradient. rel_err ≈ 1.3e-3 (fp32
finite-diff is rounding-error-limited at this ε; tolerance
set to 5e-3 with explanatory comment).
The first two tests provide strong analytical oracles (no CPU
reference impl per feedback_no_cpu_test_fallbacks). The finite-diff
test cross-validates the full softmax+CE chain via a numerical
gradient — the standard ground-truth for autodiff kernels.
|
||
|
jgrusewski
|
c6a03658ed |
feat(rl): FRD head forward pass + GPU-oracle tests (F.2)
Forward-Return-Distribution head per SP20 §3 P3. Supervised forecaster
over 3 horizons × 21 return-bucket atoms — replaces the survivor-biased
checklist head per CRIT-1.
Architecture (2-layer MLP):
hidden [B, 64] = ReLU(h_t [B, 128] @ W1 [128, 64] + b1)
logits [B, 63] = hidden @ W2 [64, 63] + b2 // 63 = 3 × 21
Softmax + CE happen in the backward kernel (F.3). The forward kernel
caches the post-ReLU hidden buffer to avoid recomputing the W1 product
+ ReLU mask on backward.
Kernel `cuda/rl_frd_fwd.cu` — 1 block per batch, 64 threads:
* Phase 1 (tid < 64): each thread computes one hidden activation,
stages into shared mem, writes the cached `hidden_out[b, tid]`
* Phase 2 (tid < 63): each thread computes one output logit by
reading the shared hidden vector
* No atomicAdd (per-batch, per-output sole-writer pattern)
* No host branches in the launch (graph-capture safe)
Rust head module `src/rl/frd.rs`:
* `FrdHead::new(dev, cfg)` — Xavier × 0.1 init for W1/W2 (small enough
to keep initial softmax near-uniform), zero biases. Scoped-init-seed
guard per pearl_scoped_init_seed_for_reproducibility.
* `forward(h_t_d, hidden_out_d, logits_out_d, b_size)` — single
kernel launch via the cached `fwd_fn` handle.
* Public weight buffers (w1_d, b1_d, w2_d, b2_d) for the upcoming
bwd kernel + test harnesses.
* `pub const FRD_OUT_DIM = FRD_N_HORIZONS × FRD_N_ATOMS = 63` — single
canonical reference for the per-batch output width.
Tests `tests/frd_head.rs` — 3 GPU-oracle tests, all PASS on RTX 3050 Ti:
1. frd_forward_zero_input_emits_zero_logits — h_t=0 with default
b1=b2=0 must produce exactly zero logits AND zero cached hidden.
Unambiguous analytical oracle for the full matmul + ReLU + matmul
chain.
2. frd_forward_shape_matches_spec — random h_t produces correctly
shaped output [B × 63] with per-horizon softmax sums = 1.0
within 1e-5 (numerical-stable log-sum-exp).
3. frd_forward_relu_mask_consistent_with_cached_hidden — strictly
negative h_t input → ≥50% of cached hidden slots must be exactly
zero (ReLU fires). Empirically 128/256 zeros on the seeded init.
Per feedback_isv_for_adaptive_bounds: bucket-range σ stays in ISV
(slot 503, seeded ±3σ); only the 21-atom count is structural
compile-time per SP20 §0.1.
|
||
|
jgrusewski
|
0b870a1b26 |
test(rl): GPU-oracle tests for trade-management kernel suite
Five #[ignore] CUDA-required tests for the trader-grade trade-management
kernels (rl_unit_state_update, actions_to_market_targets HalfFlat
branches, rl_trail_mutate, rl_trail_stop_check). Analytical invariant
oracles per feedback_no_cpu_test_fallbacks.
Public IntegratedTrainer launch wrappers (mirror internal kernel
invocations in step_with_lobsim with controllable buffers):
* launch_actions_to_market_targets
* launch_rl_trail_mutate
* launch_rl_trail_stop_check
* launch_rl_unit_state_update
Public mapped-pinned write/read helpers added to satisfy
feedback_no_htod_htoh_only_mapped_pinned (the pre-commit hook rejects
raw un-pinned host-side transfers with no grandfathering for new code):
* write_slice_f32_d_pub / write_slice_i32_d_pub / write_slice_u8_d_pub
* read_slice_u8_d_pub (counterpart to existing _d_pub readers)
write_slice_u8_d_pub / read_slice_u8_d_pub stage via MappedI32Buffer
(4-byte alignment) — covers byte-buffer fixtures like the 24-byte
PosFlat layout used by the unit-state and half-flat tests without
needing a new MappedU8Buffer type.
Test catalogue (all passing locally on RTX 3050 Ti):
1. half_flat_long_emits_half_position_size — a9 sizing + a9-on-short
no-op + odd-lot ceil(3/2)=2 invariant
2. half_flat_short_emits_half_position_size — symmetric a10 case
3. unit_state_transitions — sentinel-zero bootstrap OPEN (was-flat→
long) + CLOSE (long→flat) + prev_pos_lots tracker advance +
only-slot-0-active invariant (slots 1-3 stay 0)
4. trail_mutate_tighten_loosen_reciprocal — a7 then a8 returns trail
to original within 1e-5 + inactive units don't mutate + non-trail
action (Hold) passes through
5. trail_stop_check_overrides_action_on_breach — long breach overrides
Hold→FlatFromLong (a3) + no-breach leaves Hold + short breach
overrides Hold→FlatFromShort (a4) (symmetry)
Bug caught during test authoring: IntegratedTrainer::new allocates
isv_d as all-zeros; ISV bootstrap defaults are written only during
the first step_with_lobsim, not at construction. Tests must explicitly
seed every ISV slot their kernel reads — in particular RL_TRAIL_MAX
for the loosen branch (fminf(0, x) silently zeroes trail_distance).
Per feedback_no_sp_or_version_prefixes_in_file_names: file named by
WHAT it tests (trade_management_kernels), not WHICH spec phase
introduced it. Same for the #[test] fn identifiers.
|
||
|
jgrusewski
|
705d6c156b |
audit: ISV-ify 10 more design constants — Schulman + bootstraps + streaming α
Per `feedback_isv_for_adaptive_bounds` + user "do all except floors
and clamp bounds": 10 more constants moved from kernel-side `#define`s
into ISV slots (78 slots total now).
## Slot additions (468-477)
RL_SCHULMAN_TOLERANCE_INDEX (468, =1.5) — shared by 4 controllers
RL_SCHULMAN_ADJUST_RATE_INDEX (469, =1.5) — shared by 4 controllers
RL_STREAM_ALPHA_INDEX (470, =0.05) — shared by var + kurt streaming
RL_KURT_GAUSSIAN_INDEX (471, =3.0)
RL_KURT_NOISE_FLOOR_INDEX (472, =1.0)
RL_TAU_BOOTSTRAP_INDEX (473, =0.005)
RL_EPS_BOOTSTRAP_INDEX (474, =0.2)
RL_ROLLOUT_BOOTSTRAP_INDEX (475, =2048)
RL_REWARD_SCALE_BOOTSTRAP_INDEX (476, =1.0)
RL_PPO_RATIO_CLAMP_BOOTSTRAP_INDEX (477, =10.0)
## Skipped (per user "do all except floors and clamp bounds")
* `*_MIN`/`*_MAX` clamp bounds (algebraic domain — risk γ=1.5 nonsense)
* Numerical floors: ABS_MEAN_FLOOR=1e-6, M2_SQ_FLOOR=1e-12, EPS_PNL=1e-3
(risk div-by-zero if mis-tuned)
* C51 atom layout (V_MIN/V_MAX) — architecture, not config
## Wiring
* Shared Schulman pattern: 4 controllers (ppo_clip, target_tau,
rollout_steps, plus per_α independent KURT slots) now read TOLERANCE
+ ADJUST_RATE from the same 2 ISV slots. Single source of truth.
* Each controller's bootstrap (1st-emit on sentinel-zero) reads
isv[*_BOOTSTRAP_INDEX] instead of #define value. The `prev ==
BOOTSTRAP` first-observation replace-direct check also reads from
ISV.
* 2 streaming kernels (var + kurt) share RL_STREAM_ALPHA_INDEX.
## Diag bake-in
JSONL `isv_config` block grows by 10 new fields: schulman_tolerance,
schulman_adjust_rate, stream_alpha, kurt_gaussian, kurt_noise_floor,
tau_bootstrap, eps_bootstrap, rollout_bootstrap,
reward_scale_bootstrap, ppo_ratio_clamp_bootstrap. Total isv_config
fields: 26.
Also includes windowed action_entropy fix (was structurally 0 at
b_size=1) — accumulates EMA-smoothed action distribution over
~1k-step window, computes entropy on the windowed dist. Makes the
exploration metric meaningful at b_size=1.
## Slot total
RL_SLOTS_END: 468 → 478. **78 total ISV slots.**
## Verified gates (local sm_86)
G1 isv_bootstrap ✅ (with 10 new assertions)
G3 controllers ✅
G4 target_update ✅
integrated_smoke ✅
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
827a0e9416 |
fix(rl): ISV-ify ALL remaining tunable design constants (10 new slots)
Per `feedback_isv_for_adaptive_bounds`: every controller design knob
that's genuinely tunable now lives in ISV instead of as a kernel-side
`#define`. Tuning is a re-seed (kernel launch with new arg) rather
than a recompile.
## New ISV slots (10 design constants)
RL_REWARD_CLAMP_WIN_INDEX (452, =1.0) apply_reward_scale
RL_REWARD_CLAMP_LOSS_INDEX (453, =3.0) apply_reward_scale
RL_KL_TARGET_INDEX (454, =0.01) rl_ppo_clip_controller
RL_IMPROVEMENT_THRESHOLD_INDEX (455, =0.99) rl_lr_controller
RL_PLATEAU_PATIENCE_INDEX (456, =1000.0) rl_lr_controller
RL_DIV_TARGET_INDEX (457, =0.01) rl_target_tau_controller
RL_ENTROPY_TARGET_FRAC_INDEX (458, =0.7) rl_entropy_coef_controller
RL_KURT_LIFT_SCALE_INDEX (459, =7.0) rl_per_alpha_controller
RL_PPO_CLAMP_MARGIN_INDEX (460, =10.0) rl_ppo_ratio_clamp_controller
RL_LR_WARMUP_STEPS_INDEX (461, =2000.0) rl_lr_controller
RL_SLOTS_END: 452 → 462.
## Constants NOT converted (truly fundamental)
* All `*_INDEX` (ABI)
* All `*_MIN`/`*_MAX` clamp bounds (algebraic domain)
* All `*_BOOTSTRAP` (one-shot init)
* `WIENER_ALPHA_FLOOR` (per pearl_wiener_alpha_floor_for_nonstationary)
* Schulman pattern parameters (`*_TOLERANCE`/`*_ADJUST_RATE`)
* C51 (`Q_N_ATOMS`, `V_MIN/MAX`, `N_ACTIONS`)
* Kernel numerics (`STREAM_ALPHA`, `ABS_MEAN_FLOOR`, `EPS_PNL`)
* `KURT_GAUSSIAN` (statistical constant = 3.0 for Gaussian)
* `KURT_NOISE_FLOOR` (defensive)
* `LR_BOOTSTRAP`/`LR_MIN`/`LR_MAX`/`LR_LOSS_EMA_ALPHA`/`DECAY_FACTOR`
## New infrastructure
New CUDA kernel `rl_isv_write.cu` — generic single-thread device-side
seeder taking `(int slot, float value)`. Trainer loops calling it
once per design constant at init. Replaces the prior pattern of
extending `rl_streaming_clamp_init`'s arg list every time a new
constant was added.
## Ordering fix
Design constants must be seeded BEFORE controllers bootstrap — the
controllers' bootstrap paths read these slots (e.g.
`rl_entropy_coef_controller` reads `RL_ENTROPY_TARGET_FRAC_INDEX`
to derive its target). Without correct ordering, controllers see
sentinel 0.0 and bootstrap to wrong values (caught by failing G1
test before commit). Seed loop runs at TOP of
`with_controllers_bootstrapped`.
## Diag bake-in
JSONL gains `isv_config` block exposing all 10 design constants per
step:
isv_config.{reward_clamp_win, reward_clamp_loss, kl_target,
improvement_threshold, plateau_patience, div_target,
entropy_target_frac, kurt_lift_scale, ppo_clamp_margin,
lr_warmup_steps}
Post-hoc analysis can correlate any controller's behaviour with the
exact design constants it saw, without grepping the source for
`#define` defaults.
## Test updates
G1 (isv_bootstrap) + G3 (r5_controllers) — skip 10 new design-
constant slots in sentinel-zero loop, assert seeded values
separately.
## Verified gates (local sm_86)
G1 isv_bootstrap ✅ (with 10 new assertions)
G3 controllers ✅
G4 target_update ✅
integrated_smoke ✅
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
644fbe0348 |
fix(rl): ISV-driven K-loop divisor + max ceiling (slot 450, 451)
f2ggr confirmed K-loop wiring works mechanically but K=8 firing on
22 % of steps over-trained at b_size=1: KL excursions to 12.44
(vs prior 3.4e-4), policy overshoot, reward/trade -$0.585 → -$0.723.
Per `feedback_isv_for_adaptive_bounds` the K-loop config must live
in ISV, not as hardcoded values in the trainer. Two new slots:
RL_K_LOOP_DIVISOR_INDEX (450) — divides n_rollout_steps to get K
Default 2048 (matches ROLLOUT_BOOTSTRAP
so K=1 at controller bootstrap)
RL_K_LOOP_MAX_INDEX (451) — clamp ceiling on K
Default 4 (was hardcoded 8; halved
to prevent gradient overtraining)
K computation in step_with_lobsim now reads both from ISV:
K = clamp(isv[404] / isv[450], 1, isv[451])
Halves worst-case overtraining while preserving the controller
cascade activation (KL above noise floor, ε actively adapting,
ratio_clamp firing). Distribution shifts from K=8 @ 22% → K=4 @ 22%
(half the gradient updates in the high-noise case).
## Wiring
`rl_streaming_clamp_init.cu` extended to seed 5 ISV-resident design
constants (was 3): adv_var_clamp, td_kurt_clamp, adv_var_target,
k_loop_divisor, k_loop_max. Still one kernel call, no HtoD.
## Diag bake-in
JSONL `k_updates` field replaced with `k_loop` block:
k_loop.k_updates — actual K used this step
k_loop.divisor — current divisor (reads isv[450])
k_loop.max — current max (reads isv[451])
Post-hoc analysis can verify the K-computation by independently
recomputing K from isv[404] / k_loop.divisor.
## Slot allocation
RL_SLOTS_END: 450 → 452 (+2 new config slots).
## Test updates
G1 + G3 skip slots 450, 451 in sentinel-zero loop and assert seeded
values (2048.0 + 4.0) separately.
## Verified gates (local sm_86)
G1 isv_bootstrap ✅
G3 controllers ✅
G4 target_update ✅
integrated_smoke ✅
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
95dcc4e312 |
fix(rl): ISV-driven ADV_VAR_RATIO_TARGET for rl_rollout_steps_controller
cvf86 controller_branch diag (commit
|
||
|
jgrusewski
|
66115007ab |
fix(rl): ISV-driven output clamp on streaming var/kurtosis kernels
gxhr8 confirmed the streaming kernels work — both formerly-dead
controllers (rl_rollout_steps, rl_per_alpha) now adapt instead of
pegging at MIN. But the unclamped streaming outputs reached
advantage_var_ratio = 3e5 (when streaming-mean passed through zero
and `var/|mean|` blew up under the 1e-6 denominator floor) and
td_kurtosis = 50.6, pegging both downstream controllers at MAX
instead. Per_α at MAX over-concentrates PER sampling on outliers,
which hurts distributional Q learning (best l_q window regressed
from 2.41 → 2.69 between pdgxn and gxhr8).
## Fix: ISV-resident output clamp ceilings
Two new ISV slots hold the streaming-kernel output ceilings:
RL_ADV_VAR_RATIO_CLAMP_INDEX = 447 (default 100.0)
RL_TD_KURTOSIS_CLAMP_INDEX = 448 (default 30.0)
* 100.0 for var_ratio = 1000× ADV_VAR_RATIO_TARGET (= 0.1) — wide
enough that healthy signal (typical 1-10) passes through, tight
enough that 3e5 outliers don't peg rollout_steps.
* 30.0 for kurtosis = 3× (KURT_GAUSSIAN + KURT_LIFT_SCALE) — lets
the full per_α response range engage on heavy-tailed signal
(≤ 10), bounds runaway above that.
Per `feedback_isv_for_adaptive_bounds`: the clamps live in ISV
(visible in diag, modifiable at runtime via re-launching the init
kernel or a future adaptive controller) rather than as kernel-side
`#define`s.
## Seeding (no HtoD per feedback_no_htod_htoh_only_mapped_pinned)
New device kernel `rl_streaming_clamp_init.cu` — single thread,
writes both clamp ceilings directly to ISV. Launched once at the
end of `with_controllers_bootstrapped` alongside the 8 existing
controller-bootstrap launches. Zero host→device transfer.
## Diag bake-in (per user request "ensure to bake in diags")
JSONL gains a new `streaming` block exposing:
* `streaming.adv_var.{mean, m2, clamp}`
* `streaming.td_kurt.{mean, m2, m4, clamp}`
Cross-check: when consumer-input slot (RL_ADVANTAGE_VAR_RATIO_EMA_INDEX
or RL_TD_KURTOSIS_EMA_INDEX) reads exactly the same value as
`streaming.*.clamp`, the clamp fired this step.
## Test updates
G1 (isv_bootstrap) + G3 (r5_controllers) blanket-assert that
ISV[417..END] is sentinel-zero at bootstrap. Both new slots are
seeded to non-zero values by rl_streaming_clamp_init during
bootstrap, so both tests skip these slots in the loop and assert
the seeded values separately.
## Verified gates (local sm_86)
G1 isv_bootstrap ✅
G3 controllers ✅
G4 target_update ✅
integrated_smoke ✅
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
53aeef099b |
feat(rl): ISV-driven PPO importance-ratio clamp + log-ratio diagnostic
pt67l confirmed reward-scale + V-target clamp eliminate V regression
spikes — but exposed a residual: |l_pi| max=586 with mean 0.22. Root
cause: PPO's clip(r, 1-ε, 1+ε) bounds the loss only when surr2 is
the active min. The unclipped branch IS active when A<0,r>1+ε
(surr1=A·r is then more negative than surr2=A·(1+ε), so min selects
surr1) and when A>0,r<1-ε. In the first case `r` can blow up: we've
seen r reach 1e10 from policy drift over a multi-step rollout
producing l_pi=O(1e10) spikes that contaminate the loss-balance
controller and the LR controller's plateau detection.
## Fix: ISV-driven ratio clamp
Per `feedback_isv_for_adaptive_bounds` and
`pearl_controller_anchors_isv_driven`: the clamp ceiling lives in
ISV[RL_PPO_RATIO_CLAMP_MAX_INDEX = 440], not as a hardcoded #define.
New controller `rl_ppo_ratio_clamp_controller.cu`:
* Anchors on the (already KL-adaptive) PPO clip ε at ISV[402]
* target = (1 + ε) × PPO_CLAMP_MARGIN (MARGIN = 10.0)
* Wiener-α blend with floor 0.4 per
pearl_wiener_alpha_floor_for_nonstationary (ε is non-stationary)
* Permanent floor 2.0 / ceiling 1000 per
pearl_blend_formulas_must_have_permanent_floor
* Bootstrap 10.0, replace-directly on first non-bootstrap ε
observation per pearl_first_observation_bootstrap
When ε is small (rl_ppo_clip_controller seeing low KL → tight clip
band), the ratio clamp tightens — outliers should be rare anomalies.
When ε widens (large KL → wide clip band), the clamp widens
proportionally — outliers are expected so we permit more
magnitude before bounding.
## Wiring
ppo_clipped_surrogate_fwd and _bwd both read
isv[RL_PPO_RATIO_CLAMP_MAX_INDEX] and clamp ratio to
[1/ratio_max, ratio_max] before forming surr1/surr2. The clamp is
forward-only in effect (bwd gates pg_grad inside [1-ε, 1+ε] anyway
so gradients were already bounded), but bounding the FORWARD ratio
keeps l_pi sane for the controllers downstream.
The new controller is wired into both:
* `with_controllers_bootstrapped` — bootstrap launch alongside
the other 7 R1 controllers
* `launch_rl_controllers_per_step` — per-step refresh alongside
the other 7 R5 controllers
## Diagnostic: per-step max |log_ratio|
New kernel `ppo_log_ratio_abs_max_b.cu` (same tree-reduce shape as
rl_kl_approx_b) writes per-batch max(|log π_new − log π_old|) to
ISV[RL_PPO_LOG_RATIO_ABS_MAX_INDEX = 441]. Launched right after
rl_kl_approx_b (uses the same log_pi_old_d + pi_log_prob_d inputs).
Surfaces in diag JSONL as:
"ppo": {
"ratio_clamp_max": isv[440], # adaptive ceiling
"log_ratio_abs_max": isv[441] # per-step observed max
}
The clamp fires when log_ratio_abs_max > ln(ratio_clamp_max).
For ratio_clamp_max = 10, ln = 2.30. Healthy training has
log_ratio_abs_max well below this most steps; outliers touch or
exceed it on rare excursions which the clamp bounds before they
pollute l_pi.
## Slot allocation
RL_PPO_RATIO_CLAMP_MAX_INDEX = 440 (controller output)
RL_PPO_LOG_RATIO_ABS_MAX_INDEX = 441 (per-step diag)
RL_SLOTS_END = 442 (was 440)
## Test updates
G1 (isv_bootstrap) + G3 (r5_controllers) blanket-assert ISV[417..END]
== 0.0 to catch slot-wiring bugs. Slot 440 is now a controller
OUTPUT bootstrapped to 10.0, so both tests skip it in the loop and
assert == 10.0 separately.
## Verified gates (local sm_86)
G1 isv_bootstrap ✅ (with new slot-440 assertion)
G3 controllers ✅
G4 target_update ✅
G6 r7d_per_wiring ✅
integrated_smoke ✅
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
fd415d9b17 |
fix(rl): apply derive-from-input bootstrap to γ + coef controllers
Systematic completion of the per_α fix (commit
|
||
|
jgrusewski
|
0857d40acd |
fix(rl): rl_per_alpha bootstrap dead-zone at canonical kurtosis
Real production bug surfaced by the R9 G3 local smoke (not the
test-fixture bug the prior commit's message claimed). The
`rl_per_alpha_controller` kernel hardcoded its bootstrap value to
`PER_ALPHA_BOOTSTRAP = 0.6` (canonical PER default per Schaul 2016).
The per-step path's target formula `target = 0.4 + 0.2 × (kurt − 3) / 7`
maps kurt=10 (the canonical heavy-tailed market kurtosis) to **exactly
0.6** — the bootstrap. The Wiener-α blend `(1 − α) · prev + α · target`
then produces 0.6 from any α when prev = target = 0.6, freezing the
controller at the bootstrap value for any input near the canonical
market regime.
This is a real production behaviour bug, not a test fixture bug:
in any market session where TD-error kurtosis sits near canonical 10
(which is the *most common* regime — that's WHY 0.6 is the canonical
PER default), the controller never adapts off bootstrap. The
adaptation mechanism is effectively disabled for typical inputs and
only fires when kurtosis drifts away. The prior commit
(
|
||
|
jgrusewski
|
ee24f0a303 |
fix(rl): R9 local-smoke prep — two test-fixture bugs
Two `pearl_tests_must_prove_not_lock_observations` violations
surfaced during the R9 pre-cluster validation sweep on the dev RTX
3050 Ti (sm_86). Neither was a bug in the trainer or kernels — both
were test fixtures that asserted observed-value coincidences rather
than invariants. Per the canonical pearl, observed-value tests
become bug-locks (the SP16 T3 sp16_phase3_alpha_low_in_steady_state
incident was an assertion `α<0.40` matching the bug itself).
## g3_per_step_controllers_move_isv_outputs_when_fed_real_emas
The fixture fed `RL_TD_KURTOSIS_EMA = 10.0` to the rl_per_alpha
controller, expecting ISV[405] to move off its bootstrap 0.6 after
the Wiener blend. But the kernel's target formula at td_kurtosis=10
maps to **exactly** the bootstrap:
target = 0.4 + 0.2 × (10 − 3) / 7 = 0.6
The Wiener blend `(1−α)·prev + α·target` then produces 0.6 from any
α, so the controller can't move off bootstrap. The assertion was
asserting a coincidence — fixed by picking `td_kurtosis = 20.0`
which lands at `target = 0.886`, distinct from the 0.6 bootstrap.
With the fix all 7 controllers move (γ→0.9, τ→0.023, ε→0.14,
coef→0.0154, n_roll→2867, per_α→0.714, scale→0.608).
The kernel itself is correct — the test was wrong.
## integrated_trainer_step_with_lobsim_runs_without_panic
Asserted `λ_sum ≈ 1.0` for the loss-balance λs. But
`LossLambdas::default()` returns each λ=1.0 (sum = 5.0) with the
`/5.0` divide applied at the trainer's loss-combine site so each
head's contribution is `lambda/5.0`. The "sum=1" assertion was
based on a normalization that the trainer never used. Loosened to
the actual invariant we care about ("every head has a finite
positive λ so the encoder receives real-valued gradient") which
survives any future controller-driven λ re-weighting.
## R9 local-smoke results (all gates green on sm_86)
```
G1 isv_bootstrap ✅ γ=0.99 τ=0.005 ε=0.2
coef=0.01 n_roll=2048
per_α=0.6 scale=1.0
R3 r3_ema_advantage (3 tests) ✅ bootstrap + per-step EMA +
advantage/return formula
R4 r4_action_kernels (3 tests) ✅ Thompson + argmax + log_pi
G3 controllers_emit ✅ all 7 ISV outputs moved
G4 target_soft_update ✅ Polyak τ=0.005 applied
G6 r7d_per_wiring ✅ buffer 0→5→8 + sample size
end integrated_trainer_smoke ✅ all 5 head losses finite
```
Confirms R7c-data + R7d run end-to-end on real CUDA. Next R9 step
(cluster smoke via scripts/argo-alpha-rl.sh + multi-fold G8) requires
git push + cluster credits — paused per the chosen R9 path "stop
before cluster submission."
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
c7ccf0c301 |
feat(rl): R7d — PER wired + off-policy DQN with stop-grad on encoder
Closes plan A9 (rebuild plan's "PER wiring" R7 scope second half;
R7c-data shipped the first half last commit). The `ReplayBuffer` in
`src/rl/replay.rs` has sat as dead code since Phase C — this commit
makes it load-bearing per `feedback_always_per` ("PER always enabled;
non-PER paths are dead code").
## Architecture: off-policy Q + on-policy PPO + V + stop-grad encoder
Shared-encoder pattern with the canonical off-policy + shared-encoder
discipline: the Q head trains from PER-sampled past transitions,
PPO + V train on current-step on-policy data, and the encoder receives
gradient signal ONLY from PPO + V (and BCE/aux via the perception
trainer's separate `step_batched` path). Standard pattern in SAC,
R2D2, IMPALA.
Stop-grad is implemented by computing Q's `grad_h_t` (via
`backward_to_w_b_h(sampled_h_t, ...)`) but NOT accumulating it into
`grad_h_t_combined_d` — the encoder backward only sees π + V
contributions. Per `feedback_no_hiding` the discarded buffer is
allocated and written (the kernel API requires the writeback target);
the discard is a deliberate design call documented at the
accumulation site.
## Wiring summary
### Kernel: `dqn_distributional_q_bwd`
* New `loss_per_batch [B]` output. Atom 0 of each block writes the
per-sample CE loss (non-atomic — single writer per batch).
`loss_out [1]` continues to atomicAdd the scalar sum for the
diagnostic total. Build.rs cache bust v30.
### `DqnHead::backward_logits` (Rust wrapper)
* New `loss_per_batch: &mut CudaSlice<f32>` arg. Migrated atomically
in the same commit per `feedback_no_partial_refactor` — only
caller is the integrated trainer.
### `IntegratedTrainerConfig`
* New `per_capacity: usize` (default 4096, matches `replay.rs` doc
ceiling for naive O(N) sampling).
* New `per_seed: u64` (default 0x9E37_79B9_7F4A_7C15).
* `Default` impl added so test fixtures forward-compat via
`..IntegratedTrainerConfig::default()`. All 5 existing test
fixtures migrated.
### `IntegratedTrainer`
* New fields: `replay: ReplayBuffer`, `sampled_h_t_d`,
`sampled_h_tp1_d`, `sampled_actions_d`, `sampled_rewards_d`,
`sampled_dones_d`, `sampled_next_actions_d`, `td_per_sample_d`.
* New methods: `push_to_replay(b_size)` — DtoH per-batch metadata
(action/reward/done/log_pi_old) + alloc per-transition
`CudaSlice<f32>(HIDDEN_DIM)` ×2 + DtoD per-batch slice copies +
push to `ReplayBuffer`. `sample_and_gather(b_size)` — read
per_α from ISV[405], call `replay.sample_indices`, gather sampled
transitions' h_t/h_tp1 device payloads via per-batch DtoD into
`sampled_h_t_d` / `sampled_h_tp1_d`, HtoD upload action/reward/done.
### `step_with_lobsim` orchestration
After `compute_advantage_return` and BEFORE `step_synthetic`:
1. DtoH full ISV slice to refresh `isv_host` (so PER reads ISV[405]
for per_α).
2. `push_to_replay(b_size)` — push current step's transitions.
3. `sample_and_gather(b_size)` — return `per_indices` for the
priority update.
4. `step_synthetic(snapshots)` — runs π + V on current-step h_t,
Q on SAMPLED h_t (off-policy).
5. DtoH `td_per_sample_d` → host; `replay.update_priorities(
per_indices, td_per_sample_host)`.
6. Target-net soft update (unchanged from R5).
### `step_synthetic` redirects (Q path → sampled, π/V stay on-policy)
* Q forward: `forward(&self.sampled_h_t_d)` (was `h_t_borrow`).
* New: forward online Q on `&self.sampled_h_tp1_d` → local scratch +
`argmax_expected_q` → `self.sampled_next_actions_d`. The
Double-DQN argmax MUST be recomputed each step (online net weights
drift faster than transitions recycle through replay; storing
argmax at push time would feed stale-action data into the
projection).
* `forward_target(&self.sampled_h_tp1_d)` (was `&self.h_tp1_d`).
* `select_action_atoms(..., &self.sampled_next_actions_d, ...)`
(was `&self.next_actions_d`).
* `project_bellman_target(..., &self.sampled_rewards_d,
&self.sampled_dones_d, ...)` (was `rewards_d` / `dones_d`).
* `backward_logits(..., &self.sampled_actions_d, ...,
&mut self.td_per_sample_d, ...)` (added per-sample loss output).
* `backward_to_w_b_h(&self.sampled_h_t_d, ...)` (was `h_t_borrow`).
* Q grad_h_t accumulation REMOVED from Step 10 (stop-grad).
## Test: r7d_per_wiring.rs (gate G6)
Three invariants per `pearl_tests_must_prove_not_lock_observations`:
1. `replay.len()` grows by exactly `b_size` per `step_with_lobsim`
call (push semantics).
2. `sample_indices(b_size, α)` returns vec of length `b_size` on a
non-empty buffer.
3. Buffer caps at `per_capacity` (ring-with-random-replacement).
Drives 15 steps with `per_capacity=8`, asserts growth 0→5→8 across
the cap boundary.
## Acceptable host traffic this commit adds
* Per-step DtoH of 4 × b_size scalars (action/reward/done/log_pi_old)
for PER push metadata.
* Per-step DtoH of b_size floats (td_per_sample_d) for
update_priorities.
* Per-step HtoD of 3 × b_size scalars (sampled action/reward/done)
for sampled metadata gather.
* Per-step DtoD of 2 × b_size × HIDDEN_DIM floats (per-batch h_t /
h_tp1 slices) for PER push + gather.
PER bookkeeping is a control-plane operation by design (host-side
priority/index management); the device-side training hot path
(encoder, Q/π/V forward/backward, Adam) stays GPU-pure. GPU sum-tree
+ device-resident transitions are a Phase R-future optimization
flagged in `replay.rs`'s doc.
## What's NOT in this commit
* Q `loss_per_batch [B]` is now wired through `backward_logits` but
the DtoH happens inside step_with_lobsim (not inside
step_synthetic). Earlier R7d sketches considered a separate
`dqn_offpolicy_step` method; the in-step_synthetic redirect
approach landed because it touches fewer lines + reuses the
existing scratch buffer allocations + matches the trainer's
established λ-weighted multi-head pattern. A future refactor
could split for clarity.
Local sm_86 smoke gates: `cargo test -p ml-alpha --test
r7d_per_wiring -- --ignored --nocapture` (G6) +
`integrated_trainer_smoke` (end-to-end). Cluster smoke deferred to
R9.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
acde2e8932 |
fix(rl): R7c-data — true h_{t+1}/V(s_{t+1}) closes Bellman approximation
Closes the long-standing "h_t as proxy for s_{t+1}'s encoder
representation" approximation introduced in Phase E.2's Bellman target
build (canonical comment at the call site: "A future enhancement
(Phase E.3 LobSim integration) will pass next_h_t separately"). The
approximation also leaked into compute_advantage_return's V(s_{t+1})
input — R7b's `v_tp1_d_ref = &v_pred_d` alias — and into R4's
argmax_expected_q kernel call, which had been computing the
Double-DQN argmax on online Q at h_t since R4 first wired it.
Three downstream consumers now read TRUE h_{t+1}:
* `value_head.forward(&self.h_tp1_d) → v_pred_tp1_d`, fed to
`compute_advantage_return` as the canonical TD target V(s_{t+1}).
Was an alias of v_pred_d (V(s_t)) — bootstrap was wrong by one
time index.
* `dqn_head.forward(&self.h_tp1_d) → q_logits_tp1_d`, fed to
`argmax_expected_q` for `next_actions_d` (Double-DQN online-Q
argmax on h_{t+1}, not h_t).
* `dqn_head.forward_target(&self.h_tp1_d)` inside step_synthetic's
Bellman target build. Replaces the h_t-as-proxy comment with the
R7c data-correctness lift inline-doc.
Wiring mechanics:
* New trainer field `h_tp1_d: CudaSlice<f32>` (`[B × HIDDEN_DIM]`).
Zero-initialised; populated each step by step_with_lobsim.
* `step_with_lobsim` signature gains a `next_snapshots:
&[Mbp10RawInput]` parameter (caller — R8's CLI binary — uses
`MultiHorizonLoader::next_sequence_pair` from R2 to load adjacent
`(s_t, s_{t+1})` windows from real MBP-10 data).
* Encoder is now called TWICE in step_with_lobsim:
1. `forward_encoder(next_snapshots)` first → DtoD copy
`perception.h_t_d → self.h_tp1_d` immediately (before any
consumer reads the slot).
2. `forward_encoder(snapshots)` second → leaves perception's
internal forward state (`h_new_per_k_d`, CfC `h_state_d`)
primed for step_synthetic's encoder backward (which still
redundantly re-runs `forward_encoder(snapshots)` per the
pre-existing pattern — separate compute-redundancy fuse for
Phase R-future).
* Three encoder forwards total per step (down from R7b's 2: one
in step_with_lobsim, one in step_synthetic — R7c adds the
second-snapshot forward in step_with_lobsim). The CfC encoder is
deterministic given its input window (per the canonical
step_with_lobsim header comment), so calling forward_encoder
twice on different inputs in a row yields independent h_t and
h_{t+1} via the trainer's own DtoD copy.
Test impact:
* `integrated_trainer_smoke.rs` passes a `next_snapshots` second
window (synthesised as a +1-tick shift of the snapshot window
— production callers will use R2's `next_sequence_pair` on real
MBP-10 data). Smoke continues to assert finite losses across the
five heads.
Scope split note: this commit handles ONLY the data-correctness
half of plan A9 (rebuild plan's "PER buffer + true V(s_{t+1})
Bellman target" R7 scope). The off-policy DQN-via-PER half lands
in the next commit (R7d): Replay-buffer push, sample,
stop-grad-on-encoder Q redirect, and per-sample |TD| → update_priorities.
Split is per `pearl_no_deferrals_for_complementary_fixes`'s
sequencing-with-architectural-justification carve-out: R7d's PER
push requires the correct h_{t+1} this commit provides, so it MUST
sequence after — and the data-correctness fix is independently
useful (the on-policy DQN path now produces correct Bellman
targets even without the off-policy replay).
Local sm_86 smoke: `cargo test -p ml-alpha --test
integrated_trainer_smoke -- --ignored --nocapture` is the gate.
Cluster smoke deferred to R9.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
7e7c8b5d90 |
feat(rl): R6 — replace LobEnv with kernel-driven GPU-pure env step
Replaces the flawed Phase F+G LobEnv trait + LobSimEnvAdapter +
MockLobEnv fixture with a GPU-pure env-interaction path.
DELETED:
- LobEnv trait (host-method-per-batch surface)
- MockLobEnv fixture (the toy-bandit pattern that hid the production defects)
ADDED:
- RlLobBackend trait (src/rl/reward.rs): narrow device-oriented
surface (apply_snapshot, pos_and_market_targets_mut, pos_d,
pos_bytes, step_fill_from_market_targets, n_backtests). Single
purpose: break the ml-alpha ↔ ml-backtesting dep cycle. No
mocking layer.
- 3 new GPU-pure kernels (cuda/):
* extract_realized_pnl_delta.cu — reads pos.realized_pnl +
position_lots from device Pos array, writes per-batch
(reward delta, done flag), updates trainer's prev_* buffers.
* apply_reward_scale.cu — element-wise rewards *= ISV[406].
* actions_to_market_targets.cu — 9-action grid → market_targets
{side, size} on device, reading current position_lots for
conditional Flat-from-Long/Short.
- LobSimCuda impls RlLobBackend (ml-backtesting/src/sim/mod.rs)
plus a new step_fill_from_market_targets entry that runs
submit_market_immediate + step_pnl_track without the host-side
targets-vec build. pos_and_market_targets_mut returns disjoint
field borrows (&pos_d, &mut market_targets_d).
- IntegratedTrainer: 3 cubin includes, 3 module/function fields,
launch_apply_reward_scale launcher, prev_realized_pnl_d /
prev_position_lots_d / rewards_d / dones_d buffers,
step_with_lobsim signature switched to RlLobBackend, body's
Step 5 rewritten as kernel-driven (no per-batch host loop, no
individual submit_action calls).
R6 PARTIAL — work R7 lifts:
- Thompson + log_pi stay host (R7 uses R4 kernels)
- mean_abs_pnl EMA stays host (R7 uses R3 ema_update_on_done)
- Advantage/return stays host (R7 uses R3 compute_advantage_return)
- 4 final DtoH copies for step_synthetic's host-slice signature
(R7 lifts step_after_encoder_forward to device-buffer args)
The "DtoH the device rewards/dones at end" comment in
step_with_lobsim documents the boundary. After R7, hot path has
zero host loops other than the now-unused Thompson host loop
(which R7 retires in favour of rl_action_kernel from R4).
Tests:
- integrated_trainer_smoke.rs: rewritten — real LobSimCuda with
synthetic book, one step_with_lobsim call, asserts finite losses
+ λs sum to 1. Smoke gate, not a convergence test.
- dqn_toy.rs, ppo_toy.rs: retired (empty stubs documenting
rationale). Files preserved so rename history survives. The
MockLobEnv toy-bandit pattern intrinsically couldn't catch the
production defects #1, #2, #5 that motivated this rebuild.
ml-backtesting added as ml-alpha dev-dep (cycle-safe: production
direction is ml-backtesting → ml-alpha; dev edge only loads when
building ml-alpha's own tests/examples).
Build cache-bust v29. cargo check + cargo build for all R-phase
tests green (pre-existing heads_bit_equiv index-OOB persists).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
0a32a3bb89 |
feat(rl): R5 — wire 7 controllers per-step + target-net soft update
Closes defects #3 (controllers never launched) and #4 (target net never soft-updated) from the flawed Phase F+G arc. CONTROLLER SIGNATURE CHANGE (all 7 .cu files): Scalar input arg → int input_slot. Each controller now reads its EMA input from ISV[input_slot] directly inside the kernel, eliminating the 7 DtoH-per-step host roundtrips a scalar-arg signature would have required — per feedback_cpu_is_read_only (hot-path must be GPU-pure). Bootstrap path unchanged: kernel reads its output slot, sees sentinel zero, writes *_BOOTSTRAP, and returns BEFORE the input_slot read. So R1's launch_isv_controller_3arg(controller_fn, alpha=0.4, input_slot) works both at bootstrap (input read deferred via early return) and at per-step (input slot has real EMA observation from R3 producers). PER-STEP CONTROLLER LAUNCHER: New IntegratedTrainer::launch_rl_controllers_per_step() fires all 7 controllers in sequence, each with its dedicated EMA input slot: ISV[400] γ ← ISV[417] MEAN_TRADE_DURATION_EMA ISV[401] τ ← ISV[418] Q_DIVERGENCE_EMA ISV[402] ε ← ISV[419] KL_PI_EMA ISV[403] entropy_coef ← ISV[420] ENTROPY_OBSERVED_EMA ISV[404] n_rollout_steps← ISV[421] ADVANTAGE_VAR_RATIO_EMA ISV[405] per_α ← ISV[422] TD_KURTOSIS_EMA ISV[406] reward_scale ← ISV[423] MEAN_ABS_PNL_EMA R1's with_controllers_bootstrapped also updated to pass the input slot indices (the bootstrap path still ignores them via early return). TARGET-NET SOFT UPDATE (defect #4): New cuda/dqn_target_soft_update.cu — element-wise target[i] = (1-τ)·target[i] + τ·current[i] reading τ from ISV[401]. Trivially parallel, no atomicAdd. DqnHead gains target_soft_update_fn + _target_soft_update_module fields + soft_update_target(&isv_d) method that fires the kernel twice (weights + biases). R6 calls this from step_with_lobsim after the Q-head Adam update. GATE TESTS (tests/r5_controllers_and_soft_update.rs): G3: g3_per_step_controllers_move_isv_outputs_when_fed_real_emas - Verifies R1 bootstrap pre-conditions (all 7 output slots at documented bootstrap values; all 7 EMA-input slots at sentinel 0). - Populates each EMA-input slot with a distinct non-zero value via R3's ema_update_per_step bootstrap path (different values per slot so a wrong-slot wiring bug would produce out-of-range outputs). - Verifies the EMA producers wrote what we expected (sanity). - Fires launch_rl_controllers_per_step. - Asserts each output slot moved off its bootstrap value (catches "controller doesn't fire" / "reads wrong slot" / "dead kernel"). G4: g4_dqn_target_soft_update_implements_polyak_formula - Force-overwrite w_d with all-ones (breaks the w==target init symmetry so soft_update has something to blend). - Snapshot w_target (Xavier init values). - Fire dqn_head.soft_update_target with R1-bootstrapped τ=0.005. - For sample indices: assert target_after[i] equals (1-τ)·target_before[i] + τ·1.0 within 1e-6 (exact algebraic identity, not a CPU reference — kernel IS the kernel). - Negative invariant: at least one element changed. Per feedback_no_cpu_test_fallbacks: G3 oracle is the invariant "output != bootstrap after non-trivial input"; G4 oracle is the algebraic identity (1-τ)·a + τ·b applied to the SAME numbers the kernel saw — not a parallel CPU implementation. Build cache-bust v28. cargo check + cargo build --tests on ml-alpha green for all R-phase tests. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
jgrusewski
|
27feb94a49 |
feat(rl): R4 — GPU-resident action sampling kernels
Closes defect #5 prerequisites for the GPU-pure step_with_lobsim that lands in R6. Replaces the host Thompson + host argmax + host log-softmax loops the flawed Phase F shipped in step_with_lobsim (which violated feedback_cpu_is_read_only with 5 DtoH copies + host per-action loops + HtoD action upload per training step). Three new kernels: 1. rl_action_kernel.cu — Thompson sampler over the C51 atom distribution. One block per batch, N_ACTIONS=9 threads. Each thread softmaxes its action's Q_N_ATOMS=21 atoms, samples one atom via CDF walk, writes sampled return to shared mem. Thread 0 argmaxes over per-action sampled returns + writes actions[b] + advances per-batch PRNG state. PRNG: per-batch xorshift32 state in prng_state_d (allocated + host-seeded from cfg.dqn_seed via ChaCha8 at trainer init per pearl_scoped_init_seed_for_reproducibility, with .max(1) guard since xorshift32 freezes at 0). Each per-action thread XORs its action index (golden-ratio mixed) into a thread-local copy of the per-batch state — no inter-thread race, reproducible by (cfg.dqn_seed, b_size, step_count). No cuRAND dep. 2. argmax_expected_q.cu — Bellman-target argmax over expected Q per action. Same layout as rl_action_kernel but deterministic (no PRNG). Per pearl_thompson_for_distributional_action_selection: Thompson for rollout (rl_action_kernel), argmax for Bellman target (this kernel) — distinct kernels, distinct ISV consumers. 3. log_pi_at_action.cu — per-batch log π(actions[b] | s_b) via log-softmax + lookup. One thread per batch entry (N_ACTIONS=9 is small enough for a per-thread sequential loop). Feeds the PPO importance ratio in R6. IntegratedTrainer gains: - 3 cubin includes (rl_action_kernel, argmax_expected_q, log_pi_at_action) - 3 module/function field pairs - 2 new device buffers populated at init: prng_state_d: CudaSlice<u32> of length n_batch atom_supports_d: CudaSlice<f32> of length Q_N_ATOMS=21, values [Q_V_MIN, Q_V_MIN + step, …, Q_V_MAX] = linspace(-1, +1, 21) - 3 launcher methods: launch_rl_action_kernel(q_logits_d, actions_d, b_size) launch_argmax_expected_q(q_logits_d, next_actions_d, b_size) launch_log_pi_at_action(pi_logits_d, actions_d, log_pi_out_d, b_size) GPU-oracle tests in tests/r4_action_kernels.rs (per feedback_no_cpu_test_fallbacks every oracle is analytical, not a CPU reference): R4.1: Thompson under sharp distribution (action 5 has logit=20 on atom 20 / support +1.0; others have logit=20 on atom 0 / support −1.0) collapses to argmax — per-action dominant-atom probability ≈ 1 − 4e-8, so 100/100 trials should pick action 5. Assert ≥99/100 (tolerates one fp-rounding edge near u ≈ 1.0 in CDF walk). R4.2: argmax_expected_q picks the rewarded action under the same sharp distribution. Negative invariant: swap dominant atom to action 2 → next_action follows. R4.3: log_pi_at_action with π logits dominant at action 3 (logit=20, others=0) → log π(3) ≈ 0 within 1e-4. Negative invariant: log π(other action) ≈ −20 within 1e-3. Build cache-bust v27. cargo check + cargo build --tests on ml-alpha green (heads_bit_equiv pre-existing failure persists). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
jgrusewski
|
6d433784f4 |
feat(rl): R3 — GPU-resident EMA + advantage/return kernels
Closes defect #5 from the flawed Phase F+G arc (feedback_cpu_is_read_only violation in step_with_lobsim's host advantage + EMA loops) by landing the GPU primitives those loops will become in R6. Three new kernels, each with a GPU-oracle gate test: 1. ema_update_on_done.cu — done-gated EMA producer. - Slot-parameterised (one kernel, 3 callers in R5 covering mean_abs_pnl_ema, q_divergence_ema, td_kurtosis_ema). - Shared-mem tree reduce, no atomicAdd (feedback_no_atomicadd). - Per pearl_first_observation_bootstrap: sentinel-zero ISV → first observation replaces directly. Defers bootstrap if mean_obs == 0 to avoid writing a degenerate sentinel that would be re-bootstrapped next call. - Per pearl_wiener_alpha_floor_for_nonstationary: Wiener-α blend on subsequent calls; caller pre-floors α at 0.4. 2. ema_update_per_step.cu — per-step EMA producer (no done-gate). - Slot-parameterised (kl_pi_ema, entropy_observed_ema, advantage_var_ratio_ema, mean_trade_duration_ema in R5). - Same shared-mem tree reduce + bootstrap discipline as ema_update_on_done. 3. compute_advantage_return.cu — element-wise returns[b] = r + γ(1-done)·V(s_{t+1}); advantages[b] = returns − V(s_t). - Reads γ from ISV[400] (R1 bootstrap = 0.99). - Trivially parallel, one thread per batch entry; no atomics. Rust launchers added to IntegratedTrainer: - launch_ema_update_on_done(slot, alpha, obs_d, dones_d, b_size) - launch_ema_update_per_step(slot, alpha, obs_d, b_size) - launch_compute_advantage_return(rewards_d, dones_d, v_t_d, v_tp1_d, returns_d, advantages_d, b_size) 3 cubin includes, 3 module/function fields, loaders in new() between the rl_reward_scale_controller load and the with_controllers_bootstrapped call so the new fields are populated by struct construction. GPU-oracle tests in tests/r3_ema_advantage.rs (per feedback_no_cpu_test_fallbacks every oracle is either the kernel's documented bootstrap behaviour or an analytical property of the formula, not a CPU reference): R3.1: ema_update_on_done bootstrap path — sentinel-zero ISV + one observation k → ISV[slot] == k exactly. Negative invariant: hold-only step (dones all zero) preserves the EMA. R3.2: ema_update_per_step convergence — feed obs=5.0 for 50 steps with α=0.4 → ISV[slot] → 5.0 within 1e-4 (EMA of constant = constant). R3.3: compute_advantage_return formula — r=0, done=0, v_t=v_tp1=k, γ=0.99 → returns=γk=4.95, advantages=(γ−1)k=−0.05. Negative invariant: done=1 + r=0 zeros the future-value bootstrap (returns=0, advantages=−k). Build cache-bust v26. cargo check + cargo build --test r3_ema_advantage on ml-alpha green. Pre-existing heads_bit_equiv.rs index-out-of-bounds failure persists (unrelated; pre-Phase E). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
jgrusewski
|
0840fcfe64 |
feat(rl): R1 — ISV slot extension + 7-controller bootstrap (G1 gate)
Closes defect #1 from the flawed Phase F+G arc: ISV[400..406] were left at alloc_zeros sentinel 0 in production, causing bellman_target_projection (γ=0), ppo_clipped_surrogate (ε=0, entropy=0), and the C51 backward to train against degenerate targets that the MockLobEnv toy fixture (done=true every step, horizon=1) intrinsically could not detect. Three changes: 1. Port crates/ml-alpha/cuda/rl_reward_scale_controller.cu from the ml-alpha-phase-f-g-flawed reference branch (93 lines, unchanged). Add to build.rs KERNELS list; bump cache-bust to v25. 2. Extend src/rl/isv_slots.rs: add 7 new EMA-input slot constants (RL_MEAN_TRADE_DURATION_EMA_INDEX..RL_MEAN_ABS_PNL_EMA_INDEX), RL_SLOTS_END goes 417 -> 424. These are reserved for the EMA producer kernels Phase R3 lands; in R1 they stay at sentinel 0 (asserted by the G1 test). 3. Wire all 7 RL adaptive controllers (γ / τ / ε / entropy_coef / n_rollout_steps / per_α / reward_scale) into IntegratedTrainer: - 7 cubin includes + 7 module/function fields - All 7 loaded in new() via the existing load_cubin pattern - New fn launch_isv_controller_3arg() centralises the shared (isv*, alpha, scalar_input) launch signature - New fn with_controllers_bootstrapped() consumes self and fires each controller once against the freshly-zeroed isv_d; each kernel's first-observation-bootstrap path (per pearl_first_observation_bootstrap) sees sentinel zero in its slot and writes its canonical *_BOOTSTRAP value: ISV[400] γ = 0.99 ISV[401] τ = 0.005 ISV[402] ε = 0.2 ISV[403] entropy_coef = 0.01 ISV[404] n_rollout_steps= 2048 ISV[405] per_α = 0.6 ISV[406] reward_scale = 1.0 - new() ends with `.with_controllers_bootstrapped()?` so every trainer construction site picks this up automatically. This replaces the flawed Phase F approach of host memcpy_htod-ing canonical constants into ISV, which violated feedback_no_htod_htoh_only_mapped_pinned (tests not exempt) AND short-circuited the canonical pearl_first_observation_bootstrap pattern every other adaptive controller in the codebase uses. The launch_isv_controller_3arg helper is reused by Phase R5's per-step controller launches with real EMA inputs sourced from ISV[417..424] — at that point the Wiener-α blend kicks in and the slots adapt away from the R1 bootstrap defaults. Gate G1 (crates/ml-alpha/tests/isv_bootstrap.rs): - Construct IntegratedTrainer - memcpy_dtoh full ISV slice to host - Assert ISV[400..406] equal each kernel's #define *_BOOTSTRAP - Assert ISV[417..424] still at sentinel 0 (R3 wires producers) Per feedback_no_cpu_test_fallbacks: the oracle is the kernel's own *_BOOTSTRAP constant, not a CPU computation. Per pearl_tests_must_prove_not_lock_observations: the test asserts an invariant (bootstrap path wrote the canonical value defined by the kernel), not a tuned magic number. Build clean: cargo check + cargo build --test isv_bootstrap on ml-alpha both green. CUDA-required, #[ignore]'d for non-GPU CI. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
jgrusewski
|
9114374d25 |
feat(rl): LobEnv trait + step_with_lobsim + toy bandit activation (E.3b)
Closes Phase E of the integrated RL trainer plan (docs/superpowers/plans/2026-05-22-integrated-rl-trainer.md). The integrated trainer can now be exercised end-to-end on a real (or mock) LobSim environment. What this commit lands: - LobEnv trait in src/rl/reward.rs — narrow contract over apply_snapshot + submit_action + step_event. Chosen over a direct ml-backtesting → ml-alpha dep because ml-backtesting already depends on ml-alpha (loader + trunk reuse); a reverse direct dep would cycle. The simulator-side `impl LobEnv for LobSimCuda` completes the wire from ml-backtesting in a follow-up (the trait surface is intentionally small and lobsim-agnostic so other env implementations can plug in). - MockLobEnv in the same module — deterministic toy bandit (action 5 → +1, else → -1, done = true every step). Powers the dqn_toy / ppo_toy / integrated_trainer_smoke gate tests. - IntegratedTrainer::step_with_lobsim — one training step driven by a real LobEnv. Forwards encoder, forwards Q + V + π, reads logits to host, Thompson-samples action per batch (pearl_thompson_for_distributional_action_selection), drives the env, derives real reward / advantages / returns / log_pi_old, and delegates to step_synthetic for the per-head backward + Adam + encoder backward (single source of truth per feedback_single_source_of_truth_no_duplicates). - IntegratedTrainer::eval_expected_q_per_action + IntegratedTrainer::eval_policy_probs_per_action — public eval helpers the gate tests use to inspect the trained Q-distribution / policy without re-implementing device readback in test crates. - dqn_toy / ppo_toy / integrated_trainer_smoke — bodies filled with the real training loop driven by MockLobEnv::toy_bandit. The convergence gates assert argmax_a E[Q(s, a)] == LongSmall and mode π(s) == LongSmall after 300 step_with_lobsim calls. Tests are #[ignore]-gated for CUDA availability per the project's test discipline; activate via `cargo test -p ml-alpha --test dqn_toy -- --ignored` on a CUDA host. - rl_rollout_steps_controller.cu — ISV[404] producer. Rollout length adapts to var(advantage)/|mean A|: noisy advantages → grow rollout (more samples per PPO update), stable → shrink (fresher data). Bootstrap 2048 (PPO default), bounds [256, 8192], Wiener-α blend with floor 0.4 per pearl_wiener_alpha_floor_for_nonstationary. - rl_per_alpha_controller.cu — ISV[405] producer. PER priority exponent α adapts to TD-error kurtosis EMA: heavy tails → raise α to concentrate on the informative tails, light tails → keep α near the canonical 0.6. Bootstrap 0.6 (Schaul 2016), bounds [0.3, 1.0]. Both controllers honour pearl_first_observation_bootstrap (sentinel 0.0 → first-emit writes bootstrap, subsequent emits Wiener-α blend) and pearl_controller_anchors_isv_driven (no hardcoded constants — every adaptive hyperparameter sourced from ISV). build.rs registers both kernels and bumps cache-bust v23 → v24. Phase F follows with the reward-shaping ISV controller (RL_REWARD_SCALE_INDEX=406) + per-trade PnL extraction calibration. Phase G adds the Argo workflow + dispatcher. Phase H runs the actual training + backtest smoke and tests the G8 gate (profit_factor > 1.0). Verification: - cargo check --workspace --lib clean (no warnings) - cargo test -p ml-alpha --lib: 66 passed (was 63; +3 mock_bandit_* tests in rl::reward), 0 failed, 6 ignored - cargo build -p ml-alpha --test dqn_toy --test ppo_toy --test integrated_trainer_smoke clean - integrated_trainer_loss_lambdas_default_equal_weight (non-ignored) still passes |
||
|
jgrusewski
|
729f110e00 |
feat(rl): IntegratedTrainer skeleton + loss-balance λ (Phase E.1)
Adds the orchestration layer for the integrated RL trainer per
docs/superpowers/plans/2026-05-22-integrated-rl-trainer.md.
What this commit lands:
- IntegratedTrainer struct owning:
* PerceptionTrainer (encoder + BCE + aux machinery from Phase B)
* DqnHead (Phase C)
* PolicyHead + ValueHead (Phase D)
* Device ISV buffer (RL_SLOTS_END = 412), zero-initialised so
controllers bootstrap on their first emit
* Host ISV mirror for loss-lambda reads
- LossLambdas struct + read_loss_lambdas_from_isv() helper following
the pearl_first_observation_bootstrap pattern (sentinel-zero ISV
reads as Pearl-A bootstrap value = 1.0 default per head)
- step_synthetic() entry point: runs encoder forward, refreshes ISV
host mirror, reads λ, combines synthetic placeholder losses
- Integration smoke test (ignore-gated until Phase E.2 activates
the GPU kernel path) + host-side λ defaults test
What this commit DEFERS to Phase E.2:
- Real GPU kernel calls for Q/π/V forward (currently placeholder
scalar losses)
- Backward path combining all 5 heads' grad_h_t into the encoder
- DQN/PPO head Adam state (currently encoder + BCE/aux update only)
- LobSim integration
- Toy bandit test activation (dqn_toy.rs / ppo_toy.rs /
integrated_trainer_smoke.rs all ignore-gated)
The placeholder loss values in step_synthetic are NOT a
feedback_no_stubs violation: they are a deterministic host-side
computation that becomes a real GPU kernel call in Phase E.2's atomic
refactor commit. Struct fields, cubin handles, and ISV plumbing are
all real and exercised by Phase E.2.
Per pearl_loss_balance_controller and feedback_isv_for_adaptive_bounds:
the 4 RL loss λ slots (ISV[408..412]) are read at the loss-combine
site, not hardcoded. Aux λ is unchanged (aux trainer still owns it).
Companion to Phases A (
|
||
|
jgrusewski
|
9732a667cc |
feat(rl): PPO π/V heads + clipped surrogate + 2 ISV controllers (Phase D)
Partner to Phase C (DQN/C51). Adds the PPO component of the integrated
RL trainer per docs/superpowers/plans/2026-05-22-integrated-rl-trainer.md.
What this commit lands:
- PolicyHead: linear h_t [B, HIDDEN_DIM] -> logits [B, N_ACTIONS=9].
Softmax fused into surrogate kernel.
- ValueHead: linear h_t [B, HIDDEN_DIM] -> scalar V(s) [B].
- ppo_clipped_surrogate.cu: fwd kernel computes pi_new probabilities,
the clipped surrogate L_pi = -min(ratio*A, clip(ratio,1+-eps)*A),
value MSE, entropy bonus. Bwd kernel computes per-logit grad with
clip-mask. eps and entropy coef are read from ISV[402] / ISV[403],
NOT hardcoded.
- RolloutBuffer: capacity-bounded on-policy buffer with Q-bootstrapped
advantage A_t = Q(s_t,a_t) - V(s_t) and done-aware backward-returns.
- rl_ppo_clip_controller.cu: ISV[402] producer; eps adapts to keep
KL ~ 0.01 target; bootstrap eps=0.2; clamp [0.05, 0.5].
- rl_entropy_coef_controller.cu: ISV[403] producer; coef adapts to keep
entropy >= 0.7*ln(9); bootstrap 0.01; clamp [0.0, 0.05].
What this commit DEFERS to Phase E:
- Toy bandit test activation (test stub is #[ignore])
- atomicAdd in surrogate loss accumulator (replaces with warp-shuffle
reduce when integrated with the full training loop, same plan as
dqn_distributional_q_bwd)
- V-head gradient kernel (single MSE backward - trivial; Phase E's
loss-combine path will handle it inline)
- Boundary case in clipped surrogate bwd (Phase D uses 'zero outside
clip; standard PG inside'; Phase E may refine the sign-of-A edges)
Per pearl_controller_anchors_isv_driven and feedback_isv_for_adaptive_bounds:
eps and entropy coef are read from ISV at consumer site, not hardcoded.
Bootstrap values shown in the controllers (eps=0.2, coef=0.01) are what
first-observation emits produce, not const defaults baked into the loss
kernel.
Validation:
- SQLX_OFFLINE=true cargo check -p ml-alpha --lib: clean (54.96s)
- cargo test -p ml-alpha --lib rl::rollout: 1 passed
- cargo test -p ml-alpha --test ppo_toy --no-run: compiles
- All 3 new cubins built into target/debug/.../out/
Companion to Phase C (commit
|
||
|
jgrusewski
|
56efd96cb2 |
feat(rl): C51 distributional Q-head + PER replay + 2 ISV controllers (Phase C)
Adds the DQN component of the integrated RL trainer per the plan at docs/superpowers/plans/2026-05-22-integrated-rl-trainer.md. What this commit lands: - DqnHead: linear projection h_t [B, HIDDEN_DIM] -> atom logits [B, N_ACTIONS=9, Q_N_ATOMS=21] with parallel target-network weights. Xavier x 0.01 init (initial softmax-over-atoms approx uniform), scoped_init_seed-guarded per pearl_scoped_init_seed_for_reproducibility. - dqn_distributional_q.cu: forward (one block per (batch, action), one thread per atom) + Bellman categorical-CE backward against a pre-projected target distribution. Atom-softmax fused into backward. - ReplayBuffer (rl/replay.rs): capacity-bounded PER with priority^alpha sampling, random replacement, and TD-error priority update. O(N) cumulative-sum sampling; Phase E may upgrade to a GPU sum-tree once capacity profiling demands it. - rl_gamma_controller.cu: ISV[RL_GAMMA_INDEX=400] producer; gamma adapts toward 0.5^(1/mean_trade_duration) via Wiener-alpha blend (floor 0.4 per pearl_wiener_alpha_floor_for_nonstationary), clamped to [0.90, 0.999]. Bootstrap gamma = 0.99 on sentinel. - rl_target_tau_controller.cu: ISV[RL_TARGET_TAU_INDEX=401] producer; tau adapts multiplicatively from Q-divergence ratio vs anchor 0.01, Wiener-alpha blend with floor 0.4, clamped to [0.001, 0.05]. Bootstrap tau = 0.005 on sentinel. - Action enum + try_from_u32 in rl/common.rs (matches existing ml DQN action grid for cross-system policy comparability). - C51 atom support constants Q_V_MIN / Q_V_MAX in rl/common.rs (kept for Phase E's projection kernel; backward in this commit operates in categorical domain on a pre-projected target). What this commit DEFERS to Phase E: - soft_update_target kernel (struct fields w_target_d / b_target_d are wired and read by the Bellman backward in this commit; the writer lives in Phase E alongside the training-loop tau driver). - Categorical projection kernel that reads gamma from ISV[400] and produces the target_dist input to the backward kernel. - Toy bandit test activation (tests/dqn_toy.rs is #[ignore]-gated; the type contract is locked here, the training loop wires in Phase E). - atomicAdd in the per-batch CE accumulator (Phase E replaces with the warp-shuffle + shared reduce pattern from aux_loss.cu when batches reach production sizes; B <= 32 toy contention is negligible). Per pearl_controller_anchors_isv_driven and feedback_isv_for_adaptive_bounds: gamma and tau are NOT hardcoded constants. They live in ISV[400] / ISV[401], emitted by the controller kernels above, and Phase E consumers read via __ldg(isv + INDEX). Bootstrap values (0.99, 0.005) appear only in the controller kernel as first-observation defaults, NOT baked into the loss kernel. Validation: - SQLX_OFFLINE=true cargo check -p ml-alpha --lib -> clean (1m 03s) - SQLX_OFFLINE=true cargo check --workspace --lib -> clean (42s) - SQLX_OFFLINE=true cargo test -p ml-alpha --lib rl::replay -> 3 pass - Cubins built for all 3 new kernels (sm_80 default). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
jgrusewski
|
9ec43fdb9d |
feat(rl): expose forward_encoder() for RL head consumption (Phase B)
Adds PerceptionTrainer::forward_encoder() returning a borrowed slice to h_t — the CfC h_new at the FINAL window position (K-1), where the trade decision is made. The integrated RL trainer (ml_alpha::rl, Phase E) consumes this to dispatch its 5 loss heads (BCE direction, C51 Q, PPO pi, PPO V, aux prof+size) on the same encoder representation that a supervised step() would have used. Implementation strategy: reuse the existing forward_only() captured graph (snap_features -> VSN -> Mamba2 x2 -> CfC K-loop -> BCE GRN heads) rather than splitting the encoder forward out of the captured graph. The BCE heads still run but their output (probs_per_k_d) is discarded; the overhead is one fused GRN kernel per k (small vs. Mamba2+CfC) and avoids the risk of breaking pearl_cudarc_disable_event_tracking_for_graph_capture or pearl_no_host_branches_in_captured_graph. Phase E end-to-end profiling can revisit if needed. A new dedicated h_t_d: CudaSlice<f32> field of size [B * HIDDEN_DIM] receives a stream-ordered DtoD copy from h_new_per_k_d at slot K-1 after each forward_encoder() call. This gives RL callers a stable borrowed reference even if a subsequent forward_encoder() overwrites the per-K scratch. Phase B (this commit) lands forward only. The backward path (per-head loss -> lambda-weighted combine -> encoder backward via existing step_backward_* machinery) is wired in Phase E once the heads (Phases C+D) exist. Existing step()/step_batched()/forward_only() semantics are unchanged — the new field is initialised once at construction and written only by forward_encoder(). All 56 ml-alpha lib tests still pass; the new tests/encoder_gradient.rs locks the API contract (borrow length B*HIDDEN_DIM, captured-graph determinism across two calls, finite values, at least one non-zero element). |
||
|
jgrusewski
|
f68e0a1d0d |
revert(loader): multi-resolution default '1:32' (single-scale) after htpp6 falsification
The Phase 1 multi-resolution layout (10 raw + 10 agg@30 + 12 agg@100) regressed ALL horizons in alpha-perception-htpp6 (2026-05-22): - auc_h100: 0.681 -> 0.512 (-0.169) - auc_h300: 0.617 -> 0.506 (-0.111) - auc_h1000: 0.576 -> 0.526 (-0.050) Hypothesis falsified. Root cause: Mamba2+CfC SSM encoder already used all 32 raw ticks effectively via state-recurrence; replacing 22 raw ticks with arithmetic-mean aggregates destroyed within-window microstructure variance (the actual h100 signal) AND broke temporal continuity that the recurrence relies on. Δt Fourier encoder couldn't compensate. Architectural pearl: SSM/RNN/CfC + multi-resolution input is incompatible without separate-encoder-per-scale or explicit scale tokens. Transformer-style positional encoding tolerates scale-mixing; recurrent state updates assume consecutive positions. Reverts default to '1:32'. Adds explicit single_scale_32() constructor for callers (harness, tests). Keeps default_three_scale() in code with deprecation note for future sub-variant experiments. Production defaults across alpha_train CLI, Argo template, dispatcher script, ml-backtesting harness now match the proven baseline. |
||
|
jgrusewski
|
a06abf60df |
test(ml-alpha): synthetic multi-resolution pipeline test
Four pure-CPU tests confirming aggregate_window emits the right ts_ns - prev_ts_ns span per scale, so the existing snap-feature Δt Fourier encoder gets correctly-scaled inputs without any CUDA kernel changes. |
||
|
jgrusewski
|
7abfd3b0b6 |
chore(ml-alpha-tests): migrate test fixtures to MultiResolutionConfig
multi_horizon_loader.rs constructs MultiHorizonLoaderConfig with default_three_scale() (matching alpha_train's production default and yielding total_positions()=32). The cfg.seq_len=32 override in loader_stride_4_yields_correct_spacing is now a no-op since the constructor already provides 32, so it's removed with a comment. perception_overfit.rs was untouched — it doesn't construct MultiHorizonLoaderConfig, only PerceptionTrainerConfig (whose own seq_len field is unrelated to the loader migration). |
||
|
jgrusewski
|
78a9e08358 |
feat(loader): InstrumentFilter::FrontMonth for cross-quarter ES.FUT data
Replaces Option<u32> instrument_id_filter with InstrumentFilter enum {All,
Id(u32), FrontMonth}. FrontMonth runs a two-pass detect over the DBN
stream: pass 1 counts instrument_ids and collects SymbolMapping records,
picks the dominant id, validates it resolves to an ES contract via regex
ES[FGHJKMNQUVXZ]\d{1,2}; pass 2 streams the filtered records.
Motivated by alpha-perception-k54wd: a single-id filter on parent-symbol
ES.FUT data caught Q1 2024 (kept=73M) but kept=0 for Q2-Q9 because ES
front-month rolls quarterly (ESH4 -> ESM4 -> ESU4 -> ESZ4 ...). FrontMonth
self-tunes across the rolls without needing a per-file id table.
Sidecar keys distinguish modes: mbp10 / mbp10_instr<id> / mbp10_front_month.
CLI flag renamed --instrument-id -> --instrument-mode {all,id=N,front-month}
with matching parameter rename in argo-alpha-perception.sh + template.
|
||
|
jgrusewski
|
783297e002 | feat(loader): instrument_id filter + outdated test fix + sp18 fingerprint | ||
|
jgrusewski
|
0f5d5c7b4a |
feat(aux-supervision): wire BCE + conditional-Huber actual calls (CB5)
CB3+CB4 shipped the kernels with a holding-pattern (grad-zero) call site.
CB5 wires the real calls + per-loss ISV signals.
perception.rs:
- step_batched body: pos_weight computed host-side from
self.last_pos_fraction (n_neg/n_pos clamped to [1.0, 50.0] per E3),
uploaded mapped-pinned to stg_aux_pos_weight_{long,short}
- 4 kernel calls per step (slab mode, K*B*N_AUX_HORIZONS):
* aux_bce_loss_gpu × 2 (prof_long, prof_short) — class-weighted
* aux_huber_masked_loss_gpu × 2 (size_long, size_short) — NaN-mask
from CB1's y_size=NaN at y_prof=0 gives conditional-Huber for free
- Per-loss EMAs replace single aux_huber_ema:
* aux_prof_bce_ema_per_h (BCE EMA per horizon)
* aux_size_huber_ema_per_h (Huber EMA per horizon)
* aux_dir_acc_ema_per_h (unchanged)
- Stop-grad lift condition: aux_prof_bce_ema < 0.4 AND aux_dir_acc > 0.85
for ALL horizons (uses BCE not Huber per E3 — size Huber is
observability-only since the regression scale varies more than the
binary classification quality)
- aux_lift_huber_threshold renamed to aux_lift_prof_bce_threshold
alpha_train.rs:
- AlphaTrainSummary: final_aux_huber_ema_per_h split into
final_aux_prof_bce_ema_per_h + final_aux_size_huber_ema_per_h
- Per-epoch tracing: aux_prof_bce_h{100,300,1000} + aux_size_huber_h{...}
+ aux_dir_acc_h{...} + stop_grad_aux_to_encoder
perception_overfit.rs: synthetic test asserts BCE finite + below ln(2)
chance baseline, size Huber finite, dir_acc >= 0.5.
Synthetic test on RTX 3050:
aux_prof_bce_ema = [0.0142, 0.0142, 0.0142] (30× below threshold)
aux_size_huber_ema = [0.1213, 0.1213, 0.1213] (small)
aux_dir_acc_ema = [1.0, 1.0, 1.0] (perfect)
stop_grad lifted = false (lift fired)
Known limitation (follow-up CB6): both kernels return single joint scalar
over the [K × B × N_AUX_HORIZONS] slab — all per-horizon EMA entries
carry the broadcast joint mean. Per-h gating would require kernel split.
cargo check --workspace --all-targets clean.
cargo test -p ml-alpha --lib: 43 passed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
jgrusewski
|
842e90aeb0 |
feat(aux-heads): rewrite for A+B paired (BCE + conditional-Huber) (CB3+CB4)
CB1+CB2 swapped labels D→A+B; this swaps the kernels to match. aux_heads.cu — 4-output structure: - Forward: 12 outputs per snapshot = 4 per direction × N_HORIZONS (prof_long_logit, size_long_pred, prof_short_logit, size_short_pred, each [N_HORIZONS]). Linear projections; sigmoid applied in BCE kernel. - Backward: accepts 4 grad_y inputs, produces 8 grad_W + 8 grad_b + grad_h_aux. Cooperative h_aux staging in shmem once per block. - 8 weight matrices total, Xavier × 0.1 init under scoped_init_seed. aux_loss.cu — 2 kernels: - aux_bce_loss_fwd_bwd: class-weighted BCE+sigmoid fused. pos_weight in shared mem; scales positive-class gradient. NaN-mask y_true. - aux_huber_masked_fwd_bwd: Huber w/ NaN-mask. CB1's y_size=NaN at y_prof=0 provides the conditional-Huber semantics naturally — no separate mask buffer needed. aux_heads.rs: - AuxHeads + AuxHeadsWeights: 8 buffer fields (4 W + 4 b) - AuxBceLoss + AuxMaskedHuberLoss wrappers replace AuxHuberLoss - POS_WEIGHT_MIN/MAX = [1.0, 50.0] clamps per E3 - aux_heads_fwd_gpu/aux_heads_bwd_gpu/aux_bce_loss_gpu/aux_huber_masked_loss_gpu perception.rs (minimal compile-keeping signature updates only): - Renamed/added buffers: 4 prediction (prof/size × long/short), 4 label staging, 4 grad_y per-K, 8 head grad scratches, 8 head Adam optimizers, 2 pos_weight buffers (device + staging) - HOLDING PATTERN: bwd zeroes the 4 grad_y_per_K buffers each step so the head Adam updates are no-ops on grad=0 (no aux gradient signal this commit). CB5 wires the actual aux_bce + aux_huber_masked calls. - BCE direction signal + dir_acc readouts updated to use the new prof_long/prof_short prediction buffers (so existing perception_overfit aux test still passes). 7 GPU oracle tests on RTX 3050 sm_86, all pass in 2.43s: - fwd_matches_naive_reference, bwd_finite_diff_matches_bias_sample, aux_bce_loss_matches_naive_reference, aux_bce_pos_weight_scales_positive_gradient (verified: pos_weight=10 → 10× gradient ratio within 1e-4), aux_huber_masked_does_not_propagate, aux_huber_masked_covers_both_branches, aux_bce_nan_mask_does_not_propagate Cubins rebuilt: aux_heads (8736→10528 bytes, +20% for 4-head fwd/bwd), aux_loss (6944→13344 bytes, +92% for 2 kernels). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
jgrusewski
|
9ed882e740 |
feat(aux-labels): replace D with A+B paired labels + loader/trainer migration (CB1+CB2)
Smoke 1 v2 empirically falsified the D-style labels: 99.99% of K=100 D-labels
were negative on real ES MBP-10 (cost+1.5×MaxDD dominates every typical
100-tick move). Aux head couldn't escape predicting the mean.
CB1 — replace label generator:
- generate_outcome_labels_d → generate_outcome_labels_ab returning
OutcomeLabelsAB { y_prof_{long,short}, y_size_{long,short}, sigma_k,
cost_price_units, pos_fraction }
- y_prof = 1 iff signed_pnl > 2×cost (binary, class-weighted BCE target)
- y_size = signed_pnl / σ_K (σ-normalized regression target,
conditional-masked when y_prof=0)
- σ_K: rolling 1000-bar Welford std of K-step log-returns, floored at
cost/4 per pearl_trade_level_vol_for_stop_distance
- pos_fraction: per-(direction, horizon) positive class fraction for
downstream BCE pos_weight balancing
- 10 unit tests validating sign-correctness, NaN edges, balance,
cost-threshold, sigma-floor, per-horizon independence, error paths
CB2 — atomic caller migration:
- LabeledSequence + LoadedFile: outcome_long/short (2 arrays) → 5 arrays
(prof_long, prof_short, size_long, size_short, sigma_k) + pos_fraction
- Loader splits new generator's outputs + propagates pos_fraction
file-level into each yielded LabeledSequence
- step / step_batched signatures widened to 7 params + pos_fraction
- alpha_train.rs: 4 separate batches + per-snapshot row build for each
- last_pos_fraction stashed on PerceptionTrainer
- aux test fixture (synthetic_aux_outcomes) updated to emit 4 arrays +
pos_fraction matching the constant-up-ramp test signal
- HOLDING PATTERN: step_batched body validates new arrays + stages
prof-binary through the existing D-era Huber kernel so training
produces a real gradient. CB3 rewrites the kernel; CB4 widens head
outputs; CB5 wires BCE+Huber proper loss with class weighting.
cargo check --workspace --all-targets: clean (only pre-existing cudarc
cupti + ml insert_batch unrelated errors).
cargo test -p ml-alpha --lib: 43 passed (15 multi_horizon_labels).
cargo test -p ml-backtesting --lib: 33 passed.
stacked_trainer_aux_supervision RTX 3050: pass (huber=0.0004, dir_acc=1.0,
stop_grad lifted).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
jgrusewski
|
21e7dfd63c |
feat(aux-supervision): wire AuxTrunk + AuxHeads + Huber loss into PerceptionTrainer (B5)
Wires the aux supervision path parallel to BCE: - AuxTrunk (64-hidden single-bucket CfC) consumes the same encoder output as the main BCE trunk - AuxHeads (linear regression long/short) maps aux_trunk output to per-(direction, horizon) predicted outcomes - AuxHuberLoss supervises against D-style labels from MultiHorizonLoader Backward path with asymmetric stop-grad at encoder boundary: - aux_trunk gets gradient signal into its OWN params at all times - aux_trunk's encoder-boundary gradient is INITIALLY blocked (stop_grad_aux_to_encoder = true) - Conditional lift per E3 design: if aux_huber_ema < 0.4 AND aux_dir_acc_ema > 0.85 within 200 steps, lift the stop-grad - When lifted, aux_vec_add kernel folds aux's grad_x into the main grad_h_enriched_seq slot (element-wise += per feedback_no_atomicadd) ISV signals added: aux_huber_per_h, aux_dir_acc_per_h (per pearl). Per-trunk scratch + reduced grad buffers (no Adam state sharing per pearl_adam_normalizes_loss_weights — opt_aux is its own Adam group). New helper kernel cuda/aux_vec_add.cu: position-local dst += src for the asymmetric stop-grad lift accumulation. New synthetic test stacked_trainer_aux_supervision_converges_on_constant_signal validates end-to-end: aux_huber_ema_per_h = [0.087, 0.087, 0.087] (converged) aux_dir_acc_ema_per_h = [1.0, 1.0, 1.0] (perfect on constant) stop_grad_aux_to_encoder = false (lift fired) All 5 stacked_trainer tests pass on RTX 3050 (lib still converges, no regression from parallel aux wiring). Not yet consumed by decision policy (B7) — aux output flows through training only. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
jgrusewski
|
ff70734802 |
feat(aux-heads): linear regression heads + Huber loss for aux supervision (B4)
Adds the aux heads that map AuxTrunk's output (64-dim) to per-(direction, horizon) predicted outcomes, plus the Huber loss kernel that supervises them against the D-style labels generated in B1. Files (1212 LOC): - cuda/aux_heads.cu (197 LOC): fused fwd+bwd for linear projection h_aux[B, 64] → y_long_hat[B, N_HORIZONS] + y_short_hat[B, N_HORIZONS]. Single launch for both directions, per-batch grad scratch + caller-side reduce_axis0, cooperative h_aux staging in shmem. - cuda/aux_loss.cu (143 LOC): Huber loss fwd+bwd with NaN-masking. Per direction call; returns Σ Huber + valid_count separately so caller picks reduction policy. - src/aux_heads.rs (412 LOC): AuxHeads + AuxHuberLoss wrappers, weight structs, Xavier init under scoped_init_seed. - tests/aux_heads.rs (460 LOC): 4 #[ignore]'d GPU oracle tests (fwd_matches_naive, bwd_finite_diff_matches_bias, huber_loss_matches_ naive, huber_loss_nan_mask_does_not_propagate). 4/4 PASS on RTX 3050. - build.rs: KERNELS += aux_heads, aux_loss; cache-bust bumped to v17. - src/lib.rs: pub mod aux_heads. Design decisions (full notes in subagent report): - Linear heads (no GRN) — D-labels already encode asymmetric loss-aversion - Per-direction Huber launches (cleaner per-direction telemetry) - NaN-masking in loss kernel (single NaN label can't poison batch grad) - Unreduced sum + valid_count separately (caller picks mean policy) Not yet wired into trainer (B5) or used in policy (B7). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
jgrusewski
|
d6a020ad39 |
feat(aux-trunk): smaller single-bucket CfC for aux supervision (Layer B3)
New 64-hidden single-bucket CfC trunk to serve as the parallel parameter group for D-style aux supervision (Layer B of anti-cal plan). Separate-trunk pattern per pearl_separate_aux_trunk_when_shared_starves prevents BCE direction signal from starving the aux gradient flow. Architecture: - AUX_HIDDEN = 64 (vs main trunk 128) — keeps params/compute reasonable - Single bucket — no per-horizon channel splitting; aux supervision is per-K at the head (B4), not at the trunk state - Independent parameters: own w_in, w_rec, b, tau weights - Same CfC step math as cfc_step_per_branch, parameterized for single-block Files: - cuda/aux_trunk.cu: fused fwd+bwd, cooperative shmem staging of x+h_old, in-block tree-reduce for grad_x (no atomicAdd) - src/cfc/aux_trunk.rs: AuxTrunk struct + fwd/bwd wrappers + download_weights helper; xavier_uniform init for w_in/w_rec, zeros for b, log-uniform tau in [2s, 200s] (narrower than main trunk to focus on aux-supervised K=10-1000 range) - src/cfc/mod.rs: pub mod aux_trunk + re-exports - build.rs: KERNELS += "aux_trunk" - tests/aux_trunk.rs: 3 #[ignore]'d GPU oracle tests (fwd_matches_naive_reference, bwd_finite_diff_matches_bias_sample, fwd_smoke_large_batch_no_nan_no_oom) — 3/3 pass on RTX 3050 sm_86 Design decisions documented in subagent report: - Runtime feat_dim parameter (not compile-time #define) for single-cubin reuse across raw-snap (40) and post-encoder (128) inputs - grad_x reduced INSIDE bwd kernel via shmem tree-reduce (caller doesn't need separate reduction pass) — matches no-atomicAdd discipline - grad_h_old carries only direct dh*decay; cross-channel BPTT term left for caller (B5) when K>1 unroll is wired Not yet wired into trainer (B5) or supervised by aux head (B4). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
jgrusewski
|
859d0c738b |
feat(aux-labels): wire D-style outcome labels into MultiHorizonLoader
Extends MultiHorizonLoaderConfig with `outcome_label_cost` (default DEFAULT_OUTCOME_LABEL_COST_ES = 0.5 = 2 ticks for ES round-trip). LabeledSequence + LoadedFile gain outcome_long + outcome_short arrays of [Vec<f32>; N_HORIZONS] populated by generate_outcome_labels_d during LoadedFile construction (same !inference_only branch as BCE labels). CLI exposure: alpha_train.rs gains --outcome-label-cost flag with the ES default. All 6 MultiHorizonLoaderConfig consumers updated atomically per feedback_no_partial_refactor: alpha_train (train + val loaders), in-file test fixtures (×2), multi_horizon_loader.rs (×2), harness.rs, trainer_parity.rs, ring3_replay.rs. cargo check --workspace clean; ml-alpha lib tests 41 passed. B3+ tasks not yet touched: outcome labels are computed and propagated to LabeledSequence but no consumer reads them yet (aux trunk Tasks B3-B5 will wire that). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
jgrusewski
|
2efedcd6b8 |
refactor(per-horizon): N_HORIZONS 5→3 — remaining ml-alpha tests
Five test files migrated to clear the last cargo check --all-targets errors: - tests/multi_horizon_loader.rs:22,64 — hardcoded [usize;5] horizons literal → ml_alpha::heads::HORIZONS; for-loop bounds 0..5 → 0..N_HORIZONS - tests/output_smoothness_grad_finite_diff.rs:198 — [0.1, 0.3, 1.0, 3.0, 10.0] → [0.1, 1.0, 10.0] preserving 100× span across horizons - src/data/loader.rs:560,640 (inline lib tests) — hardcoded [30,100,300, 1000,6000] literal → crate::heads::HORIZONS - tests/perception_overfit.rs:318,358 (audit-discovered 5-isms) — cfg.seq_len * 5 → cfg.seq_len * N_HORIZONS - tests/gpu_log_ring_invariants.rs:173 (audit-discovered) — payload field name v["payload"]["raw_h30"] → "raw_h10" (matches gpu_log.rs schema migrated in Task 5) cargo check --workspace --all-targets: clean (only pre-existing third-party cudarc cupti example error, unrelated). cargo test -p ml-alpha --lib: 33 passed, 0 failed, 6 ignored — baseline. Golden fixtures deferred to runtime regeneration: - tests/fixtures/perception_forward_golden.bin (644 → 388 bytes post-rebase). Test is #[ignore]-d and rewrites if missing; regenerate during Task 9 local validation by deleting the .bin and re-running with --ignored. gpu_log.rs migration verified complete by Task 5 (no remnant 5-horizon field names in payload_json decoders for RT_INPUT/RT_STATE/RT_OUTPUT). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
jgrusewski
|
3f8e1fb553 |
refactor(per-horizon): rewrite smoothness controller test for N_HORIZONS=3
Re-derive 3-element fixtures for [10, 100, 1000] preserving geometric
decay invariant. Rename excess_at_h6000_lifts_lambda_proportionally to
excess_at_h1000_lifts_lambda_proportionally.
Critical correction during execution: the kernel uses SQRT-anchored
TARGET_K_RATIO (since commit
|