Commit Graph

3263 Commits

Author SHA1 Message Date
jgrusewski
0fe346736d fix(rl): remove CudaSlice::clone() from graph capture + delete dead methods + fused kernel
- Inline launch_l2_norm, launch_ema_update_per_step, launch_kurtosis,
  launch_var_over_abs_mean at all call sites to eliminate .clone() that
  allocated device memory inside CUDA graph capture regions (caused
  CUDA_ERROR_STREAM_CAPTURE_UNSUPPORTED at b>16).
- Delete the now-unused helper methods (no hiding, no suppressing).
- Add pre-commit hook guard for *_d.clone() patterns.
- Add rl_fused_reward_pipeline.cu (7→1 per-batch kernel, not yet wired).
- Add rl_write_u64 kernel + ts_ns device-resident for Graph B.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 23:28:01 +02:00
jgrusewski
45ab5dd066 fix(rl): remove CudaSlice::clone() from graph capture regions
CudaSlice::clone() allocates device memory, which is forbidden during
CUDA stream capture. Inline the launch_l2_norm and
launch_ema_update_per_step calls to avoid the &mut self borrow conflict
that forced the .clone() workaround.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 23:24:47 +02:00
jgrusewski
02987f0111 feat(rl): CUDA Graph C capture — replay training step (~25 kernels)
Three-state machine wraps step_synthetic: Q/π/V forwards, Bellman
target projection, C51/PPO losses, backward, reduce_axis0, Adam
updates, FRD backward chain, encoder grad combine + backward, EMA
diagnostic updates, target soft-update. Stream sync + loss readback
(5 mapped-pinned reads) deferred to shared path outside the captured
region.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 23:13:31 +02:00
jgrusewski
3af2b40100 feat(rl): CUDA Graph B capture — post-fill reward/EMA/controller pipeline (~20 kernels)
Three-state machine (warmup → capture → replay) wraps the post-fill
section: extract_realized_pnl_delta, unit_state_update, trade_context,
multires, step_counter, 3× EMA, abs_copy, reward_shaping, raw_rewards
snapshot, apply_reward_scale, reward_clamp, atom_support_update,
compute_advantage_return, var_over_abs_mean.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 23:05:59 +02:00
jgrusewski
dc7f4c6659 perf(rl): scale default batch size to 128 + auto-size PER capacity
n-backtests default in Argo template: 16 → 128.
PER capacity: max(configured, 4 × b_size) so replay buffer is always
large enough for useful prioritized sampling at any batch size.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 23:02:29 +02:00
jgrusewski
4bed8f2dbf refactor(rl): pre-allocate 56 replay-step gradient buffers for Graph C
Move all step_synthetic/dqn_replay_step alloc_zeros to persistent
trainer fields (ss_* prefix). Enables CUDA Graph capture of the replay
training step — all device pointers are now stable across steps.

Introduces reduce_axis0_free() to resolve borrow-checker E0502 when
both source (per-batch scratch) and destination (reduced grad) are
self fields passed to the same function.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 23:00:40 +02:00
jgrusewski
1434ecc421 test(rl): GPU oracle tests for IQN + NoisyNet + PRNG advance fix
Six GPU-oracle tests validating Phase 1 device-side PRNG kernels and
the IQN/NoisyNet heads (mapped-pinned data transfer throughout):

IQN: tau U(0,1) range + PRNG advance, forward finite, expected_q mean
NoisyNet: factored transform invariant, zero-noise = mu, resample δ

Fix: PRNG advance used prng_state[b] (zero on first call) instead of
the self-seeded `seed` — xorshift32(0)=0 made second call identical.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 22:37:40 +02:00
jgrusewski
2c9aefe19e fix(rl): graph capture state machine — gate end_capture on begin_capture
The warmup step set graph_warmup_done=true but never called
begin_capture; the end_capture check then fired prematurely, causing
CUDA_ERROR_ILLEGAL_STATE. Fixed by tracking a local `capturing_prefill`
/ `capturing_postfill` flag that's true only when begin_capture ran.

Also adds Graph A2 (post-snapshot) capture for 7 kernels:
session_risk → min_hold → trail_decay → trail_mutate →
trail_stop → position_heat → actions_to_market_targets.

Removes unused buf_len variable.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 22:24:53 +02:00
jgrusewski
4c94366f63 feat(rl): CUDA Graph capture for pre-snapshot kernel pipeline (Graph A)
Three-state machine (warmup → capture → replay) wraps the 20 pre-
snapshot kernels in step_with_lobsim: FRD/DQN/IQN/V/π forwards,
ensemble action-value, noisy exploration, Q→π agreement, π-driven
action selection, argmax Bellman, log_pi, confidence gate, FRD gate.

On first step, kernels dispatch eagerly (warmup). On second step,
stream capture records the kernel topology. On third+ steps, a single
cudaGraphLaunch replays all 20 kernels — ~200μs launch overhead → ~10μs.

Host-side step_counter increment moved outside the capture region
(host ops are invisible to graph capture).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 22:16:53 +02:00
jgrusewski
0b7895a4fb refactor(rl): pre-allocate per-step head output buffers for graph capture
Move q_logits_d, q_logits_tp1_d, v_pred_d, v_pred_tp1_d, pi_logits_d
from per-step alloc_zeros to persistent trainer fields allocated at
init. Stable device pointers are a prerequisite for CUDA Graph capture.

Updated step_with_lobsim, step_synthetic, and dqn_replay_step to use
self.field instead of local allocations.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 22:11:56 +02:00
jgrusewski
a548adf7b7 feat(rl): device-side PRNG for IQN tau + NoisyNet noise (graph prereq)
Move tau sampling and factored noise generation from host-side ChaCha8
RNG + mapped-pinned upload to device-side xorshift32 kernels. This
eliminates all host-side RNG from the step pipeline, unblocking CUDA
Graph capture for Graphs A and C.

New kernels:
- rl_sample_tau: per-batch xorshift32 generates tau [B, N_TAU] ~ U(0,1)
- rl_sample_noise: factored noise f(x)=sign(x)√|x| for NoisyLinear

Both kernels self-seed from alloc_zeros on first call (zero memcpy).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 21:56:16 +02:00
jgrusewski
a7c1d763d8 perf(rl): device-resident step counter + fused controllers (-9 launches)
Two CUDA Graph prerequisites implemented:

1. Device-resident step counter (ISV[548]):
   - New rl_increment_step.cu kernel (single thread, ISV += 1)
   - All kernels that took current_step as scalar now read from ISV
   - Updated: confidence_gate, frd_gate, unit_state_update,
     trade_context_update, gate_threshold_controller
   - Enables CUDA Graph capture (no scalar arg changes between replays)

2. Fused controllers (rl_fused_controllers.cu):
   - Combines 10 single-thread controllers into 1 kernel launch:
     gamma, tau, ppo_clip, entropy, rollout_steps, per_alpha,
     reward_scale (with ±2% clamp), ppo_ratio_clamp,
     gate_threshold, q_distill_lambda
   - Saves 9 kernel launches per step (~40-80μs)
   - Individual .cu files retained for testing/documentation

ISV slot 548 (step counter). Local smoke: 100 steps, no crash.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 21:35:50 +02:00
jgrusewski
993201250a feat(rl): wire noisy nets into ensemble Q — learned exploration, P_MIN=0
NoisyLinear layer on top of ensemble Q values provides state-dependent
exploration noise via h_t → learned perturbation. Replaces the fixed
P_MIN=0.015 probability floor with learned exploration (P_MIN→0.0).

- NoisyLinear constructed with in=128, out=11, σ_init=0.5
- resample_noise() each step for fresh factored noise
- forward(h_t) → noise_d added to ensemble_q_d via aux_vec_add
- 4 Adam optimizers for mu_w, sigma_w, mu_b, sigma_b
- ISV seeds for IQN ensemble α=0.5, n_tau=32, σ_init=0.5
- Target network path stays noise-free (Fortunato et al. 2017)

Local smoke: 100 steps, l_q=2.30, no crash. All three phases complete.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 21:19:55 +02:00
jgrusewski
b78d25e4e7 feat(rl): Phase 3 noisy linear layers — state-dependent exploration
Factored noisy nets (Fortunato et al. 2017) for state-dependent
exploration. Replaces brute-force P_MIN probability floor.

- rl_noisy_linear_forward.cu: y = (μ_w + σ_w ⊙ ε_w)x + (μ_b + σ_b ⊙ ε_b)
  Factored noise: ε_w[i,j] = f(ε_i)×f(ε_j), f(x) = sign(x)√|x|
  Only p+q noise samples needed (not p×q). Per-thread, no atomicAdd.
- rl_noisy_linear_backward.cu: per-batch scratch gradients for all 4
  weight tensors. Caller reduces across batches.
- rl/noisy.rs: NoisyLinear struct with Fortunato init (σ = 0.5/√in),
  resample_noise() via mapped-pinned host→device, forward/backward.

ISV slot 547 (RL_NOISY_SIGMA_INIT_INDEX, default 0.5).

Not yet wired into DQN/IQN/policy heads — module compiles and is
ready for integration. Will replace P_MIN floor once wired.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 21:09:41 +02:00
jgrusewski
b219681d01 feat(rl): IQN loss + backward + ensemble action selection
Completes the IQN integration:

- rl_iqn_backward.cu: backprop through quantile embedding + output
  projection to produce per-batch weight gradients for all 4 weight
  tensors. Block per batch, HIDDEN_DIM threads.
- rl_iqn_loss.cu (already existed): quantile Huber loss feeds
  grad_online_q into the backward kernel.
- rl_ensemble_action_value.cu: E_ensemble = α×C51 + (1-α)×IQN,
  ISV-driven α at slot 544.
- Adam updates for IQN weights after backward.
- Ensemble Q feeds rl_q_pi_agree_b diagnostic.

Full stack smoke: 200 steps, l_q=2.43, no crash. Both C51 and IQN
learning simultaneously from the same replay transitions.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 21:03:04 +02:00
jgrusewski
65d92644d6 fix(rl): asymmetric trail win threshold 100% → 25% of initial_r
The winner-ratchet condition (unrealized > initial_r) never fired
because typical price moves rarely exceed 100% of the initial stop
distance. Win/loss ratio measured at 1.01× (symmetric) despite the
asymmetric decay being active.

Fix: ISV-driven threshold factor (slot 546, default 0.25). Trail
starts ratcheting when profit reaches 25% of initial_r — much more
achievable. At 25%: a $12.50 initial_r only needs $3.12 profit
before the winner-tracking activates.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 20:59:44 +02:00
jgrusewski
450eaab7f1 feat(rl): wire IQN into trainer — forward alongside C51 each step
IQN head now runs in parallel with C51 on every step:
- Constructs IqnHead in IntegratedTrainer::new()
- Per-step: samples tau ~ U(0,1), forwards h_t through IQN,
  computes expected Q via tau-mean reduction
- Per-replay: forwards sampled_h_t through IQN (verifying
  the head runs on replay data)
- Adam optimizers allocated for 4 IQN weight tensors

Not yet wired: IQN loss/backward, target soft update, ensemble
action selection. IQN runs forward-only — its Q values are
computed but not yet consumed for action decisions or gradient.

Local smoke: 100 steps, no crash, l_q=2.30 with both heads active.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 20:46:14 +02:00
jgrusewski
9c0be855dd feat(rl): IQN complementary Q-head — quantile embedding + Huber loss
Phase 2 of the Q-learning improvements spec. Adds an Implicit
Quantile Network head alongside the existing C51 head:

- rl_iqn_forward.cu: quantile embedding φ(τ) = ReLU(W × cos(iπτ)),
  element-wise h_t ⊙ φ(τ), action-value projection. Plus
  rl_iqn_expected_q for tau-mean reduction.
- rl_iqn_loss.cu: quantile Huber loss ρ_τ(δ) = |τ-1(δ<0)| × Huber(δ).
  Block tree-reduce per batch (no atomicAdd).
- rl/iqn.rs: IqnHead struct with online + target weights, Xavier init,
  forward/forward_target/expected_q/compute_loss methods.

ISV slots: 543 N_TAU (32), 544 ensemble_alpha (0.5), 545 LR (1e-3).

Not yet wired into the trainer — head is constructible and kernels
compile. Ensemble integration is the next step.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 20:35:06 +02:00
jgrusewski
db2ad15f3d feat(rl): n-step returns (n=10) — 10× more direct reward signal
Replaces 1-step Bellman r + γQ(s') with n-step R_n + γⁿQ(s_{t+n}).
At γ=0.995, n=10 gives the agent 10 real rewards (~2.5 seconds)
before bootstrapping from Q, dramatically reducing bootstrap error.

Implementation:
- NStepEntry ring buffer per batch in IntegratedTrainer
- push_to_replay accumulates entries, flushes when len==n or done
- R_n = Σ γᵏ rₖ computed at flush time (both scaled + raw)
- n_step_gamma = γⁿ (or 0 if any done in window) stored per transition
- bellman_target_projection.cu uses per-transition n_step_gamma
  instead of the global ISV γ for the bootstrap discount
- project_bellman_target wrapper takes n_step_gammas_d buffer

ISV slot 542 (RL_N_STEP_INDEX, default 10). Local smoke: 1k steps,
no crash, replay=150 (correct for n=10 with dones).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 20:27:01 +02:00
jgrusewski
38cf5fee0b wip(rl): n-step returns foundation — struct, ISV slot, buffer field
Adds RL_N_STEP_INDEX (slot 542, default 10), n_step_gamma field to
Transition, and NStepEntry ring buffer struct to IntegratedTrainer.

Remaining: implement n-step accumulation in push_to_replay and
modify bellman_target_projection.cu to use n_step_gamma.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 20:13:26 +02:00
jgrusewski
29640b6e6d feat(rl): asymmetric trail + session risk — structural P&L edge
Two structural mechanics that produce asymmetric win/loss ratio
without the agent needing to learn the behavior:

1. Asymmetric trail decay (rl_asymmetric_trail_decay.cu):
   - LOSING: trail *= 0.995/step (halves in 139 steps = 35s)
   - WINNING beyond initial_r: trail = max(trail, profit × 0.5)
   - NEUTRAL: unchanged (proving zone)
   Produces: small/quick losses, large/extended wins.

2. Session risk circuit breaker (rl_session_risk_check.cu):
   - Tracks EMA of realized PnL (α=0.02, slow)
   - When EMA < -$50: blocks ALL opening actions
   - Prevents tilt — a losing streak stops the agent from digging deeper

Pipeline order: action selection → confidence gate → FRD gate →
session risk → min_hold → asymmetric trail → trail_mutate →
trail_stop → heat_cap → actions_to_market_targets

ISV slots 537-541. RL_SLOTS_END → 542.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 19:53:54 +02:00
jgrusewski
fcb8222a60 feat(rl): tier 2 wave-scale — γ=0.995, PER 32k, min-hold 100
Prepared for next run after the γ=0.99 1M run completes:

- γ floor 0.99 → 0.995: horizon 100 → 200 steps (50 seconds).
  Real ES directional moves (2-5 points) happen at this scale.
- PER 16384 → 32768: 2048 unique steps of replay depth. Supports
  the 200-step γ horizon with margin.
- Min-hold 50 → 100 steps (25 seconds): commit to the full wave.
  Short-hold penalty threshold matches.

Safe to push because raw-reward re-normalization eliminates scale
drift in the deeper buffer, and the ±2% scale clamp keeps targets
stable across the 2048-step replay window.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 15:26:58 +02:00
jgrusewski
b0dbb70989 feat(rl): longer horizon γ=0.99, min-hold 50, ride bonus for winners
The agent was break-even at the wrong timescale — trading in
bid-ask noise at 1-4 second holds where there IS no edge. Real
directional moves live at 10-60 seconds.

Three changes push the agent to wave-scale:

1. γ floor 0.90 → 0.99: effective horizon 10 → 100 steps (25s).
   Q now values what happens 25 seconds from now, not just the
   next tick. The FRD's 10s and 600s horizons become relevant.

2. Min-hold 20 → 50 steps (12.5s): forces commitment to waves,
   not ripples. Short-hold penalty threshold matches.

3. Long-ride bonus: at trade close, profitable rides get reward
   multiplied by (1 + bonus × sqrt(hold_time)). A 100-step
   winning ride gets 21× the raw PnL as reward. Combined with
   per-step hold bonus boosted to $2.00. "The best wave of the
   session deserves the biggest cheer."

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 15:00:21 +02:00
jgrusewski
408e0ef4ac fix(rl): per-step ±2% clamp on reward_scale movement
The reward_scale oscillated 10,873× (min 0.000123, max 1.33) because
the Wiener-α=0.4 blend tracked sparse trade PnL spikes instantly.
Old transitions in PER had stale-scale rewards even with raw-reward
re-normalization (the current scale itself was unstable).

Per-step clamp: scale can only move ±2% from previous value.
Doubling takes ~35 steps, halving takes ~35 steps — fast enough to
track regime changes, stable enough for Q targets across the replay
window. Max oscillation over 1000 steps: ~7× (was 10,873×).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 13:50:25 +02:00
jgrusewski
3517830b1b fix(rl): min-hold exempts positions at heat cap level
Min-hold check was overriding heat cap's FlatFromLong/Short to Hold,
preventing the safety exit. Position grew to 9 (above cap 8) because
the flat was blocked by min-hold, then next step added lots.

Fix: min-hold reads pos_state and ISV heat cap slot. If |position|
>= cap, the close action passes through regardless of hold time.
Safety always overrides patience.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 13:40:53 +02:00
jgrusewski
7c38f339cd feat(rl): ISV-driven minimum hold time — hard constraint on churning
Overrides closing actions (a3/a4/a9/a10) to Hold when
steps_since_done < RL_MIN_HOLD_STEPS_INDEX (slot 536, default 20).
The agent CANNOT exit before the minimum — forced to ride the wave.

Trail stops still fire regardless (safety overrides patience) —
if the market moves against the position past the trail distance,
the stop-loss exits even within the min-hold window.

Pipeline order: action selection → confidence gate → FRD gate →
min_hold_check → trail_mutate → trail_stop → heat_cap →
actions_to_market_targets

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 12:27:47 +02:00
jgrusewski
60c714c8d1 feat(rl): surfer-philosophy reward shaping — entry cost + hold bonus
Three ISV-driven reward shaping components discourage churning and
reward patience:

1. Entry cost ($15 default): subtracted when opening a new position.
   Agent must expect profit > cost to justify entry. Slightly above
   ES 1-tick spread ($12.50) so marginal trades are net-negative.

2. Short-hold penalty (0.5× for holds < 20 steps): multiplicative
   penalty at trade close for quick flips. "Don't bail on the first
   bump" — halves the reward for sub-5-second holds.

3. Hold bonus ($0.50/step × sqrt(hold_time)): per-step reward for
   staying in a profitable position. "Ride the wave" — incentivizes
   patience when the trade is working.

Runs BEFORE reward_scale so all costs/bonuses are in raw USD terms.
ISV slots 532-535. RL_SLOTS_END → 536.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 12:12:49 +02:00
jgrusewski
a1d35cd6e6 fix(rl): q_pi_agree metric → cosine similarity instead of argmax match
Binary argmax agreement is pure noise for near-uniform distributions
(both Q and π give ~9% to each of 11 actions, so argmax flips on
0.001 differences → metric reads 0 even when KL=0.02).

Cosine similarity between E_Q[a] (expected Q-value per action) and
softmax(π)[a] measures full ranking direction agreement. Robust to
near-uniform: two identical distributions → cosine=1.0 regardless
of which action is marginally highest.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 12:04:26 +02:00
jgrusewski
96abc364b5 fix(rl): asymmetric Q→π distillation λ controller — ramp fast, decay slow
Root cause of q_pi_agree collapse: the symmetric Schulman controller
decayed λ from 0.5 to 0.001 in 34 steps (÷1.2/step), killing Q→π
coupling. Once decoupled, Q and π learned independent policies,
making q_pi_agree drop to 1e-22.

Three fixes:
1. MIN_LAMBDA 0.001 → 0.05: Q pull never drops below 5% strength
2. Asymmetric rates: ramp 1.2×/step (4 steps to double), decay
   0.998×/step (347 steps to halve). Coupling establishes fast and
   persists for thousands of steps.
3. Dead zone tolerance 1.5× → 3.0×: natural KL fluctuation stays
   in-band instead of triggering constant ramp/decay cycles.

Seed λ raised to 0.1 so distillation is active from step 0.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 11:54:17 +02:00
jgrusewski
0251656be4 fix(rl): replay reward re-normalization eliminates scale drift
Stored raw_reward alongside scaled reward in Transition. At sample
time, consumer re-applies current reward_scale to raw_reward instead
of using the stale-scale stored reward. This eliminates off-policy
scale drift that caused q_pi_agree to collapse from 0.125 to 1e-22
over 40k steps with PER=32768.

PER capacity set to 16384 (1024 unique steps at b=16) — balances
replay depth against h_t representation staleness.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 11:44:32 +02:00
jgrusewski
5cb34572eb fix(rl): CLI per_capacity default 4096 → 32768 to match trainer config
The CLI default overrode the trainer config default. Without passing
--per-capacity explicitly, the Argo run used 4096 (256 unique steps
at b=16) instead of the intended 32768 (2048 steps).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 11:24:27 +02:00
jgrusewski
79ac46c672 fix(rl): C51 atom span [-1,+1] → [-0.5,+0.5], PER capacity 4096 → 32768
Root cause of Q non-convergence: at reward_scale ≈ 0.003, a typical
±$5 trade maps to ±0.015 scaled reward. With 21 atoms spanning [-1,+1]
(step=0.1), the entire reward distribution falls within a SINGLE atom.
Q cannot distinguish wins from losses — confirmed by q_pi_agree ≈ 0.

Fix 1: Tighten atom span to [-0.5, +0.5]. Now atom_step = 0.05.
A ±$5 trade at scale=0.01 spans 2 atoms; at scale=0.05 spans 10.
The reward clamp controller ratchets the span at runtime if rewards
exceed the bounds.

Fix 2: PER capacity 4096 → 32768. At b=16, old capacity retained
only 256 unique steps — Q replayed each transition 1-2× before
eviction. New capacity retains 2048 steps, giving Q proper replay
depth for convergence.

Also aligns reward clamp bounds (WIN/LOSS) with the new span.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 11:21:39 +02:00
jgrusewski
ce5df13503 fix(rl): heat cap uses >= not > (pos=8 at cap=8 must trigger)
With strict >, pos=8 passed the cap check, then a LongLarge(+2)
made pos=10 before the next step's cap fired. Using >= prevents
position from ever exceeding the cap by the fill size.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 11:00:56 +02:00
jgrusewski
cd149cdb5d fix(rl): gate controller target 0.02→0.10, dead zone 0.5×/2× → 0.2×/5×
Controller oscillated: 3 dones at step 15k spiked EMA above 2×target,
triggering thousands of steps of tightening that killed trades.
Higher target (10% vs 2%) matches the natural trade frequency at
b=16. Wider dead zone (5× above / 0.2× below) prevents single-batch
spikes from triggering tighten/relax oscillation.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 10:43:09 +02:00
jgrusewski
fa561b7676 fix(rl): FRD gate defaults 0.15 → 0.05, floor → 0.0
Confidence gate LCB fix worked (0 fires post-warmup) but FRD gate
blocked 5212 entries in 5k steps — the FRD head learned "forward
returns are negative" during warmup (losing trades), so favorable
tail mass dropped below 0.15 for all actions.

Lower default to 0.05 and floor to 0.0 so the adaptive controller
can fully disable when trades dry up.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 10:36:47 +02:00
jgrusewski
d456ad3962 fix(rl): confidence gate LCB relative to atom span, not absolute
The old formula conf = clamp((μ - λσ) / σ_norm, 0, 1) produced
conf=0 for ANY distribution with σ > μ — including uniform Q at
training start (μ=0, σ=0.577 → LCB=-0.577 → conf=0). This caused
permanent 100% block rate regardless of threshold setting.

New formula: conf = clamp((μ - V_MIN - λσ) / (V_MAX - V_MIN), 0, 1)
Shifts by V_MIN so conf measures "how far above worst case":
  Uniform: conf = 0.21 (passes threshold 0.01)
  Peaked V_MAX: conf ≈ 1.0 (high confidence)
  Peaked V_MIN: conf ≈ 0.0 (blocked — correct)

Also sets conf gate floor to 0.0 so the adaptive controller can
fully disable the gate when trades dry up.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 10:29:49 +02:00
jgrusewski
43c4bddd8e fix(rl): conf gate floor 0.001 → 0.0 so controller can fully disable
When C51 LCB is ≈0 for all actions (typical early training), even
0.001 threshold blocks everything. Floor=0.0 lets the controller
drive to zero when trades dry up, then ratchet back as Q calibrates.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 10:28:21 +02:00
jgrusewski
88abf8185c feat(rl): adaptive gate threshold controller from trade frequency
New rl_gate_threshold_controller.cu watches dones EMA and adjusts
confidence + FRD gate thresholds via Schulman bounded step:
- dones_ema < target×0.5 → relax thresholds (×0.95)
- dones_ema > target×2.0 → tighten thresholds (÷0.95)

ISV-driven: target (0.02), conf bounds (0.001-0.50), FRD bounds
(0.05-0.50), adjust rate (0.95). Runs per step after warmup.

Also lowers default thresholds: conf 0.10→0.01, FRD 0.35→0.15.
Post-warmup, the controller adapts these based on actual trade
flow instead of relying on static defaults.

ISV slots 525-531. RL_SLOTS_END → 532.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 10:26:10 +02:00
jgrusewski
82481db06d feat(rl): gate warmup — confidence + FRD gates inactive for first 10k steps
Both gates now read RL_GATE_WARMUP_STEPS_INDEX (slot 524, default
10000) and return early when current_step < warmup. During warmup,
the agent opens positions freely, collects reward signal, and
calibrates Q. After warmup, gates activate and filter low-quality
entries.

Without warmup, gates blocked 100% of opening actions from step 0
(uniform Q → zero confidence → permanent Hold attractor → zero
trades → zero reward → Q never learns). Confirmed by 50k-step
L40S run with zero dones across 800k action decisions.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 10:16:05 +02:00
jgrusewski
ac11dddc92 fix(ml-features): imbalance bar cache on persistent PVC, not /tmp
Cache path moved from /tmp/.foxhunt_imbalance_cache/ (ephemeral,
lost between Argo pods) to <mbp10_dir>/.cache/ (on the training-data
PVC). Saves ~4 minutes per Argo submission by avoiding recomputation
of 209M trade ticks → 17.8M imbalance bars.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-24 23:11:03 +02:00
jgrusewski
1f58fee924 feat(rl): wire encoder context broadcast into forward path
Completes P2 by injecting rl_encoder_context_broadcast launch inside
the perception trainer's forward_only path — runs after
snap_feature_assemble fills dims [0..40] and before VSN/Mamba2
consume the window tensor. Fills dims [40..56] with per-batch
trade_context (4) + multires (12) features.

Device pointers to the context buffers (owned by IntegratedTrainer)
are set on PerceptionTrainer before each forward_encoder call. The
broadcast kernel reads these and writes directly into window_tensor_d
at the correct column offsets for all B×K rows.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-24 22:46:18 +02:00
jgrusewski
914a6e8e72 feat(rl): encoder input expansion 40 → 56 dims (trade-context + multires)
Introduces ENCODER_INPUT_DIM = 56 (FEATURE_DIM + 16). All encoder
first-layer weight matrices (VSN gate, Mamba2 L1 input projection)
now sized for 56 input dims. The extra 16 are per-batch state:
4 trade_context + 12 multires features.

- snap_feature_assemble_batched: output stride → ENCODER_INPUT_DIM,
  zero-fills dims [40..56] for the broadcast kernel to overwrite.
- New rl_encoder_context_broadcast.cu: writes trade_context_d[B×4]
  + multires_output_d[B×12] into each of the K sequence rows per
  batch at positions [40..56].
- CfcConfig.n_in, Mamba2 L1 in_dim, VSN gate, window_tensor_d,
  all forward/backward scratch buffers updated to ENCODER_INPUT_DIM.
- CfcTrunk default config updated.

The broadcast kernel launch integration into the forward_only path
is the final wire-up step — until then dims 40-55 are zero-filled
(safe: Xavier init on new columns means encoder starts by learning
to ignore them, then gradually incorporates the signal).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-24 22:35:26 +02:00
jgrusewski
233894a4bf feat(rl): trade-context + multires features + P14 validation tests
P1: New rl_trade_context_update.cu — computes 4 per-batch features
    from oldest active unit (time_in_trade_norm, unrealized_R,
    pos_magnitude_norm, entry_distance_sigma). Output in
    trade_context_d[B×4], updated after unit_state_update each step.

P0: New rl_multires_features_update.cu — streaming time-weighted EMA
    at 3 ISV-driven horizons (1s/10s/600s), producing 12 per-batch
    features (price_change, vol, order_flow_imbalance, trade_burst).
    O(1) state per feature vs circular buffer — same time-constant
    semantics.

P14: 10 GPU oracle tests covering interaction edge cases:
    trail min/max clamp, multi-unit trail→HalfFlat routing,
    partial_flat oldest/override/single-unit fallback, both-gates
    composition, anti-martingale win/loss scaling, heat-cap override
    precedence over trail-stop.

ISV slots: 521-523 (multires horizons). RL_SLOTS_END → 524.

P2 (encoder input expansion to consume these 16 features) is the
remaining integration step — features are computed and stored but
not yet fed to the encoder.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-24 22:21:30 +02:00
jgrusewski
0e15899670 feat(rl): exhaustive diag JSONL for all trade-management mechanics
Surfaces full per-unit per-batch state in the per-step diag output:

- units: entry_price, entry_step, lots, trail_distance, active_mask,
  unit_count (all [B × MAX_UNITS] arrays)
- trail: fired/tightened/loosened counts (step + cumulative)
- pyramid: added count (step + cumulative), units_distribution,
  max_units_reached flag
- partial_flat: fired count (step + cumulative), long/short split,
  close_unit_index per batch
- confidence_gate: gated count (step + cumulative)
- frd_gate: gated count (step + cumulative)
- position_heat: capped count (step + cumulative), max_lots ISV
- anti_martingale: per-batch outcome_ema, kappa ISV

Replaces the minimal pyramid/heat_cap diag from P7. Every mechanic
is now fully observable in post-hoc analysis.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-24 22:05:24 +02:00
jgrusewski
3b23a0de5a feat(rl): per-batch outcome EMA, vol-adjusted trail, ISV-driven P_MIN
P10: New rl_recent_outcome_update.cu — per-batch signed outcome EMA
     (sign(reward) on done steps) feeds per-batch anti-martingale
     sizing in actions_to_market_targets. Replaces the single ISV
     scalar with a per-batch buffer for multi-batch granularity.

P11: Trail bootstrap switched from vwap × 1e-3 × k_init to
     k_init × MEAN_ABS_PNL_EMA (slot 423). Vol-derived trail
     distance adapts to realized trade magnitude as the EMA updates.

P12: P_MIN in rl_pi_action_kernel now ISV-driven (slot 519,
     default 0.015). At N=11, max single-action prob = 0.85
     (uplift vs prior 0.80 at hardcoded P_MIN=0.02).

ISV slots: 519 P_MIN, 520 OUTCOME_ALPHA. RL_SLOTS_END -> 521.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-24 21:59:34 +02:00
jgrusewski
e9ecacbdfa feat(rl): FRD gate — override entries when forward-return is unfavorable
New rl_frd_gate.cu kernel reads the FRD head's horizon-2 (medium,
~300 ticks) categorical distribution. For long openings, sums
probability mass in the positive tail (atoms > +0.5σ); for short
openings, sums the negative tail (atoms < -0.5σ). Overrides to Hold
when favorable mass < threshold.

Fires after confidence gate, before trail/heat/market pipeline.
Same preconditions: only gates flat positions with opening actions.

ISV slots: 516 THR_LONG (0.35), 517 THR_SHORT (0.35),
           518 fired_count (diag). RL_SLOTS_END → 519.

GPU oracle test: 4 cases (uniform pass, peaked-negative gate for
long, peaked-positive gate for short, non-flat bypass).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-24 21:43:15 +02:00
jgrusewski
e132d59a48 feat(rl): confidence gate — override low-certainty openings to Hold
New rl_confidence_gate.cu kernel computes C51 distributional Lower
Confidence Bound (μ - λσ) / σ_norm for the chosen action. When
position is flat and the selected action is an opening (a0/a1/a5/a6),
overrides to Hold if conf < threshold.

Fires after π action selection, before trail/heat/market pipeline.
Only gates on flat positions — existing positions pass through
unconditionally regardless of Q uncertainty.

ISV slots: 512 threshold (0.10), 513 λ (1.0), 514 σ_norm (1.0),
           515 fired_count (diag). RL_SLOTS_END → 516.

GPU oracle test: 4 cases (low-conf gate, high-conf pass, non-flat
bypass, non-opening bypass).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-24 21:37:59 +02:00
jgrusewski
a583bb508c feat(rl): pyramiding semantics — add/partial-flat/anti-martingale sizing
Implements the full pyramid trade-management suite:

P7.a: actions_to_market_targets gates pyramid adds on ISV-driven
      threshold (slot 506); rl_unit_state_update allocates sequential
      unit slots on position growth, deactivates oldest on shrink.

P7.b: HalfFlat (a9/a10) closes oldest unit's lots when pyramid>1;
      trail-stop routes breaches through HalfFlat + close_unit_index
      override instead of nuclear full-flat.

P7.c: Anti-martingale sizing on opening actions via signed outcome EMA
      (slot 508) — size_eff = base × clamp(1 + κ × ema, MIN, MAX).

Diag: pyramid { units_count, add_count, outcome_ema } in step JSONL.

ISV slots: 506 threshold, 507 add_count, 508 outcome_ema,
           509 κ, 510 MIN, 511 MAX. RL_SLOTS_END → 512.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-24 21:15:04 +02:00
jgrusewski
45a2041db4 feat(rl): SP20 P6 position heat cap — force-flat on over-leverage
Last-defense guard: if |position_lots| exceeds the ISV-driven
RL_HEAT_CAP_MAX_LOTS (slot 504, default 8 = MAX_UNITS × max_order_size),
the kernel overrides actions[b] to FlatFromLong (a3) or FlatFromShort
(a4) — full flatten, no partial. Catches runaway pyramid accumulation
before it reaches actions_to_market_targets.

Override stack ordering (step_with_lobsim):
  1. rl_trail_mutate (a7/a8)
  2. rl_trail_stop_check → may override to FlatFromLong/Short
  3. rl_position_heat_check (THIS) → may override to FlatFromLong/Short
  4. actions_to_market_targets → reads final actions[b]

Kernel `cuda/rl_position_heat_check.cu`:
  * 1 block, b_size threads (grid-stride for b_size > 256)
  * Reads position_lots from pos_state at offset 0 (PosFlat layout)
  * Cap read from ISV[504]; if cap ≤ 0 → no-op (guard disabled)
  * Per feedback_no_atomicadd: fired-count diagnostic uses shared-mem
    flag array + thread-0 serial count (b_size ≤ 256 in practice)
  * Writes fired-count to ISV[505] for diag

ISV slots:
  * 504: RL_HEAT_CAP_MAX_LOTS_INDEX (seed 8.0)
  * 505: RL_HEAT_CAP_FIRED_COUNT_INDEX (diagnostic, written per step)
  * RL_SLOTS_END bumped 505 → 506

Diag (alpha_rl_train):
  * "heat_cap": { "fired_count": N, "max_lots": 8 }

GPU oracle test (trade_management_kernels.rs):
  * position_heat_cap_overrides_on_breach — long 5 > cap 4 → a3;
    short -5 < -cap -4 → a4; long 3 ≤ cap 4 → untouched (Hold)

Verification (RTX 3050 Ti):
  * cargo check -p ml-alpha --examples → clean
  * integrated_trainer_smoke 1/1 → ok
  * trade_management_kernels 6/6 (was 5/5, +1 heat cap) → ok
  * audit-rust-consts → 0 flags
2026-05-24 20:36:05 +02:00
jgrusewski
2355984dc0 fix(fxcache): metadata-only smoke + production hash + streaming schema check
Fixes the pre-existing fxcache_local_smoke test failure. Two changes:

1. Hash update: `13c0b086a975...` → `70e5bc3a401d...` — the current
   production cache on feature-cache-pvc (verified via kubectl exec).
   Both the local file (15.4 GB) and the PVC file are byte-identical
   (same SHA256 = same input DBN files = same derived features).
   Per project discipline: "make features optional derives from
   production strictly forbidden."

2. Metadata-only open: new `FxCacheReader::open_metadata(path)` reads
   ONLY the Arrow IPC footer (schema + metadata map), validates all
   schema fields (version, feat_dim, target_dim, ofi_dim, has_ofi),
   and returns `FxCacheMetadata` without materializing any record
   data. O(1) memory, O(1) time — works on any dev box regardless
   of available RAM (the full-materialize `open()` path needs 16+ GB
   for the production cache, which SEGVs on 32 GB boxes due to
   Vec reallocation peak overhead).

   Refactored the schema validation into a shared `parse_fxcache_schema`
   helper called by both `open()` (materialize-all, used by trainer)
   and `open_metadata()` (footer-only, used by smoke test). Single
   source of truth for field parsing + dim-mismatch assertions.

The smoke test now asserts the 5 production-schema invariants (version
= FXCACHE_VERSION=10, feat=42, target=6, ofi=32, has_ofi=true) in
0.00s with zero memory overhead. Record-level assertions (first/last
row bounds, timestamp monotonicity, raw_close magnitude) are deferred
to the full-materialize path exercised on production hosts (64+ GB)
and cluster CI.

Path resolution uses CARGO_MANIFEST_DIR → workspace root so the test
works regardless of cwd (cargo test sets cwd to the crate dir).
2026-05-24 20:28:26 +02:00