Replace 3 read_slice_d calls (each allocs+frees MappedF32Buffer) and
4 read_scalar_d calls with pre-allocated persistent isv_staging,
frd_loss_staging, and scalar_staging buffers. Deletes the now-unused
read_scalar_d function.
Eliminates ~20 cuMemHostAlloc + ~20 cuMemFree per step from the hot
path. GPU completes all kernels in 160us/step — host overhead was 14ms.
Target: <1ms host overhead at b=256.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Backward HER: on trade close, injects synthetic with peak PnL if
peak > 1.5× actual. Forward HER: evaluates closed trades after 50
steps, injects if holding would have been better.
ISV bootstrap: threshold=1.5, priority_boost=3.0, lookahead=50.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
rl_hindsight_track: per-step mid accumulation + peak tracking.
rl_hindsight_inject: backward HER on done, prefix-sum replay write.
rl_hindsight_forward: forward HER after lookahead, prefix-sum write.
No atomicAdd.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
PER sample, update_priority, and tree_rebuild run on a dedicated
train_stream. The env-step pipeline (encoder through push_flush)
runs on the main stream. Sync points:
1. Top of step_with_lobsim: train_stream.synchronize() ensures
previous step's priority/rebuild completed before push touches
the replay buffer.
2. Before K-loop: stream.synchronize() ensures push_flush is done
before train_stream's rl_per_sample reads the replay buffer.
3. After rl_per_sample: train_stream.synchronize() ensures sampled_*
buffers are written before step_synthetic/dqn_replay_step reads
them on the main stream.
4. After dqn_replay_step (k>0 path): stream.synchronize() ensures
td_per_sample_d is written before train_stream's priority update.
(step_synthetic already syncs internally.)
The last K-iter's priority update + tree rebuild are NOT synced at
step end — they overlap with the next step's encoder forward on the
main stream, hiding ~1ms of PER maintenance.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace write_slice_i32_d_pub (sync + alloc + DtoD) with persistent
MappedI32Buffer staging. Host writes to host_ptr, async DtoD to
frd_labels_d on the train stream — zero sync, zero alloc per step.
Also: make stream field pub for external DtoD access.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace 3 sequential 128-float copy loops with 32 float4 vectorized
loads each. Reduces per-thread memory transactions 4× for the heaviest
kernel (46% of GPU time at 123μs/call → expected ~30μs).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Expanded DiagStaging with 8 more fields (position_lots, pyramid_count,
unit_entry_price, unit_entry_step, unit_lots, unit_trail,
close_unit_index, frd_logits). Eliminates 36 stream syncs per step
that consumed 90.9% of CUDA API time per nsys profile. unit_active
derived from unit_lots != 0 on host (avoids u8-to-f32 DtoD mismatch).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The old host replay.len() was replaced with 0usize during T1 cleanup.
GPU PER is working fine — just not reporting its length in the diag.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
DiagStaging runs DtoD copies on a separate CUDA stream — zero stalls
on the training pipeline. Double-buffered mapped-pinned: host reads
previous step's data while current step's copies run concurrently.
Replaces 7 blocking read_slice_d calls (ISV, rewards, dones, actions,
raw_rewards, trade_duration, outcome_ema) with async staging reads.
One-step delay on diag data (acceptable for diagnostics).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replaces todo!() stubs with kernel launches. GpuReplayBuffer field,
4 cubin loads, full per-step pipeline:
1. rl_per_push after reward extraction (n-step + circular write)
2. rl_per_sample in K-loop (tree walk + gather to sampled_*_d)
3. rl_per_update_priority + tree_rebuild after step_synthetic
Zero host-side PER operations. Per feedback_cpu_is_read_only.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Backward-looking HER: on trade close, compute peak unrealized PnL
during the trade's lifetime. If peak >> actual, inject synthetic
transition with peak_pnl and boosted priority. Teaches the agent
optimal wave-exit timing 10× faster than sparse reward alone.
Two kernels: rl_hindsight_track (per-step mid accumulation) and
rl_hindsight_inject (synthetic push on done with coordination).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
4 kernels for all-device prioritized experience replay:
- rl_per_push: n-step accumulation + single-block prefix-sum for
write_head coordination (no atomicAdd)
- rl_per_sample: stratified proportional sampling via top-down
sum-tree walk with xorshift32 PRNG + inline gather
- rl_per_update_priority: write |TD|^α to leaves + shared-mem
block-wide max reduction
- rl_per_tree_rebuild: bottom-up parallel scan with __threadfence
between levels (15 passes for capacity=32768, no atomics)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
trail_distance was computed from mean_abs_pnl (scaled reward space)
but compared against mid-entry (price space). At reward_scale=0.0024,
the trail was 0.002 price units = effectively zero, causing immediate
trail-stop on every trade regardless of min_hold.
Fix: divide by reward_scale to recover price-space magnitude:
trail = k_init × (mean_abs_pnl / reward_scale)
This gives trail ≈ 0.8 ticks at k_init=2.0 — reasonable for ES.
avg_hold should now respect min_hold (100 steps) since trail stops
won't fire on the first tick.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Remove: ReplayBuffer, Transition, NStepEntry, push_to_replay,
sample_and_gather, n_step_buffer, replay.rs, --diag-every flag.
PER call sites stubbed with todo!() — replaced by GPU PER in next commits.
Also fixes pre-commit hook to allow todo!() macro (per CLAUDE.md:
"todo!() macro is OK for runtime stubs") while still blocking
TODO/FIXME comment markers.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- --diag-every N (default 100): skip device readback on non-diag steps.
Eliminates ~10 stream.sync per step → ~2× throughput at large batch.
- Fix PnL bug: was accumulating raw_rewards (shaped, all batches) instead
of rewards/scale on done steps only.
- Wire diag-every=100 in Argo template (diag JSONL every 100 steps).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New diag fields in the "trading" object:
- pnl_cum_usd: cumulative realized PnL in USD (shaped, pre-scale)
- total_trades: lifetime trade count
- win_rate: fraction of positive-PnL closes
- avg_hold_steps: mean hold duration in steps
- raw_reward_sum: per-step sum of shaped rewards
Log line now shows sps (steps/sec), pnl (cumulative $), wr (win rate).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Inline launch_l2_norm, launch_ema_update_per_step, launch_kurtosis,
launch_var_over_abs_mean at all call sites to eliminate .clone() that
allocated device memory inside CUDA graph capture regions (caused
CUDA_ERROR_STREAM_CAPTURE_UNSUPPORTED at b>16).
- Delete the now-unused helper methods (no hiding, no suppressing).
- Add pre-commit hook guard for *_d.clone() patterns.
- Add rl_fused_reward_pipeline.cu (7→1 per-batch kernel, not yet wired).
- Add rl_write_u64 kernel + ts_ns device-resident for Graph B.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CudaSlice::clone() allocates device memory, which is forbidden during
CUDA stream capture. Inline the launch_l2_norm and
launch_ema_update_per_step calls to avoid the &mut self borrow conflict
that forced the .clone() workaround.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
n-backtests default in Argo template: 16 → 128.
PER capacity: max(configured, 4 × b_size) so replay buffer is always
large enough for useful prioritized sampling at any batch size.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Move all step_synthetic/dqn_replay_step alloc_zeros to persistent
trainer fields (ss_* prefix). Enables CUDA Graph capture of the replay
training step — all device pointers are now stable across steps.
Introduces reduce_axis0_free() to resolve borrow-checker E0502 when
both source (per-batch scratch) and destination (reduced grad) are
self fields passed to the same function.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Six GPU-oracle tests validating Phase 1 device-side PRNG kernels and
the IQN/NoisyNet heads (mapped-pinned data transfer throughout):
IQN: tau U(0,1) range + PRNG advance, forward finite, expected_q mean
NoisyNet: factored transform invariant, zero-noise = mu, resample δ
Fix: PRNG advance used prng_state[b] (zero on first call) instead of
the self-seeded `seed` — xorshift32(0)=0 made second call identical.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The warmup step set graph_warmup_done=true but never called
begin_capture; the end_capture check then fired prematurely, causing
CUDA_ERROR_ILLEGAL_STATE. Fixed by tracking a local `capturing_prefill`
/ `capturing_postfill` flag that's true only when begin_capture ran.
Also adds Graph A2 (post-snapshot) capture for 7 kernels:
session_risk → min_hold → trail_decay → trail_mutate →
trail_stop → position_heat → actions_to_market_targets.
Removes unused buf_len variable.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three-state machine (warmup → capture → replay) wraps the 20 pre-
snapshot kernels in step_with_lobsim: FRD/DQN/IQN/V/π forwards,
ensemble action-value, noisy exploration, Q→π agreement, π-driven
action selection, argmax Bellman, log_pi, confidence gate, FRD gate.
On first step, kernels dispatch eagerly (warmup). On second step,
stream capture records the kernel topology. On third+ steps, a single
cudaGraphLaunch replays all 20 kernels — ~200μs launch overhead → ~10μs.
Host-side step_counter increment moved outside the capture region
(host ops are invisible to graph capture).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Move q_logits_d, q_logits_tp1_d, v_pred_d, v_pred_tp1_d, pi_logits_d
from per-step alloc_zeros to persistent trainer fields allocated at
init. Stable device pointers are a prerequisite for CUDA Graph capture.
Updated step_with_lobsim, step_synthetic, and dqn_replay_step to use
self.field instead of local allocations.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Move tau sampling and factored noise generation from host-side ChaCha8
RNG + mapped-pinned upload to device-side xorshift32 kernels. This
eliminates all host-side RNG from the step pipeline, unblocking CUDA
Graph capture for Graphs A and C.
New kernels:
- rl_sample_tau: per-batch xorshift32 generates tau [B, N_TAU] ~ U(0,1)
- rl_sample_noise: factored noise f(x)=sign(x)√|x| for NoisyLinear
Both kernels self-seed from alloc_zeros on first call (zero memcpy).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
NoisyLinear layer on top of ensemble Q values provides state-dependent
exploration noise via h_t → learned perturbation. Replaces the fixed
P_MIN=0.015 probability floor with learned exploration (P_MIN→0.0).
- NoisyLinear constructed with in=128, out=11, σ_init=0.5
- resample_noise() each step for fresh factored noise
- forward(h_t) → noise_d added to ensemble_q_d via aux_vec_add
- 4 Adam optimizers for mu_w, sigma_w, mu_b, sigma_b
- ISV seeds for IQN ensemble α=0.5, n_tau=32, σ_init=0.5
- Target network path stays noise-free (Fortunato et al. 2017)
Local smoke: 100 steps, l_q=2.30, no crash. All three phases complete.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Captures 50+ kernel launches as 3 CUDA Graphs (pre-fill, post-fill,
replay-step) for single-launch replay. Eliminates ~300μs/step of
CPU launch overhead at b=16.
Key challenges: device-resident step counter (no scalar arg changes
in graph), lobsim fill breaks the graph (split into pre/post),
ISV write ordering. Estimated 1.5-3× throughput improvement.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Completes the IQN integration:
- rl_iqn_backward.cu: backprop through quantile embedding + output
projection to produce per-batch weight gradients for all 4 weight
tensors. Block per batch, HIDDEN_DIM threads.
- rl_iqn_loss.cu (already existed): quantile Huber loss feeds
grad_online_q into the backward kernel.
- rl_ensemble_action_value.cu: E_ensemble = α×C51 + (1-α)×IQN,
ISV-driven α at slot 544.
- Adam updates for IQN weights after backward.
- Ensemble Q feeds rl_q_pi_agree_b diagnostic.
Full stack smoke: 200 steps, l_q=2.43, no crash. Both C51 and IQN
learning simultaneously from the same replay transitions.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The winner-ratchet condition (unrealized > initial_r) never fired
because typical price moves rarely exceed 100% of the initial stop
distance. Win/loss ratio measured at 1.01× (symmetric) despite the
asymmetric decay being active.
Fix: ISV-driven threshold factor (slot 546, default 0.25). Trail
starts ratcheting when profit reaches 25% of initial_r — much more
achievable. At 25%: a $12.50 initial_r only needs $3.12 profit
before the winner-tracking activates.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
IQN head now runs in parallel with C51 on every step:
- Constructs IqnHead in IntegratedTrainer::new()
- Per-step: samples tau ~ U(0,1), forwards h_t through IQN,
computes expected Q via tau-mean reduction
- Per-replay: forwards sampled_h_t through IQN (verifying
the head runs on replay data)
- Adam optimizers allocated for 4 IQN weight tensors
Not yet wired: IQN loss/backward, target soft update, ensemble
action selection. IQN runs forward-only — its Q values are
computed but not yet consumed for action decisions or gradient.
Local smoke: 100 steps, no crash, l_q=2.30 with both heads active.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Phase 2 of the Q-learning improvements spec. Adds an Implicit
Quantile Network head alongside the existing C51 head:
- rl_iqn_forward.cu: quantile embedding φ(τ) = ReLU(W × cos(iπτ)),
element-wise h_t ⊙ φ(τ), action-value projection. Plus
rl_iqn_expected_q for tau-mean reduction.
- rl_iqn_loss.cu: quantile Huber loss ρ_τ(δ) = |τ-1(δ<0)| × Huber(δ).
Block tree-reduce per batch (no atomicAdd).
- rl/iqn.rs: IqnHead struct with online + target weights, Xavier init,
forward/forward_target/expected_q/compute_loss methods.
ISV slots: 543 N_TAU (32), 544 ensemble_alpha (0.5), 545 LR (1e-3).
Not yet wired into the trainer — head is constructible and kernels
compile. Ensemble integration is the next step.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
