n-backtests default in Argo template: 16 → 128.
PER capacity: max(configured, 4 × b_size) so replay buffer is always
large enough for useful prioritized sampling at any batch size.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Move all step_synthetic/dqn_replay_step alloc_zeros to persistent
trainer fields (ss_* prefix). Enables CUDA Graph capture of the replay
training step — all device pointers are now stable across steps.
Introduces reduce_axis0_free() to resolve borrow-checker E0502 when
both source (per-batch scratch) and destination (reduced grad) are
self fields passed to the same function.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Six GPU-oracle tests validating Phase 1 device-side PRNG kernels and
the IQN/NoisyNet heads (mapped-pinned data transfer throughout):
IQN: tau U(0,1) range + PRNG advance, forward finite, expected_q mean
NoisyNet: factored transform invariant, zero-noise = mu, resample δ
Fix: PRNG advance used prng_state[b] (zero on first call) instead of
the self-seeded `seed` — xorshift32(0)=0 made second call identical.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The warmup step set graph_warmup_done=true but never called
begin_capture; the end_capture check then fired prematurely, causing
CUDA_ERROR_ILLEGAL_STATE. Fixed by tracking a local `capturing_prefill`
/ `capturing_postfill` flag that's true only when begin_capture ran.
Also adds Graph A2 (post-snapshot) capture for 7 kernels:
session_risk → min_hold → trail_decay → trail_mutate →
trail_stop → position_heat → actions_to_market_targets.
Removes unused buf_len variable.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three-state machine (warmup → capture → replay) wraps the 20 pre-
snapshot kernels in step_with_lobsim: FRD/DQN/IQN/V/π forwards,
ensemble action-value, noisy exploration, Q→π agreement, π-driven
action selection, argmax Bellman, log_pi, confidence gate, FRD gate.
On first step, kernels dispatch eagerly (warmup). On second step,
stream capture records the kernel topology. On third+ steps, a single
cudaGraphLaunch replays all 20 kernels — ~200μs launch overhead → ~10μs.
Host-side step_counter increment moved outside the capture region
(host ops are invisible to graph capture).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Move q_logits_d, q_logits_tp1_d, v_pred_d, v_pred_tp1_d, pi_logits_d
from per-step alloc_zeros to persistent trainer fields allocated at
init. Stable device pointers are a prerequisite for CUDA Graph capture.
Updated step_with_lobsim, step_synthetic, and dqn_replay_step to use
self.field instead of local allocations.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Move tau sampling and factored noise generation from host-side ChaCha8
RNG + mapped-pinned upload to device-side xorshift32 kernels. This
eliminates all host-side RNG from the step pipeline, unblocking CUDA
Graph capture for Graphs A and C.
New kernels:
- rl_sample_tau: per-batch xorshift32 generates tau [B, N_TAU] ~ U(0,1)
- rl_sample_noise: factored noise f(x)=sign(x)√|x| for NoisyLinear
Both kernels self-seed from alloc_zeros on first call (zero memcpy).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
NoisyLinear layer on top of ensemble Q values provides state-dependent
exploration noise via h_t → learned perturbation. Replaces the fixed
P_MIN=0.015 probability floor with learned exploration (P_MIN→0.0).
- NoisyLinear constructed with in=128, out=11, σ_init=0.5
- resample_noise() each step for fresh factored noise
- forward(h_t) → noise_d added to ensemble_q_d via aux_vec_add
- 4 Adam optimizers for mu_w, sigma_w, mu_b, sigma_b
- ISV seeds for IQN ensemble α=0.5, n_tau=32, σ_init=0.5
- Target network path stays noise-free (Fortunato et al. 2017)
Local smoke: 100 steps, l_q=2.30, no crash. All three phases complete.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Captures 50+ kernel launches as 3 CUDA Graphs (pre-fill, post-fill,
replay-step) for single-launch replay. Eliminates ~300μs/step of
CPU launch overhead at b=16.
Key challenges: device-resident step counter (no scalar arg changes
in graph), lobsim fill breaks the graph (split into pre/post),
ISV write ordering. Estimated 1.5-3× throughput improvement.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Completes the IQN integration:
- rl_iqn_backward.cu: backprop through quantile embedding + output
projection to produce per-batch weight gradients for all 4 weight
tensors. Block per batch, HIDDEN_DIM threads.
- rl_iqn_loss.cu (already existed): quantile Huber loss feeds
grad_online_q into the backward kernel.
- rl_ensemble_action_value.cu: E_ensemble = α×C51 + (1-α)×IQN,
ISV-driven α at slot 544.
- Adam updates for IQN weights after backward.
- Ensemble Q feeds rl_q_pi_agree_b diagnostic.
Full stack smoke: 200 steps, l_q=2.43, no crash. Both C51 and IQN
learning simultaneously from the same replay transitions.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The winner-ratchet condition (unrealized > initial_r) never fired
because typical price moves rarely exceed 100% of the initial stop
distance. Win/loss ratio measured at 1.01× (symmetric) despite the
asymmetric decay being active.
Fix: ISV-driven threshold factor (slot 546, default 0.25). Trail
starts ratcheting when profit reaches 25% of initial_r — much more
achievable. At 25%: a $12.50 initial_r only needs $3.12 profit
before the winner-tracking activates.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
IQN head now runs in parallel with C51 on every step:
- Constructs IqnHead in IntegratedTrainer::new()
- Per-step: samples tau ~ U(0,1), forwards h_t through IQN,
computes expected Q via tau-mean reduction
- Per-replay: forwards sampled_h_t through IQN (verifying
the head runs on replay data)
- Adam optimizers allocated for 4 IQN weight tensors
Not yet wired: IQN loss/backward, target soft update, ensemble
action selection. IQN runs forward-only — its Q values are
computed but not yet consumed for action decisions or gradient.
Local smoke: 100 steps, no crash, l_q=2.30 with both heads active.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Phase 2 of the Q-learning improvements spec. Adds an Implicit
Quantile Network head alongside the existing C51 head:
- rl_iqn_forward.cu: quantile embedding φ(τ) = ReLU(W × cos(iπτ)),
element-wise h_t ⊙ φ(τ), action-value projection. Plus
rl_iqn_expected_q for tau-mean reduction.
- rl_iqn_loss.cu: quantile Huber loss ρ_τ(δ) = |τ-1(δ<0)| × Huber(δ).
Block tree-reduce per batch (no atomicAdd).
- rl/iqn.rs: IqnHead struct with online + target weights, Xavier init,
forward/forward_target/expected_q/compute_loss methods.
ISV slots: 543 N_TAU (32), 544 ensemble_alpha (0.5), 545 LR (1e-3).
Not yet wired into the trainer — head is constructible and kernels
compile. Ensemble integration is the next step.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replaces 1-step Bellman r + γQ(s') with n-step R_n + γⁿQ(s_{t+n}).
At γ=0.995, n=10 gives the agent 10 real rewards (~2.5 seconds)
before bootstrapping from Q, dramatically reducing bootstrap error.
Implementation:
- NStepEntry ring buffer per batch in IntegratedTrainer
- push_to_replay accumulates entries, flushes when len==n or done
- R_n = Σ γᵏ rₖ computed at flush time (both scaled + raw)
- n_step_gamma = γⁿ (or 0 if any done in window) stored per transition
- bellman_target_projection.cu uses per-transition n_step_gamma
instead of the global ISV γ for the bootstrap discount
- project_bellman_target wrapper takes n_step_gammas_d buffer
ISV slot 542 (RL_N_STEP_INDEX, default 10). Local smoke: 1k steps,
no crash, replay=150 (correct for n=10 with dones).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds RL_N_STEP_INDEX (slot 542, default 10), n_step_gamma field to
Transition, and NStepEntry ring buffer struct to IntegratedTrainer.
Remaining: implement n-step accumulation in push_to_replay and
modify bellman_target_projection.cu to use n_step_gamma.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Enumerates all project rules that apply to the Q-learning
improvements: CPU read-only, no atomicAdd, mapped-pinned only,
ISV-driven params, first-observation bootstrap, bilateral clamp,
raw reward in replay, no stubs/TODO/feature flags, GPU oracle
tests, local smoke before cluster, surfer philosophy constraints.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three-phase plan to push Q convergence past profitability:
1. N-step returns (n=10): 10× more direct reward signal
2. IQN complementary head alongside C51: ensemble action selection
3. Noisy linear layers: state-dependent exploration
C51 stays for the confidence gate's distributional LCB. IQN adds
flexible quantile estimation. Ensemble combines both for action
selection. All ISV-driven.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Prepared for next run after the γ=0.99 1M run completes:
- γ floor 0.99 → 0.995: horizon 100 → 200 steps (50 seconds).
Real ES directional moves (2-5 points) happen at this scale.
- PER 16384 → 32768: 2048 unique steps of replay depth. Supports
the 200-step γ horizon with margin.
- Min-hold 50 → 100 steps (25 seconds): commit to the full wave.
Short-hold penalty threshold matches.
Safe to push because raw-reward re-normalization eliminates scale
drift in the deeper buffer, and the ±2% scale clamp keeps targets
stable across the 2048-step replay window.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The agent was break-even at the wrong timescale — trading in
bid-ask noise at 1-4 second holds where there IS no edge. Real
directional moves live at 10-60 seconds.
Three changes push the agent to wave-scale:
1. γ floor 0.90 → 0.99: effective horizon 10 → 100 steps (25s).
Q now values what happens 25 seconds from now, not just the
next tick. The FRD's 10s and 600s horizons become relevant.
2. Min-hold 20 → 50 steps (12.5s): forces commitment to waves,
not ripples. Short-hold penalty threshold matches.
3. Long-ride bonus: at trade close, profitable rides get reward
multiplied by (1 + bonus × sqrt(hold_time)). A 100-step
winning ride gets 21× the raw PnL as reward. Combined with
per-step hold bonus boosted to $2.00. "The best wave of the
session deserves the biggest cheer."
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The reward_scale oscillated 10,873× (min 0.000123, max 1.33) because
the Wiener-α=0.4 blend tracked sparse trade PnL spikes instantly.
Old transitions in PER had stale-scale rewards even with raw-reward
re-normalization (the current scale itself was unstable).
Per-step clamp: scale can only move ±2% from previous value.
Doubling takes ~35 steps, halving takes ~35 steps — fast enough to
track regime changes, stable enough for Q targets across the replay
window. Max oscillation over 1000 steps: ~7× (was 10,873×).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Min-hold check was overriding heat cap's FlatFromLong/Short to Hold,
preventing the safety exit. Position grew to 9 (above cap 8) because
the flat was blocked by min-hold, then next step added lots.
Fix: min-hold reads pos_state and ISV heat cap slot. If |position|
>= cap, the close action passes through regardless of hold time.
Safety always overrides patience.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Overrides closing actions (a3/a4/a9/a10) to Hold when
steps_since_done < RL_MIN_HOLD_STEPS_INDEX (slot 536, default 20).
The agent CANNOT exit before the minimum — forced to ride the wave.
Trail stops still fire regardless (safety overrides patience) —
if the market moves against the position past the trail distance,
the stop-loss exits even within the min-hold window.
Pipeline order: action selection → confidence gate → FRD gate →
min_hold_check → trail_mutate → trail_stop → heat_cap →
actions_to_market_targets
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three ISV-driven reward shaping components discourage churning and
reward patience:
1. Entry cost ($15 default): subtracted when opening a new position.
Agent must expect profit > cost to justify entry. Slightly above
ES 1-tick spread ($12.50) so marginal trades are net-negative.
2. Short-hold penalty (0.5× for holds < 20 steps): multiplicative
penalty at trade close for quick flips. "Don't bail on the first
bump" — halves the reward for sub-5-second holds.
3. Hold bonus ($0.50/step × sqrt(hold_time)): per-step reward for
staying in a profitable position. "Ride the wave" — incentivizes
patience when the trade is working.
Runs BEFORE reward_scale so all costs/bonuses are in raw USD terms.
ISV slots 532-535. RL_SLOTS_END → 536.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Binary argmax agreement is pure noise for near-uniform distributions
(both Q and π give ~9% to each of 11 actions, so argmax flips on
0.001 differences → metric reads 0 even when KL=0.02).
Cosine similarity between E_Q[a] (expected Q-value per action) and
softmax(π)[a] measures full ranking direction agreement. Robust to
near-uniform: two identical distributions → cosine=1.0 regardless
of which action is marginally highest.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Root cause of q_pi_agree collapse: the symmetric Schulman controller
decayed λ from 0.5 to 0.001 in 34 steps (÷1.2/step), killing Q→π
coupling. Once decoupled, Q and π learned independent policies,
making q_pi_agree drop to 1e-22.
Three fixes:
1. MIN_LAMBDA 0.001 → 0.05: Q pull never drops below 5% strength
2. Asymmetric rates: ramp 1.2×/step (4 steps to double), decay
0.998×/step (347 steps to halve). Coupling establishes fast and
persists for thousands of steps.
3. Dead zone tolerance 1.5× → 3.0×: natural KL fluctuation stays
in-band instead of triggering constant ramp/decay cycles.
Seed λ raised to 0.1 so distillation is active from step 0.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Stored raw_reward alongside scaled reward in Transition. At sample
time, consumer re-applies current reward_scale to raw_reward instead
of using the stale-scale stored reward. This eliminates off-policy
scale drift that caused q_pi_agree to collapse from 0.125 to 1e-22
over 40k steps with PER=32768.
PER capacity set to 16384 (1024 unique steps at b=16) — balances
replay depth against h_t representation staleness.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The CLI default overrode the trainer config default. Without passing
--per-capacity explicitly, the Argo run used 4096 (256 unique steps
at b=16) instead of the intended 32768 (2048 steps).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Root cause of Q non-convergence: at reward_scale ≈ 0.003, a typical
±$5 trade maps to ±0.015 scaled reward. With 21 atoms spanning [-1,+1]
(step=0.1), the entire reward distribution falls within a SINGLE atom.
Q cannot distinguish wins from losses — confirmed by q_pi_agree ≈ 0.
Fix 1: Tighten atom span to [-0.5, +0.5]. Now atom_step = 0.05.
A ±$5 trade at scale=0.01 spans 2 atoms; at scale=0.05 spans 10.
The reward clamp controller ratchets the span at runtime if rewards
exceed the bounds.
Fix 2: PER capacity 4096 → 32768. At b=16, old capacity retained
only 256 unique steps — Q replayed each transition 1-2× before
eviction. New capacity retains 2048 steps, giving Q proper replay
depth for convergence.
Also aligns reward clamp bounds (WIN/LOSS) with the new span.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
With strict >, pos=8 passed the cap check, then a LongLarge(+2)
made pos=10 before the next step's cap fired. Using >= prevents
position from ever exceeding the cap by the fill size.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Controller oscillated: 3 dones at step 15k spiked EMA above 2×target,
triggering thousands of steps of tightening that killed trades.
Higher target (10% vs 2%) matches the natural trade frequency at
b=16. Wider dead zone (5× above / 0.2× below) prevents single-batch
spikes from triggering tighten/relax oscillation.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Confidence gate LCB fix worked (0 fires post-warmup) but FRD gate
blocked 5212 entries in 5k steps — the FRD head learned "forward
returns are negative" during warmup (losing trades), so favorable
tail mass dropped below 0.15 for all actions.
Lower default to 0.05 and floor to 0.0 so the adaptive controller
can fully disable when trades dry up.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The old formula conf = clamp((μ - λσ) / σ_norm, 0, 1) produced
conf=0 for ANY distribution with σ > μ — including uniform Q at
training start (μ=0, σ=0.577 → LCB=-0.577 → conf=0). This caused
permanent 100% block rate regardless of threshold setting.
New formula: conf = clamp((μ - V_MIN - λσ) / (V_MAX - V_MIN), 0, 1)
Shifts by V_MIN so conf measures "how far above worst case":
Uniform: conf = 0.21 (passes threshold 0.01)
Peaked V_MAX: conf ≈ 1.0 (high confidence)
Peaked V_MIN: conf ≈ 0.0 (blocked — correct)
Also sets conf gate floor to 0.0 so the adaptive controller can
fully disable the gate when trades dry up.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
When C51 LCB is ≈0 for all actions (typical early training), even
0.001 threshold blocks everything. Floor=0.0 lets the controller
drive to zero when trades dry up, then ratchet back as Q calibrates.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New rl_gate_threshold_controller.cu watches dones EMA and adjusts
confidence + FRD gate thresholds via Schulman bounded step:
- dones_ema < target×0.5 → relax thresholds (×0.95)
- dones_ema > target×2.0 → tighten thresholds (÷0.95)
ISV-driven: target (0.02), conf bounds (0.001-0.50), FRD bounds
(0.05-0.50), adjust rate (0.95). Runs per step after warmup.
Also lowers default thresholds: conf 0.10→0.01, FRD 0.35→0.15.
Post-warmup, the controller adapts these based on actual trade
flow instead of relying on static defaults.
ISV slots 525-531. RL_SLOTS_END → 532.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Both gates now read RL_GATE_WARMUP_STEPS_INDEX (slot 524, default
10000) and return early when current_step < warmup. During warmup,
the agent opens positions freely, collects reward signal, and
calibrates Q. After warmup, gates activate and filter low-quality
entries.
Without warmup, gates blocked 100% of opening actions from step 0
(uniform Q → zero confidence → permanent Hold attractor → zero
trades → zero reward → Q never learns). Confirmed by 50k-step
L40S run with zero dones across 800k action decisions.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replaces the 4-step DAG (check-cache → compile on CPU → warmup GPU →
train on GPU) with a single pod on the L40S that does everything:
git fetch → incremental cargo build (~3s warm) → train.
Eliminates: separate compile node, node autoscale wait, binary
transfer, fxcache step. The ci-builder image has CUDA 13.0 devel +
Rust — compilation and training use the same CUDA libs.
The cargo-target-cuda PVC provides incremental build cache across
runs. LD_LIBRARY_PATH strips the stubs dir so real CUDA libs load.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The MBP-10 predecoded sidecars live on feature-cache-pvc at
/feature-cache/predecoded/, not on training-data-pvc. Points
--predecoded-dir to the correct PVC mount.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
alpha_rl_train reads MBP-10 data with its own predecoded cache —
it never uses the fxcache pipeline. Skip the 15-minute feature
extraction step entirely for model=alpha-rl by depending only on
ensure-binary + gpu-warmup.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The ml-alpha branch uses alpha_rl_train (from ml-alpha crate) with
different CLI args than train_baseline_rl (from ml crate). Adds
model=alpha-rl case that:
- Builds ml-alpha --example alpha_rl_train
- Runs with --mbp10-data-dir, --predecoded-dir, --n-steps 50000,
--n-backtests 16 (matching the local smoke config)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
