LOW 1: sel_t_buf CudaSlice + per-step memcpy_htod → sel_t_pinned
device-mapped. Zero copies, GPU reads directly from host memory.
LOW 2: is_q_gap_frozen uniform [q_gap; 4] → actual per-branch Q-gaps
from 48B DtoH readback in reduce_current_q_stats. Each branch now
independently detected as frozen/unfrozen. Liquid tau and trajectory
backtracking use real per-branch velocities.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
mag_concat now uses vsn_masked (feature-selected) instead of raw h_s2
for its first SH2 columns. strided_scatter kernel writes tight [B,SH2]
vsn_masked into wide [B,SH2+3] mag_concat preserving Q_dir in last 3.
launch_mag_concat → launch_mag_concat_from with explicit source ptr.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CRITICAL: backward_full used state_dim for goff_b_s1 offset but
compute_param_sizes uses s1_input_dim (smaller with bottleneck active).
All downstream gradient offsets were misaligned. Fixed: sd → s1d.
HIGH: q_attn_params used alloc_f32 (potentially non-zero) instead of
alloc_zeros. Cross-branch Q-attention residual connection needs near-zero
init for stability. Fixed: alloc_zeros.
MEDIUM (deferred): VSN masking bypassed for magnitude branch — mag_concat
reads raw h_s2 instead of vsn_masked. No impact while W_vsn2 is zero-init
(identity mask). Will fix when VSN backward is wired.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
selectivity_forward: sigmoid(dot(W_sel, h_s2) + b_sel) per sample.
selectivity_backward: BCE gradient on (sel, per_sample_loss/mean_loss).
Separate Adam at LR=1e-4. PER priority integration deferred.
Also fixes kernel names (selectivity_gate_fwd→selectivity_forward,
selectivity_gate_bwd→selectivity_backward) to match experience_kernels.cu.
Adds sel_norm_buf, sel_norm_partials, sel_clip_buf, sel_t_buf fields
needed by dqn_adam_update_kernel. Wired into run_full_step after main
Adam step with mean_loss=1.0 placeholder and non-fatal error handling.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
c51_grad_kernel reads liquid_mod[d] per branch instead of spread_velocity[0].
4 per-branch Q-gap EMAs with adaptive alpha drive continuous-time ODE.
Fast branch learning → large tau → slow adaptation → don't overshoot.
Stuck branch → small tau → fast adaptation → push harder.
spread_velocity infrastructure removed (-30 lines).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
launch_q_attention runs 2-head attention on 12 Q-values [B, 12].
Wired into reduce_current_q_stats after compute_expected_q.
Also fixes kernel function name: q_cross_branch_attn → cross_branch_q_attention.
Experience collection deferred — uses raw Q-values for now.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
d_h_bd → glu_backward → d_value + d_gate_pre. Two weight gradient
GEMMs per branch (value + gate). Upstream gradient sums both paths.
VSN backward DEFERRED (W_vsn init to identity, zero gradient for now).
Gradient flows to gate weights at indices 34-41.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Checkpoint ring buffer ranked by improvement rate, plateau detection
via liquid tau velocity, informed perturbation cycle (Adam reset,
shrink-perturb, temp boost, LR×2), depth-limited rewind. ~150 lines
in training_loop.rs, zero CUDA changes. Recovers from plateaus that
prevention mechanisms (spread gradient, liquid tau) fail to avoid.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
param_sizes[12] grows by 3*adv_h for direction conditioning.
mag_concat_buf [B, SH2+3] and d_mag_concat_buf allocated.
mag_concat_qdir and strided_accumulate kernels loaded.
launch_mag_concat and accumulate_d_h_s2_from_concat methods added.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
mag_concat_qdir: computes direction E[Q] from softmax over C51 atoms,
normalizes by delta_z for matched gradient scale, concatenates with h_s2
to produce [B, SH2+3] input for magnitude branch FC.
strided_accumulate: extracts first SH2 columns from d_mag_concat [B, SH2+3]
into d_h_s2 [B, SH2] with beta accumulation for backward pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Raw E[Q] ~ 0.1 while h_s2 ~ 1.0 post-ReLU. Without normalization,
Q_dir columns learn 10x slower. Dividing by delta_z scales to ~1-10.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
In-place ranking races across blocks: block X writes shaped rewards[i]
before block Y loads it in tile scan. __syncthreads is block-local.
Fix: read from rewards_out (raw), write to gpu_batch.rewards (clone).
The DtoD clone from collect_experiences_gpu provides the output buffer.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Magnitude FC input changes from h_s2[B, SH2] to [h_s2; Q_dir][B, SH2+3].
Direction E[Q] values (3 scalars per sample) are computed from branch 0
logits and concatenated. w_b1fc grows by 3*adv_h parameters (768 for AH=256).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Four improvements to the training pipeline:
1. Asymmetric spread gradient: challenger action (adjacent to taken) gets
pushed UP at half strength instead of DOWN. Creates a two-horse race
instead of single-action monopoly.
2. Per-branch spread scaling: spread_grad *= branch_scale. Direction branch
(high impact) gets more spread than urgency (low impact).
3. Distributional variance position sizing (Layer 4): Var[Q] = E[Z²] - E[Z]²
computed in compute_expected_q. Position scaled by 1/(1+sqrt(Var[Q_taken])).
High uncertainty → smaller position. Kelly criterion from C51 atoms.
4. Reward std guard: skip rank normalization when observed_reward_std ≈ 0
(epoch 0). Prevents zeroing all rewards before std is observed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds a spread gradient to the C51 advantage logits that pushes the
taken action's distribution toward higher atoms and non-taken toward
lower. Scale = inv_batch * delta_z (adaptive to per-sample atom
resolution, zero hardcoded constants).
This gradient is ORTHOGONAL to the Bellman equation — it depends on
atom position, not target match. Active on ALL samples, providing
perpetual pressure to differentiate Q-values even when the C51
cross-entropy gradient vanishes at convergence. Prevents the Q-gap
plateau where all actions have identical Q-values.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Philox seed used batch_size+sample_id (constant across steps) so the
same action was always sampled for the same sample → at the fixed point,
identical target every step → zero gradient. Now uses t_buf (Adam step
counter, device-mapped, increments every step). Each training step
samples a DIFFERENT action per sample → target is truly stochastic →
gradient variance prevents convergence.
Also moves philox_uniform to common_device_functions.cuh for reuse.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Deterministic softmax mixture produced a fixed target that the model
matched exactly → zero gradient → Q-gap frozen at 0.167, val_Sharpe
frozen at -5.62 from epoch 22. Stochastic sampling picks ONE action
per sample from the softmax weights. Expected target over many steps
is identical (unbiased) but per-step target varies → model can never
match a moving target → perpetual gradient → no convergence trap.
Also moves philox_uniform to common_device_functions.cuh so all
kernels can use it.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Floor = max(|mean_Q| * 0.01, 1e-6). At Q-mean=0.005: floor=5e-5.
At Q-mean=1.0: floor=0.01. Fully adaptive, zero hardcoded constants.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
tau=Q_gap/3 gave 20:1 weight ratio (95%/4%/1%), making the target
essentially argmax. Once the online distribution matched this sharp
target, cross-entropy gradient vanished → Q-gap froze at 0.158,
val_Sharpe locked at 2.40 from epoch 22.
tau=Q_gap gives a CONSTANT e:1 ≈ 2.72:1 ratio (63%/23%/14% for 3
actions) regardless of Q-gap magnitude. The target always maintains
meaningful mixture → perpetual gradient pressure → Q-values keep
evolving. Scale-invariant: the softmax ratio depends only on relative
Q-differences, not their absolute magnitude.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All 4 branches (dir, mag, order, urgency) now use Boltzmann sampling
in eval mode instead of argmax. argmax(softmax(Q/tau)) = argmax(Q) —
temperature is meaningless with argmax. Every state picked the same
winning action → identical trades → Sharpe=0.00 from epoch 8 onward.
Boltzmann sampling with Philox seed gives deterministic but
action-diverse evaluation. When Q-values differentiate, Boltzmann
naturally sharpens toward the best action.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Softmax weights used implicit tau=1, producing near-uniform weights
when Q-gap=0.06: exp(0.06)≈1.06 → weights [0.35, 0.33, 0.32].
The mixture averaged all distributions → zero gradient to differentiate
actions → Q-gap frozen → val_Sharpe locked at 0.00 after epoch 19.
Now uses tau = max(Q_gap/3, 0.001). With Q-gap=0.06 and tau=0.02:
exp(0.06/0.02) = exp(3) ≈ 20:1 weight ratio between best and worst.
The 0.001 floor means even tiny Q-differences produce sharp weights,
preventing the uniform-averaging fixed point.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
read_total_loss() calls cuStreamSynchronize which is illegal during
CUDA Graph stream capture. Was inside submit_aux_ops (captured in
graph_mega/graph_aux). Moved to Step 4 (conditional ops outside graph).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
branching_action_select exposure head used hard argmax while order/
urgency used Boltzmann. experience_action_select eval mode used pure
argmax for direction. Both replaced with Boltzmann softmax:
tau = max(Q_max - Q_min, 0.01)
When Q-values differentiate: converges to argmax. When flat: spreads
evenly. Eval mode takes argmax of softmax probs (deterministic).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two fixes:
1. Width: half = max(10*q_gap, 3*q_std, 0.1) instead of fixed 3*q_std+1.0.
With q_gap=0.06 and 51 atoms, gives ~5 atoms of action resolution
(was 1.5 atoms with the 1.0 floor). No more catastrophic -40 dips
from atom underresolution.
2. Baseline: uses q_std_ema (proportional to natural Q oscillation)
instead of fixed 0.01. Normal Q-oscillation (±q_std) no longer
triggers aggressive alpha — only abnormal shifts do.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Readiness jumped from 0 to ~0.8 in one step when CV first improved,
causing per-sample support to snap from [-1,1] to V(s)-centered
discontinuously. Same adaptive alpha pattern as eval_v_range:
α = |error| / (|error| + 0.05), clamped [0.01, 0.3]. Smooth ramp-in.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
EMA init used == 0.0 sentinel and snapped to first non-zero Q-stats,
causing a discontinuous v_range transition → val_Sharpe=30 spike at
epoch 3 followed by -1 correction. Now seeds from neutral center
(mean=0, std=1 → v_range=[-4,4]) and the adaptive alpha tracks
to actual Q-stats smoothly over 2-3 steps.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When Q-mean shifted from +0.01 to -0.01, fixed α=0.05 took ~20 updates
to catch up → misaligned backtest atoms → val_Sharpe dipped to -20.
Adaptive alpha: α = |error| / (|error| + 0.01), clamped [0.01, 0.5].
Tracks fast on Q-shifts (α→0.5), smooths when stable (α→0.01).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fixed v_range from hyperparams (reward_scale/gamma=±240) was 4800x too
wide for Q-values near 0. All Q-values mapped to the same 2-3 center
atoms, making backtest action selection essentially random → val_Sharpe
jumped between -11 and +0.14 arbitrarily. Now uses the same
EMA-smoothed eval_v_range that the experience collector uses.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
reduce_current_q_stats returned PREVIOUS call's results via async
double-buffer. When the eval_v_range EMA converged, stale readback
created a limit cycle: Q-mean alternated between exactly -0.0190 and
-0.2102 for 20+ epochs. Now uses synchronous cuStreamSynchronize +
cuMemcpyDtoH (5µs cost every 50 steps). Removes dead
flush_q_stats_readback and q_stats_ready field.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Hard argmax picks actions based on noise when Q-values are flat
(all near zero). This causes policy collapse — the model locks into
a single action pattern after 1 epoch (Trades=56647 locked, action
distribution frozen). Expected SARSA-style softmax mixture:
p_target(z) = Σ_a softmax(E[Q(s',a)])[a] * p_target_a(z)
When Q-values differentiate, converges to argmax. When flat, averages
across actions → stable gradient signal → breaks policy collapse.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
update_eval_v_range snapped to current Q-stats every 50 steps. With
per-sample IQL support (the circular dependency fix), Q-stats fluctuate
between steps, causing the backtest atom placement to jump → val_Sharpe
oscillated between -42 and 0. EMA (α=0.05) smooths the transition,
giving the backtest consistent atoms across evaluations.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
compute_expected_q used eval_v_range to compute Q-values, then Q-stats
updated eval_v_range from those Q-values — circular dependency that
amplified any drift. Q-mean collapsed 0 → -0.69 in 10 epochs despite
mean_reward ≈ 0. Now uses per_sample_support from IQL (independently
trained V(s)), breaking the feedback loop.
Also removes 127 bf16 identity wrapper calls across 5 CUDA kernels
(c51_loss, epsilon_greedy, dqn_utility, experience, attention_backward).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
