The hardcoded REWARD_SCALE_FLOOR=0.01 was tuned for smoketest (reward_std=0.007)
but 940× too small for production (reward_std=6.57). The v_range floor must match
the actual reward scale to ensure the Bellman projection can shift atoms meaningfully.
- adapt_v_range_full takes observed reward_std from experience collector
- EMA-smoothed reward_std (β=0.99) prevents single-epoch noise from jerking floor
- Falls back to 0.01 before first observation, then adapts automatically
- reward_std_ema field on GpuDqnTrainer, observed_reward_std on DQNTrainer
- Decaying floor uses actual reward scale: floor = R_std * exp(-|Q_mean|/R_std)
903/903 tests passing. Smoketest val_Sharpe positive (60.76 epoch 1).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Q-gap drives v_range contraction: atoms are distributed to maximize action
discrimination, not just cover the Q-range. Target: Q-gap spans 10 atoms
(20% of 52), giving C51 enough resolution for distributional value.
Decaying reward-scale floor: starts at 0.01 (breaks fixed point at init),
decays as exp(-|Q_mean|/0.01) as Q-values mature. Automatically transitions
from "Bellman projection headroom" to "tight atom resolution" without
requiring step counters or epoch awareness.
903/903 tests passing. Q-gap grows 72× (2.7e-6 → 1.93e-4) through epoch 17.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replaces dead Gem 1 micro-reward (used future price = hindsight leak) and
moves OFI-alignment from sparse (trade completion) to dense (every positioned bar).
Uses CURRENT-BAR observable OFI features:
- depth_imbalance (70%): bid vs ask volume ratio from MBP-10
- trade_imbalance (30%): inferred trade direction flow
Rewards Long when order flow says buy pressure, Short when sell pressure.
This gives the direction advantage head a STATE-CONDITIONAL signal:
"Long is better than Short WHEN order book shows buy pressure."
Without this, direction rewards are 50/50 (price is random) and the
advantage gradient cancels via dueling mean-subtraction.
Combined with dense DSR, every positioned bar now gets:
1. DSR: "is holding improving Sharpe?" (consistency signal)
2. OFI-alignment: "does order flow confirm your direction?" (microstructure signal)
3. Inventory penalty: "holding costs money" (urgency signal)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
NoisyLinear was completely dead after cuBLAS migration — neither training
nor experience collection applied ANY exploration noise. With identical
Q-values across actions, Boltzmann selection was uniform, making the
dueling mean-subtraction cancel ALL advantage gradients. The advantage
heads could NEVER learn (Q-gap permanently 0.0000).
Fix: add_advantage_noise CUDA kernel injects per-action Gaussian noise
into Q-values AFTER compute_expected_q, BEFORE action selection.
- Philox PRNG + Box-Muller for GPU-native Gaussian sampling
- Per-sample, per-action noise (breaks within-batch symmetry)
- Only during experience collection (not training targets)
- noise_sigma=0.1 (configurable, TOML + hyperopt tunable)
The noise creates non-uniform Boltzmann selection → different actions
have different frequencies → advantage gradient survives the dueling
mean-subtraction → Q-gap can grow.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Q-gap = max_a Q(s,a) - mean_a Q(s,a): measures action preference per state.
Q-var = Var_s[Q(s,a)]: measures state differentiation across the batch.
Both are 0.0000 through all 10 smoketest epochs despite Sharpe 8+ and
Q-values growing to 0.012. This proves the good metrics are from
reward-induced mechanical bias, NOT learned Q-values.
Also: c51_warmup_epochs=0 (C51 from step 1, bypass MSE dead zone),
smoketest lr=1e-4 (50× higher, makes 40 steps representative).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sparse DSR (trade completion only) left 80% of bars with near-zero reward.
Q-values couldn't grow. Dense DSR fires EVERY bar when positioned:
- r_bar = vol-normalized unrealized PnL per bar
- EMA statistics update only from positioned bars (flat excluded)
- DSR reward = how much this bar's PnL improves the running Sharpe
- Flat bars get zero DSR — safe default, no penalty for not trading
Naturally adaptive: high-vol → variance grows → DSR shrinks per unit
return → model becomes conservative. Low-vol → DSR amplifies.
Smoketest: WinRate 52-55% (was 44%), Sharpe peaks at 11.53, PF 1.67.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
With per-component clipping removed, raw gradient norms are ~4000.
The old gradient_clip_norm=1.0 (calibrated for pre-clipped norms ~7)
discarded 99.97% of gradient magnitude every step, making the effective
learning rate 570× too small. Q-values grew 9× slower than old run.
10000 lets normal gradients (~4000) pass through unclipped. Only fires
on genuine divergence spikes (>2.5× normal). Adam's internal adaptive
step sizing handles the rest.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Per-component budget clipping (C51→4.5, IQN→1.0, CQL→1.0) produced a
CONSTANT gradient norm every step (0.655 old split, 0.756 new split).
Adam's second moment v[i] converged to a constant, making
m_hat/sqrt(v_hat) → sign(m_hat) — Adam degenerated into SignSGD.
The optimizer could not adapt step sizes to loss landscape curvature.
Fix: remove all per-component clips. Loss weights (iqn_lambda=0.25,
CQL coefficient, ensemble weight) control component balance. Single
global clip inside Adam kernel is the sole safety mechanism.
Saves 9 kernel launches per training step:
- 3× grad_norm phase1+phase2 removed (IQN, CQL, ensemble)
- 3× clipped_saxpy → plain saxpy_f32 (no norm computation needed)
- 1× primary budget clip_grad_buf_inplace removed
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause of DNS deadlock (2026-03-18, 2026-04-12): node-bootstrap
DaemonSet overwrote /etc/resolv.conf with ONLY nameserver 10.32.0.10
(cluster DNS). When nodes rebuild, CoreDNS can't pull its image because
containerd resolves via 10.32.0.10 which requires CoreDNS to be running.
Fix: resolv.conf now has DUAL nameservers:
nameserver 10.32.0.10 # primary: cluster DNS
nameserver 169.254.169.254 # fallback: Scaleway metadata DNS
Containerd tries kube-dns first (fast, resolves .svc.cluster.local),
falls back to VPC resolver when kube-dns is unreachable. Breaks the
chicken-and-egg permanently.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fundamental fix: C51 atom spacing (dz=0.6) was larger than the Q-value
range (0.02), making the distributional loss unable to resolve rewards.
The network was blind to its own learning signal.
Adaptive v_range via device buffer (same pattern as tau_buf):
- v_range_buf[2] on GPU, all C51/MSE/CQL/expected_q kernels read via
pointer (graph captures stable address, reads value at replay time)
- Cold start: v_range=±1.0 → dz=2/51=0.039 → 25× atom resolution
- Monotonic expansion based on Q-stats every 50 training steps
- Never shrinks (avoids invalidating Bellman targets in replay buffer)
Kernel changes (5 files):
- c51_loss_kernel.cu, mse_loss_kernel.cu, mse_grad_kernel.cu,
cql_grad_kernel.cu, compute_expected_q: scalar v_min/v_max → pointer
Also fixed:
- PopArt warmup 100→1 (start normalizing from step 1)
- PopArt variance floor 0.0001→1e-8 (allow sigma < 0.01)
- Backtest evaluator: allocate own v_range_buf for compute_expected_q
- num_atoms 51→52 in all TOMLs (TF32 4-element alignment)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
v_min/v_max ±15 with gamma=0.99 only covers 15% of theoretical Q range
(Q_max = 1/(1-0.99) = 100 for unit-variance PopArt rewards). C51 top atom
saturates on sustained winners, degrading distributional learning. ±50
covers 95th percentile.
PopArt GPU buffers (popart_mean, popart_var, popart_count) were never
zeroed between walk-forward folds — fold 2's reward normalization was
contaminated by fold 1's statistics. Now reset alongside Adam state in
reset_adam_state().
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Flat+epsilon can hit any of the 9 dir×mag combos. Reporting against 7
(old reachable-without-epsilon count) showed 9/7 which is confusing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The monitoring summary layout was shifted by 2 positions:
- Kernel writes: summary[5..14]=exp[9], [14..17]=order[3], [17..20]=urgency[3]
- Rust read (old): raw[5..12]=exp[7], raw[12..15]=order, raw[15..18]=urgency
- Rust read (new): raw[5..14]=exp[9], raw[14..17]=order, raw[17..20]=urgency
When num_actions changed 7→9, exp[7] and exp[8] became non-zero but Rust
read them as order_counts — the "order=3/3" was actually exp[7]+exp[8]+order[0].
Also fixed:
- MonitoringSummary.action_counts: [usize; 7] → [usize; 9]
- All downstream: monitoring.rs, metrics.rs, financials.rs, training_loop.rs
- Flat cell indices 3→{3,4,5} in diversity threshold
- exposure_names array expanded to 9 entries
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
VarStore elimination completion:
- xavier_init_params_buf(): CPU Xavier init directly into flat params_buf + target_params_buf
using compute_param_sizes() for correct bottleneck dims (s1_input_dim=54, not state_dim=80)
- Pad all cuBLAS-accessed flat buffers (grad/m/v/cql/vaccine + IQN + evaluator)
with cutlass_tile_pad to prevent SGEMM 32-element M-tile overreads
- Fix forward+backward GEMM for layer s1: use s1_input_dim/s1_ldb when bottleneck
active (was state_dim=80, writing 80-wide rows into 54-wide bn_d_concat_buf)
- Pad bn_concat_buf/bn_d_concat_buf for K-tile overreads (concat_dim=54 not multiple of 32)
- Vaccine gradient: eliminate 2× DtoD copies with std::mem::swap on CudaSlice structs
- Fix IQN Xavier init to include branch_3 (was [b0,b1,b2], now [b0,b1,b2,b3])
- Pad backtest evaluator buffers (params_flat, activations, logits)
Verified: compute-sanitizer --tool memcheck → 0 errors, 286+57 tests pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add to_varstore() compatibility shims on DuelingQNetwork and
DistributionalDuelingQNetwork so test/example code can rebuild a
GpuVarStore snapshot when needed. Delete dead tests that referenced
removed DQNAgent, PrioritizedReplayBuffer, and ReplayBufferType.
Fix action index references (action_19/21 -> action_28/30) and
type annotation issues (sin ambiguity, remainder operator).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
DuelingWeightSet/BranchingWeightSet now created via from_flat_buffer()
pointing directly into GpuDqnTrainer's params_buf. No intermediate
VarStore extraction, no D2D copies, no flatten/unflatten cycle for
the primary weight sets.
bottleneck_dim=16 now works — smoke test passes. The dimension mismatch
is resolved because params_buf is allocated at the correct reduced
dimension, and weight sets point directly into it.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Added OwnedGpuLinear to ml-core (self-contained linear layer with owned
weight/bias CudaSlice, forward_with_slices bypasses GpuVarStore lookup)
- Removed GpuVarStore from all 12 ml-dqn modules: Sequential, branching,
distributional_dueling, dueling, network, rmsnorm, residual, curiosity,
attention, iql, quantile_regression, regime_conditional
- Removed get_q_network_vars/vars/store accessors from DQN, branching,
distributional_dueling, dueling, network, quantile_regression
- Replaced GpuLinear+GpuVarStore with OwnedGpuLinear in all cold-path modules
- Made load_from_safetensors a no-op (flat buffer path handles checkpoints)
- Made sync_to_varmap a no-op (flat buffer is source of truth)
- Moved branching->VarStore conversion to gpu_weights::branching_to_varstore
- Re-enabled bottleneck_dim=16 in config defaults + all 3 TOML configs
- Removed flatten_online_weights bottleneck skip workaround
- Zero GpuVarStore references in crates/ml-dqn/src + crates/ml/src/trainers/dqn
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
DuelingWeightSet and BranchingWeightSet fields changed from owned
CudaSlice<f32> to raw u64 device pointers + usize element counts.
This enables zero-copy views directly into the flat params_buf for
the training hot path (no D2D copies, no shadow allocations).
Key changes:
- Weight sets store u64 + usize pairs instead of CudaSlice<f32>
- from_flat_buffer() creates zero-copy views into params_buf
- from_slices() creates pointer views from owned CudaSlice allocations
- DuelingWeightBacking/BranchingWeightBacking hold owned CudaSlice
arrays for callers that need independent allocations (experience
collector, ensemble heads, tests)
- flatten/unflatten become no-ops when weight sets point into params_buf
- extract functions return (Backing, WeightSet) tuples
- KernelWeightPack::build() uses u64 fields directly (no raw_device_ptr)
- All sync functions updated for pointer-based signatures
- Ensemble clone functions return backing + pointer view pairs
- gradient_budget tests updated (7/7 pass)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
DuelingWeightSet becomes raw u64 pointers into params_buf, not separate
CudaSlice allocations. Eliminates ALL D2D copies between weight
representations. flatten/unflatten cycle deleted entirely.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
VarStore is a CPU-trip corrupting data + degrading performance. Must be
replaced with direct pointer buffers (flat params_buf is already the
source of truth). Checkpoint, weight extraction, and polyak updates
all need porting to direct CudaSlice operations.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
bottleneck_dim=16 causes w_s1 size mismatch between VarStore (full state_dim)
and flat params_buf (reduced bottleneck_concat_dim). EMA target sync crashes
with CUDA_ERROR_ILLEGAL_ADDRESS.
Root cause: GpuVarStore allocates weights at full state_dim but GpuDqnTrainer's
flat buffer uses bottleneck-reduced dimensions. These two weight sources are
fundamentally incompatible when bottleneck is active.
Fix: eliminate GpuVarStore entirely (next task), then re-enable bottleneck.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Launch ensemble_aggregate_kernel per training step to populate
ensemble_var_q_buf with per-atom C51 logit variance across K heads.
Expose via ensemble_var_ptr() on FusedTrainingContext. Add
ensemble_exploration_beta config (default 0.2). Wire into
set_ensemble_variance_bonus() on GpuExperienceCollector which computes
beta * sqrt(mean_var) and tiles it as a constant SAXPY bonus over all
Q-values before Boltzmann action selection. No-op when ensemble_count <= 1.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
IQR (Q75-Q25) from IQN's 32 quantile estimates is computed per-action
after each training step and added to Q-values before Boltzmann action
selection. High-IQR actions (uncertain outcomes, typically large-magnitude
positions) get an exploration boost, countering C51's structural bias
toward low-variance Small actions that causes magnitude collapse.
Pipeline: IQN train -> compute_iqr() -> set_iqr_bonus() -> SAXPY on
q_values [N, q_stride] each experience timestep. Alpha default 0.3.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Even timesteps use the existing directional mirror (Short<->Long, negated
reward). Odd timesteps substitute a different magnitude and linearly scale
the reward by alt_mag_frac / actual_mag_frac, giving the magnitude branch
direct gradient signal about what bigger or smaller positions would have
earned. Falls back to directional mirror when reward is near-zero (scaling
uninformative). No extra buffer needed -- reuses the existing N*L
counterfactual slot.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Set c51_warmup_epochs default to 0. The ramp logic in training_loop.rs
already returns alpha_max immediately when c51_warmup_epochs==0, so all
epochs now see c51_alpha=0.5 from the first step. Stable loss landscape,
no CUDA graph recaptures from alpha transitions.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ShortSmall=0, ShortHalf=1, ShortFull=2, Flat=3, LongSmall=4, LongHalf=5,
LongFull=6. Updates all array sizes, bounds checks, names, and distribution
logic across gpu_monitoring, monitoring, training_loop, metrics, and financials.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Task 3: Enable PopArt reward normalization (default true, all 3 TOML configs,
wiring confirmed in fused_training.rs submit_forward_ops_main()).
Task 4: Couple PopArt variance with tau reset — add prev_popart_var to
FusedTrainingCtx, read_popart_var() to GpuDqnTrainer, read_popart_variance()
+ should_reset_tau() to FusedTrainingCtx, tau reset injected at epoch boundary
in training_loop.rs after log_phase_timing().
Task 5: Add best_sharpe/best_epoch/best_val_loss reset to reset_for_fold() so
each walk-forward fold competes independently. Also wire v8 reward fields
(popart_enabled, micro_reward_scale, td_lambda, max_trace_length,
hindsight_fraction, hindsight_lookahead) through training_profile.rs apply_to().
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The 4-branch DQN (direction x magnitude) had 3 degenerate variants
(Short25, Flat, Long25) that all mapped to 0.0 target exposure when
direction=Flat, causing 82% Flat collapse. Collapse these into a
single Flat variant, giving 7 levels (ShortSmall/Half/Full, Flat,
LongSmall/Half/Full) and 63 total factored actions (7x3x3).
- ExposureLevel enum: 9 variants -> 7 (add direction/magnitude/from_dir_mag)
- FactoredAction: 81 -> 63 total actions, from_index/to_index updated
- DQN epsilon-greedy: use from_dir_mag() instead of dir*3+mag indexing
- DQN config: num_actions default 9 -> 7
- PPO action space: 45 -> 63 actions, action masking updated
- Signal adapter CUDA kernel: 5-bin -> 7-bin exposure aggregation
- All tests updated for new variant names and index ranges
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
10-component design to fix 82% Flat/Small action concentration:
1. Collapse degenerate 9→7 level exposure space
2. Per-branch reward decomposition (direction/magnitude/order/urgency)
3. Hindsight action relabeling for magnitude
4. Position-aware temporal exploration with per-branch epsilon
5. IQN quantile spread as anti-collapse signal
6. Activate PopArt + couple with tau reset
7. Feed the gradient vaccine with diverse batches
8. Per-branch bottleneck widths
9. Ensemble variance at inference
10. Clear PER between folds
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fuses GEMM + bias-add + ReLU into a single cublasLtMatmul kernel via
CUBLASLT_EPILOGUE_RELU_BIAS. Eliminates 7 separate add_bias_relu
kernel launches per forward pass (3 trunk + 4 branch hidden layers).
Creates separate cached descriptors with epilogue enabled at init time.
Falls back to separate kernels if the epilogue heuristic isn't available.
Bias pointer set dynamically per-call via set_matmul_desc_attribute.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Performance:
- Cache cuBLAS descriptors: pre-create matmul_desc + layouts + algo at init.
requestedAlgoCount=3 for better algorithm selection. Zero per-GEMM overhead.
- Per-branch workspace: 4 × 32MB separate workspace buffers for multi-stream
branch dispatch. Eliminates workspace contention on parallel execution.
Training stability:
- Remove hardcoded shrink-perturb that fired every epoch on short runs (3-5 epochs).
With epochs=5, interval = epochs/4 = 1 → fired every epoch, destroying epoch 1
learned weights. This caused Sharpe to collapse from +0.60 to -0.29 after epoch 1.
- Phase 3 shrink-perturb now uses config values (was hardcoded alpha=0.9, sigma=0.01).
- The config-defined shrink_perturb_interval=20 now controls all shrink-perturb timing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The backward GEMM heuristic on H100 with CUDA 13.0 requires 32MB
workspace for TF32 algorithm selection. 16MB causes
CUBLAS_STATUS_NOT_SUPPORTED on dW_only (m=256,n=128,k=16384).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
