Components were fighting a broken signal, not broken themselves.
With correct signal: adapt DSR to trade-level, rank only non-zero
rewards, keep commitment as soft signal, let E1 enrichment handle
Q-drift instead of hardcoded kernel penalty.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: training environment differs from validation backtest in 3 ways:
1. 100-bar episodes force exits (val runs continuous) → position-gated done
2. Rank normalization destroys sparse trade-level reward → remove it
3. Different Sharpe computation (per-trade vs per-bar, different annualization)
Single execution path:
Phase 1: Episode alignment (100→5000 bars, position-gated done, soft reset)
Phase 2: Reward alignment (remove rank norm, raw trade P&L to replay)
Phase 3: Metrics alignment (un-annualized per-trade Sharpe, both paths)
Expected: training Sharpe within 2× of validation with same weights.
Removes 5 unnecessary shaping components that fought the broken signal.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Every epoch's validation backtest feeds back into the next epoch:
E1: Q-value reality check (bias correction, replaces drift penalty)
E2: Adaptive epsilon (performance-driven, replaces schedule)
E3: Dynamic gamma (from trade duration, replaces manual annealing)
E4: Trade autopsy (per-branch LR scaling from error rates)
E5: Ensemble agreement tuning (auto-tune epistemic gate)
E6: Winner distillation (boost top 10% trades in replay)
E7: Hindsight optimal labels (correct actions for losers)
E8: Curriculum weights (oversample failure regimes)
E9: State confidence (tradability scores from eval)
Zero hardcoded schedules. The model adapts from its own performance.
~215 lines Rust, no new CUDA kernels. ~15s overhead per epoch.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two fundamental fixes for training Sharpe breakthrough:
1. Trade-level rewards: replace per-bar noise (SNR=0.01) with trade
P&L attribution (SNR=0.1-0.5). C51 atoms model trade outcome
distributions, not random walk noise.
2. Exploration risk budget: protection stack (CVaR, epistemic gate,
commitment, DSR) scaled by iqn_readiness². Loose during exploration,
tight when converged. Model can discover edges before being punished.
Also: homeostatic regularization spec (unified adaptive penalties).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Primary goal: val/OOS Sharpe gap < 15%. If val_Sharpe drops to 25
post-generalization, OOS should be > 21. If val drops to 15, OOS > 13.
OOS Sharpe > 10 sustained as secondary target.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Walk-forward: strictly chronological, no future leakage.
Weight decay mask: indices 0-7 (trunk + value head), not just trunk.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Task 9d: Q-mean EMA drift penalty (lambda=0.01) in C51 grad kernel.
Prevents +0.47 Q-mean drift observed in train-vpb4w.
Task 9e: 500-step LR warmup for new tensor groups (regime gate,
adaptive atoms, Mamba2, multi-horizon). Prevents gradient shock
on new components introduced to trained network.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Task 9a: Adaptive atom position training via atom entropy gradient.
Numerical finite-difference gradient on spacing_raw, SGD at LR=1e-3.
Task 9b: Mamba2 BPTT backward through K=8 scan steps.
d_W_A/B/C accumulated, separate Adam at LR=1e-4.
Task 9c: Multi-horizon C51 loss — 3 n-step passes (1/5/20 bar),
3 C51 loss launches, regime-weighted gradient blend (ADX-based).
Fixes the "untrained component" gap identified in plan review.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Observed train-vpb4w freeze at epoch 23 (val_Sharpe=51.11):
- avg_grad=10.0 (nonzero!) but Q-stats FROZEN for 4+ epochs
- Root cause: Adam momentum converged to fixed point, not zero gradient
Component 10: Targeted Adam momentum reset for branch weights only
(indices 8-41). Lighter than full rewind, preserves trunk convergence.
Component 11: Q-gap target attractor for spread gradient. Amplifies
spread when Q-gap is below target, reduces when above. Prevents
spread from being too weak at high Q-gap levels.
Component 12: Cosine LR schedule with warm restarts (T=20 epochs).
LR decay prevents overshoot, restart jumps break fixed points.
Half-momentum at restart preserves gradient direction.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Task 10: q_mean_reduce + q_mean_subtract two-phase kernel. Zero params.
Fixes Q-mean drift 0→+0.47 from bootstrapping bias.
Task 11: Adaptive entropy_coeff based on utilization_ema. Zero params.
Fixes atom utilization collapse 100%→20%.
Execution order: 10, 11, then 1-9.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Observed in train-vpb4w (live H100 run):
- Q-mean drifts 0→+0.47 over 18 epochs (bootstrapping bias)
- Atom utilization drops 100%→20% (distributional collapse)
- val_Sharpe peaks 54.59 then decays to 31.52
Component 8: Mean-center Q-values before action selection pipeline.
Zero params, preserves ranking, prevents drift accumulation.
Component 9: Adaptive atom entropy regularization. Ramps entropy_coeff
when utilization drops below 50%. Uses existing c51_grad_kernel param.
Both are zero-param fixes moved to top of implementation order.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Checkpoint ring buffer ranked by improvement rate, plateau detection
via liquid tau velocity, informed perturbation cycle (Adam reset,
shrink-perturb, temp boost, LR×2), depth-limited rewind. ~150 lines
in training_loop.rs, zero CUDA changes. Recovers from plateaus that
prevention mechanisms (spread gradient, liquid tau) fail to avoid.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Raw E[Q] ~ 0.1 while h_s2 ~ 1.0 post-ReLU. Without normalization,
Q_dir columns learn 10x slower. Dividing by delta_z scales to ~1-10.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Magnitude FC input changes from h_s2[B, SH2] to [h_s2; Q_dir][B, SH2+3].
Direction E[Q] values (3 scalars per sample) are computed from branch 0
logits and concatenated. w_b1fc grows by 3*adv_h parameters (768 for AH=256).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
6 new CUDA kernels, C51/grad/epsilon kernel mods, dual-tau IQL,
v_range dead code deletion, per-sample everything.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Covers: PER advantage-weighted staleness, per-sample C51 atom
support centered on V(s), per-branch advantage decomposition,
dual-tau expectile gap exploration, and v_range dead code removal.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fundamental fix: C51 atom spacing (dz=0.6) was larger than the Q-value
range (0.02), making the distributional loss unable to resolve rewards.
The network was blind to its own learning signal.
Adaptive v_range via device buffer (same pattern as tau_buf):
- v_range_buf[2] on GPU, all C51/MSE/CQL/expected_q kernels read via
pointer (graph captures stable address, reads value at replay time)
- Cold start: v_range=±1.0 → dz=2/51=0.039 → 25× atom resolution
- Monotonic expansion based on Q-stats every 50 training steps
- Never shrinks (avoids invalidating Bellman targets in replay buffer)
Kernel changes (5 files):
- c51_loss_kernel.cu, mse_loss_kernel.cu, mse_grad_kernel.cu,
cql_grad_kernel.cu, compute_expected_q: scalar v_min/v_max → pointer
Also fixed:
- PopArt warmup 100→1 (start normalizing from step 1)
- PopArt variance floor 0.0001→1e-8 (allow sigma < 0.01)
- Backtest evaluator: allocate own v_range_buf for compute_expected_q
- num_atoms 51→52 in all TOMLs (TF32 4-element alignment)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
DuelingWeightSet becomes raw u64 pointers into params_buf, not separate
CudaSlice allocations. Eliminates ALL D2D copies between weight
representations. flatten/unflatten cycle deleted entirely.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
VarStore is a CPU-trip corrupting data + degrading performance. Must be
replaced with direct pointer buffers (flat params_buf is already the
source of truth). Checkpoint, weight extraction, and polyak updates
all need porting to direct CudaSlice operations.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
