foxhunt

Files

jgrusewski b21c6d5cff feat: comprehensive C51 training stability overhaul — Q-stats diagnostics, adaptive atoms, gradient safety

Major architectural fixes discovered through systematic investigation:

Q-gap measurement (3 bugs):
- compute_expected_q never ran during training → q_out_buf was zeros
- epoch_q_gap reset in process_epoch_boundary before logging read it
- flush_q_stats_readback drained async readback before in-loop read

Zero-copy pinned memory (4 hot-path scalars):
- t_buf, tau_buf, v_range_buf, adaptive_clip_buf → pinned device-mapped
- GPU reads via cuMemHostGetDevicePointer, host writes directly, no HtoD

Q-stats-driven adaptive v_range:
- v_range = q_mean ± 3σ + Bellman headroom (was fixed ±1.0)
- Adaptive MIN_RANGE scales with |Q_mean| (was fixed 0.02)
- Per-step adaptation (was every 50 steps)
- 100× finer atom resolution from epoch 1

Gradient stability:
- IS-weight clamp at 10.0 in all loss/grad kernels (PER spike prevention)
- 3 power iterations in spectral norm (was 1 — underestimated sigma)
- Bottleneck w_bn added as 13th spectral-normed matrix (was missing)
- Pre-Adam grad_buf clip via clip_grad_buf_inplace (activation amplification)
- EMA-based adaptive gradient clipping (pinned device buffer)
- Consolidated grad_norm to single buffer (was 2 — eliminated grad_norm_f32_buf)

Adaptive tau from online-target Q-divergence:
- C51 loss kernel accumulates (E[Q_online] - E[Q_target])² per batch
- Tau scales with sqrt(divergence/baseline), clamped [0.5×, 10×] base
- Accelerates target convergence during discovery, stabilizes during plateau

Deterministic evaluation:
- eval_mode in action_select kernel: pure greedy argmax, no Boltzmann/RNG
- Eliminated ±40 val_Sharpe noise from near-uniform Boltzmann sampling
- Backtest evaluator uses adaptive v_range (was config v_min/v_max — 1500× mismatch)

Atom utilization metrics:
- compute_expected_q accumulates entropy + utilization per step
- q_stats_kernel extended to 7 outputs (was 5)
- Logged per epoch: atoms=98%ent/92%util

Pessimistic Q-init removed — incompatible with adaptive v_range (bias was
255× outside ±0.01 support, causing 5-epoch cold-start and late Q-value drift).

903/903 tests passing. val_Sharpe positive from epoch 1 with greedy eval.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-04-12 20:10:25 +02:00