feat(rl): IQN loss + backward + ensemble action selection

Completes the IQN integration:

- rl_iqn_backward.cu: backprop through quantile embedding + output
  projection to produce per-batch weight gradients for all 4 weight
  tensors. Block per batch, HIDDEN_DIM threads.
- rl_iqn_loss.cu (already existed): quantile Huber loss feeds
  grad_online_q into the backward kernel.
- rl_ensemble_action_value.cu: E_ensemble = α×C51 + (1-α)×IQN,
  ISV-driven α at slot 544.
- Adam updates for IQN weights after backward.
- Ensemble Q feeds rl_q_pi_agree_b diagnostic.

Full stack smoke: 200 steps, l_q=2.43, no crash. Both C51 and IQN
learning simultaneously from the same replay transitions.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-25 21:03:04 +02:00
parent 65d92644d6
commit b219681d01
4 changed files with 290 additions and 1 deletions

View File

@@ -93,6 +93,8 @@ const KERNELS: &[&str] = &[
"rl_session_risk_check", // session-level loss limit circuit breaker
"rl_iqn_forward", // IQN distributional Q-head: quantile embedding + action-value projection; complementary to C51
"rl_iqn_loss", // IQN quantile Huber loss: ρ_τ(δ) = |τ - 1(δ<0)| × Huber(δ, κ=1.0); forward + backward
"rl_iqn_backward", // IQN backward through forward pass: grad_output → grad_w_out/b_out/w_embed/b_embed per-batch scratch
"rl_ensemble_action_value", // C51+IQN ensemble: E_ensemble = α×E_C51 + (1-α)×E_IQN; α from ISV[544]
];
// Cache bust v31 — five new reduce / derive kernels populate the input