gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

fix(rl): B-10 wire G1 Q-distribution launch into step_with_lobsim_gpu

Browse Source 
Initial B-10 commit placed `launch_q_distribution_stats` after the
`dqn_head.forward(h_t)` at line 6615 — but that's inside
`step_with_lobsim`, the legacy non-GPU path. The actual GPU path
exercised by `alpha_rl_train` is `step_with_lobsim_gpu` (line 8402),
which has its own h_t-based Q forward at line 8632. The G1 diag fields
(q_dist_entropy_mean, q_value_range_mean, q_value_abs_max) returned 0
locally because the launch never ran in the GPU path.

Added the same launch_q_distribution_stats call after line 8632.
The non-GPU launch at 6615 is left in place — `step_with_lobsim` is
still callable; per `feedback_no_partial_refactor.md` both paths are
instrumented consistently.

Validated locally (RTX 3050 Ti, 200+50 b=16 fold-1):
  q_dist_entropy_mean       = 2.6482   (was 0)
  q_value_range_mean        = 4.9257   (was 0)
  q_value_abs_max           = 3.8715   (was 0)

The entropy value 2.65 is consistent with a near-uniform softmax over
Q_N_ATOMS=21 atoms with some learned concentration (ln(21)=3.04 = full
uniform, 0 = one-hot delta). Range and abs_max in 4-5 unit scale match
the locally-adapted atom support.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-06-01 12:25:47 +02:00
parent 8c7ce02da9
commit c1dc84a345
1 changed files with 18 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
crates/ml-alpha/src/trainer/integrated.rs
View File

@@ -8631,6 +8631,24 @@ impl IntegratedTrainer {
self.dqn_head
.forward(h_t_borrow, b_size, &mut self.q_logits_d)
.context("step_with_lobsim_gpu: dqn_head.forward(h_t)")?;
// B-10 G1 (2026-06-01): Q-distribution informativeness diagnostic
// on online Q logits — emits ISV slots 730/731/732. Pure
// observability; reads q_logits_d immediately after the forward,
// before any downstream consumer (distill / Thompson / PPO).
// (Sister-launch added at line 6622 in `step_with_lobsim` — the
// GPU path here is the canonical one that the alpha_rl_train
// example actually exercises.)
self.dqn_head
.launch_q_distribution_stats(
&self.isv_dev_ptr,
&self.q_logits_d,
&self.atom_supports_d,
b_size,
&mut self.q_dist_entropy_pb_d,
&mut self.q_dist_range_pb_d,
&mut self.q_dist_abs_max_pb_d,
)
.context("step_with_lobsim_gpu: dqn_head.launch_q_distribution_stats")?;
self.dqn_head
.forward_target(h_t_borrow, b_size, &mut self.q_logits_target_st_d)
.context("step_with_lobsim_gpu: dqn_head.forward_target(h_t) for distillation")?;