bac8fd28ce
Completes the controller-input EMA wiring across all 7 RL
controllers. Previously 6 of 7 EMA input slots received no signal,
freezing those controllers at bootstrap for the entire training run.
Three new derivation kernels populate the last three:
* `rl_kl_approx_b` — Schulman-style per-step KL approximation
`mean(log π_old(a) − log π_new(a))` over batches at the sampled
action. Single-block tree reduce; inputs are `log_pi_old_d`
(recorded at action sample time) and `pi_log_prob_d` (= log
π_new at the same action, output of PPO surrogate forward).
Feeds `kl_pi_ema` (ISV[419]) consumed by rl_ppo_clip.
* `rl_l2_diff_norm` — `‖W_online − W_target‖₂` over the full
DQN weight tensor (24,192 floats = 9 actions × 21 atoms × 128
hidden). Single-block grid-stride loop (256 threads, ~95
iterations each); shared-mem tree-reduce produces a scalar.
Feeds `q_divergence_ema` (ISV[418]) consumed by rl_target_tau.
Launched immediately after `soft_update_target` so the divergence
reflects the post-update gap.
* `rl_step_counter_update` — per-batch trade-duration counter +
done-gated emit. Trainer-owned `steps_since_done_d: [i32; B]`
increments every step and resets on done; on done the counter
value (= event count the position was open) is written to
`trade_duration_emit_d` for `ema_update_on_done` to fold into
`mean_trade_duration_ema` (ISV[417]) consumed by rl_gamma.
Element-wise, one thread per batch index.
## Trainer wiring placement
* trade_duration counter + EMA emit: in `step_with_lobsim`
immediately after `extract_realized_pnl_delta` populates dones_d,
BEFORE the controllers fire — γ adapts THIS step from a real
duration observation.
* q_divergence reduce + EMA: in `step_with_lobsim` immediately
after `dqn_head.soft_update_target` runs, so the divergence
captures the post-update gap. One step lag for the τ controller
(controllers fired earlier in step_with_lobsim, before
step_synthetic).
* kl_pi reduce + EMA: in `step_synthetic` end-of-function block
alongside entropy + td_kurtosis updates. Deferred to AFTER the
encoder backward so the `&self.perception` borrow held by
`h_t_borrow` releases before the `&mut self` launches. One
step lag for the ε controller.
## All 7 controller inputs now wired
| ISV slot | Input EMA | Producer |
|----------|-----------|----------|
| 417 | mean_trade_duration | rl_step_counter_update → ema_update_on_done |
| 418 | q_divergence | rl_l2_diff_norm → ema_update_per_step |
| 419 | kl_pi | rl_kl_approx_b → ema_update_per_step |
| 420 | entropy_observed | ema_update_per_step (direct mean on entropy_d) |
| 421 | advantage_var_ratio | rl_var_over_abs_mean_b → ema_update_per_step |
| 422 | td_kurtosis | rl_kurtosis_b → ema_update_per_step |
| 423 | mean_abs_pnl | abs_copy + ema_update_on_done (existing) |
Combined with the cold-start gate + replace-directly-on-first-warm
fixes from the prior two commits, all 7 controllers will:
1. Hold at bootstrap until their input EMA receives signal
(cold-start gate prevents migration to clamps during sentinel
input period).
2. Replace prev → target directly on first non-zero observation
(no 60% bootstrap contamination in the first warm step).
3. Wiener-α blend (floored at 0.4) on subsequent steps.
## Phase-prefix comment cleanup
Per directive to stop phase prefixing in code, scrubbed "R9 audit",
"Phase A", "Phase B" markers from comments I added across the
multi-commit fix sequence. The remaining "Phase B: cross-batch
param-grad reducer" in build.rs is a pre-existing comment on the
perception trainer's `reduce_axis0` kernel, unrelated to this work.
## Verified gates (local sm_86)
G1 isv_bootstrap ✅
G3 controllers_emit ✅ (test pre-seeds inputs)
G4 target_soft_update ✅
G6 r7d_per_wiring ✅
R3, R4, smoke ✅
The next cluster smoke at this SHA will produce a diag.jsonl where
ALL 7 controller-input EMAs evolve over the 1000 steps, and ALL 7
controllers visibly adapt — the first time the integrated trainer
has every adaptive controller wired since the rebuild plan was
written.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
47 lines
1.6 KiB
Plaintext
47 lines
1.6 KiB
Plaintext
// rl_l2_diff_norm.cu — `‖a − b‖₂ = sqrt(Σ(a−b)²)` over n floats → scalar.
|
||
//
|
||
// The rl_target_tau controller consumes `q_divergence_ema` (ISV[418])
|
||
// to track how far the Q target network has drifted from the Q online
|
||
// network. The controller adapts τ (Polyak soft-update fraction): high
|
||
// divergence → raise τ so the target tracks faster; low → shrink τ
|
||
// for bootstrap stability.
|
||
//
|
||
// Inputs are the trainer's `dqn_head.w_d` (online weights) and
|
||
// `dqn_head.target_w_d` (target weights), each of length
|
||
// `N_ACTIONS × Q_N_ATOMS × HIDDEN_DIM = 9 × 21 × 128 = 24192` floats.
|
||
// Single-block grid-stride loop: 256 threads × ~95 iterations per
|
||
// thread covers the full weight array without atomics (per
|
||
// `feedback_no_atomicadd`) and without multi-block coordination.
|
||
|
||
extern "C" __global__ void rl_l2_diff_norm(
|
||
const float* __restrict__ a,
|
||
const float* __restrict__ b,
|
||
float* __restrict__ out,
|
||
int n
|
||
) {
|
||
extern __shared__ float s[];
|
||
const int tid = threadIdx.x;
|
||
|
||
// Grid-stride loop: each thread accumulates (a[i]-b[i])² for its
|
||
// own subset of elements, then we tree-reduce within the block.
|
||
float acc = 0.0f;
|
||
for (int i = tid; i < n; i += blockDim.x) {
|
||
const float d = a[i] - b[i];
|
||
acc += d * d;
|
||
}
|
||
s[tid] = acc;
|
||
__syncthreads();
|
||
|
||
// Tree reduction across blockDim.x threads (must be power of 2).
|
||
for (int stride = blockDim.x / 2; stride > 0; stride >>= 1) {
|
||
if (tid < stride) {
|
||
s[tid] += s[tid + stride];
|
||
}
|
||
__syncthreads();
|
||
}
|
||
|
||
if (tid == 0) {
|
||
out[0] = sqrtf(s[0]);
|
||
}
|
||
}
|