Architectural finding during Phase 3c implementation attempt:
Original spec: α adds `W[a] * state_121[b]` to scalar Q values. Adam-
trained W via dloss/dQ gradient.
Actual codebase: C51 distributional Q-learning. Q is computed from a
probability distribution over fixed atom positions; C51 loss is KL
divergence over distributions, not MSE over scalars. Adding a constant
to logits is softmax-shift-invariant; adding to scalar expected_Q
happens post-loss-path. Either way, the KL loss is invariant under
scalar Q bias → W gradient = 0 → W never trains → α is mathematically
ineffective as originally specified.
Principled fix
──────────────
Per-(b, a) atom-position shift:
z_n_effective[b, a, n] = z_n_base + W[a] * state_121[b] for all n
Shifts atom positions state-dependently. C51 Bellman projection
re-projects target onto shifted positions → loss depends on
`W[a] * state_121[b]` smoothly → W gradient flows from projection
arithmetic.
Implementation cost grows from ~25-35 hr to ~40-60 hr because the
atom-shift threads through every kernel that uses atom positions
for the direction branch:
- compute_expected_q (action selection)
- c51_loss_kernel (Bellman projection)
- c51_grad_kernel (backward dW + dstate from projection arithmetic)
- mag_concat_qdir (magnitude-branch conditioning Q_dir compute)
- quantile_q_select / iqn_dual_head_kernel (investigate)
The Phase A `aux_to_q_dir_bias_kernel.cu` + `aux_to_q_dir_bias_backward_kernel.cu` become architecturally redundant under the revised
design. The dW + dstate gradient computations integrate into
c51_grad_kernel's existing projection backward. Phase A kernels
stay as compiled cubins (committed in 464bc5f7a); future cleanup
commit may delete them.
Initialization: structural prior `W_init = [-0.5, 0.0, +0.5, 0.0]`
reflects domain belief that aux conviction × position-sign biases
toward aligned trades. Adam refines from there.
state_121 ∈ [-1, +1] = recentered aux p_up. When +1 (aux up):
Q_short atoms shift DOWN by 0.5 → action selection avoids Short
Q_long atoms shift UP by 0.5 → action selection biases Long
Hold / Flat unchanged
Adam-vs-fixed-W: the atom-shift design subsumes the fixed-W case
as a config choice (set Adam LR for W = 0). Per user directive
"no shortcuts", the full version is the canonical Phase 3-final
spec.
β + SP11 controller + A2 eval-side + telemetry unchanged from
prior spec sections — those parts of Phase 3 are architecturally
sound. Only α needed correction.
Files touched
─────────────
- docs/plans/2026-05-12-sp22-h6-phase3-alpha-beta.md:
Replaced "α — post-encoder bypass-head" section with
"α — atom-position shift (architecturally revised 2026-05-13)".
Documents the C51 incompatibility, the per-(b, a) atom-shift
fix, the structural-prior initialization, the kernel inventory
for atom-shift threading, the Adam wireup, and the gradient
routing decision (option (i) unchanged).
- docs/dqn-wire-up-audit.md:
"Phase 3c — α architectural finding + spec revision (2026-05-13)"
entry documenting the math obstacle, the principled fix, the
Phase A kernel redundancy under the revised design, and the
scope-revision rationale.
Next session
────────────
Runbook (`docs/plans/2026-05-13-sp22-h6-phase3-alpha-beta-runbook.md`)
α tasks (B9 Steps 4-9, C1) describe the original scalar-bias
implementation and need rewriting for atom-shift threading before
execution can resume.
Refs
────
- pearl_audit_unboundedness_for_implicit_asymmetry (motivates
per-action W signs for the structural prior)
- C51 distributional Q math: Bellemare, Dabney, Munos 2017
- pearl_no_partial_refactor (atom-shift threading is atomic across
all consumer kernels)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
648078ce20