feat: plan conviction scales reward — plan head learns from epoch 1

Plan conviction (ISV-attention-driven) directly scales reward signal.
Low conviction → dampened reward, high → amplified. No hardcoded
constants. The plan head's forward path (h_s2 → ISV feature gate →
Mamba2 temporal → plan MLP → conviction) is fully attention-driven.
Learning flows: conviction → reward → Q-value → C51 loss → gradients.
Plan learns planning, holding, strategic positioning throughout training.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-04-17 08:30:12 +02:00
parent 4a75abdcbe
commit 9f896878e0

View File

@@ -1919,6 +1919,16 @@ extern "C" __global__ void experience_env_step(
reward *= -1.0f;
}
/* Plan conviction as reward scaling (always active, ISV-driven).
* The plan head receives ISV-gated h_s2 (temporal attention → feature gate → plan MLP).
* Its conviction output scales the reward → Q-value → C51 loss gradient path.
* No hardcoded constants — conviction IS the learned scaling factor.
* ISV modulates conviction indirectly through the attention-gated trunk. */
if (plan_params_ptr != NULL && fabsf(reward) > 1e-8f) {
float conviction = plan_params_ptr[i * 6 + 4]; /* [0, 1] */
reward *= conviction; /* low conviction → dampened signal, high → amplified */
}
out_rewards[out_off] = reward;
out_dones[out_off] = (float)done;