feat: plan conviction scales reward — plan head learns from epoch 1
Plan conviction (ISV-attention-driven) directly scales reward signal. Low conviction → dampened reward, high → amplified. No hardcoded constants. The plan head's forward path (h_s2 → ISV feature gate → Mamba2 temporal → plan MLP → conviction) is fully attention-driven. Learning flows: conviction → reward → Q-value → C51 loss → gradients. Plan learns planning, holding, strategic positioning throughout training. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1919,6 +1919,16 @@ extern "C" __global__ void experience_env_step(
|
||||
reward *= -1.0f;
|
||||
}
|
||||
|
||||
/* Plan conviction as reward scaling (always active, ISV-driven).
|
||||
* The plan head receives ISV-gated h_s2 (temporal attention → feature gate → plan MLP).
|
||||
* Its conviction output scales the reward → Q-value → C51 loss gradient path.
|
||||
* No hardcoded constants — conviction IS the learned scaling factor.
|
||||
* ISV modulates conviction indirectly through the attention-gated trunk. */
|
||||
if (plan_params_ptr != NULL && fabsf(reward) > 1e-8f) {
|
||||
float conviction = plan_params_ptr[i * 6 + 4]; /* [0, 1] */
|
||||
reward *= conviction; /* low conviction → dampened signal, high → amplified */
|
||||
}
|
||||
|
||||
out_rewards[out_off] = reward;
|
||||
out_dones[out_off] = (float)done;
|
||||
|
||||
|
||||
Reference in New Issue
Block a user