fix: gate conviction reward scaling by readiness — was killing gradient signal
The plan head's conviction output (sigmoid, [0,1]) scales rewards. At init, conviction ≈ 0.1 (Xavier random weights through sigmoid), which multiplied ALL rewards by 0.1 — killing the gradient signal. The model couldn't learn meaningful Q-values, defaulted to uniform random action selection → 1.7M trades on 4M bars → val_Sharpe -1000. Fix: only apply conviction scaling when readiness ≥ 0.5 (plan head mature). Before that, conviction defaults to 1.0 (identity). This matches the existing readiness gate on position sizing (line 1416). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1922,14 +1922,16 @@ extern "C" __global__ void experience_env_step(
|
||||
reward -= opportunity_cost;
|
||||
}
|
||||
|
||||
/* Plan conviction as reward scaling (always active, ISV-driven).
|
||||
* The plan head receives ISV-gated h_s2 (temporal attention → feature gate → plan MLP).
|
||||
* Its conviction output scales the reward → Q-value → C51 loss gradient path.
|
||||
* No hardcoded constants — conviction IS the learned scaling factor.
|
||||
* ISV modulates conviction indirectly through the attention-gated trunk. */
|
||||
/* Plan conviction as reward scaling — ONLY when plan head is mature.
|
||||
* Before readiness ≥ 0.5, conviction defaults to 1.0 (identity) so the
|
||||
* reward signal flows undampened. Without this gate, random init conviction
|
||||
* (~0.1) kills the gradient signal and prevents the model from learning. */
|
||||
if (plan_params_ptr != NULL && fabsf(reward) > 1e-8f) {
|
||||
float conviction = plan_params_ptr[i * 6 + 4]; /* [0, 1] */
|
||||
reward *= conviction; /* low conviction → dampened signal, high → amplified */
|
||||
float readiness_val = (readiness_ptr != NULL) ? readiness_ptr[0] : 0.0f;
|
||||
float conviction = (readiness_val >= 0.5f)
|
||||
? plan_params_ptr[i * 6 + 4] /* [0, 1] from plan head */
|
||||
: 1.0f; /* identity until plan matures */
|
||||
reward *= conviction;
|
||||
}
|
||||
|
||||
out_rewards[out_off] = reward;
|
||||
|
||||
Reference in New Issue
Block a user