fix(dqn): unstick eval Kelly cap — health-coupled warmup_floor never collapses to zero
The eval-mode policy was producing diverse Boltzmann picks (verified by the
new `val_dir_dist` HEALTH_DIAG line) but every active-direction pick (Long /
Short) was collapsing to `actual_dir = Flat` because the Kelly cap forced
`target_position = 0` at cold start.
Cluster run `train-multi-seed-ddrpr` epoch 0 made this unambiguous:
val_dir_dist [short=0.0000 hold=0.1953 long=0.0001 flat=0.8047]
Boltzmann fired correctly (sum hold + flat ≈ 100% of bars, with Hold ~ 20% =
the share of bars where the policy explicitly picked Hold; the other 80%
were active-direction picks all rerouted to Flat by `target_position = 0`).
Root cause in `trade_physics.cuh::kelly_position_cap`:
warmup_floor = clamp(conviction, 0, 1) // ← can hit 0
effective_kelly = maturity*kelly_f + (1-maturity)*warmup_floor
= 0 + 1*0 = 0 at cold start with low conviction
cap = effective_kelly * max_position * safety = 0 → no exposure permitted
The `safety_multiplier` was already protected by a `health_safety = 0.5 + 0.5×h`
floor, but `warmup_floor` had no such floor. Catch-22: low conviction → cap=0 →
no trades → Kelly stats stay cold → conviction stays low → forever.
The same bootstrap-deadlock pattern as the IQN trunk SAXPY readiness gate
(commit f86353840), and the fix is structurally identical — apply a non-zero
adaptive floor sourced from the same training-stability signal:
warmup_floor = max(conviction, health_floor)
where `health_floor = 0.5 + 0.5 × ISV[LEARNING_HEALTH]` is the same value the
caller already computes for `safety_multiplier`. Both signals are adaptive
and ISV-driven; no tuned constants. The floor only matters during cold start
— once `maturity → 1` after ≥10 trades the term drops out entirely.
Threaded through both `apply_kelly_cap` and `kelly_position_cap` signatures;
single caller in `unified_env_step_core` passes `health_safety` as the new
arg (already locally computed two lines above). Build clean at 11-warning
baseline.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -269,7 +269,8 @@ __device__ __forceinline__ float kelly_position_cap(
|
|||||||
float sum_losses,
|
float sum_losses,
|
||||||
float max_position,
|
float max_position,
|
||||||
float safety_multiplier,
|
float safety_multiplier,
|
||||||
float conviction
|
float conviction,
|
||||||
|
float health_floor
|
||||||
) {
|
) {
|
||||||
const float prior_wins = 2.0f;
|
const float prior_wins = 2.0f;
|
||||||
const float prior_losses = 2.0f;
|
const float prior_losses = 2.0f;
|
||||||
@@ -316,7 +317,22 @@ __device__ __forceinline__ float kelly_position_cap(
|
|||||||
* doesn't swing wildly on small samples. */
|
* doesn't swing wildly on small samples. */
|
||||||
float total_trades = win_count + loss_count;
|
float total_trades = win_count + loss_count;
|
||||||
float maturity = fminf(1.0f, total_trades * 0.1f);
|
float maturity = fminf(1.0f, total_trades * 0.1f);
|
||||||
float warmup_floor = fminf(1.0f, fmaxf(0.0f, conviction));
|
/* warmup_floor must never collapse to zero: at cold start (maturity=0) the
|
||||||
|
* effective Kelly cap reduces to `(1 - maturity) * warmup_floor = warmup_floor`,
|
||||||
|
* so a zero floor pins target_position to 0 and the policy can never open a
|
||||||
|
* position to accumulate the trades that would graduate Kelly stats — the
|
||||||
|
* same bootstrap deadlock pattern that gated the IQN trunk SAXPY. The
|
||||||
|
* adaptive floor is therefore the max of (a) per-sample policy conviction
|
||||||
|
* and (b) the caller's training-stability floor (`health_floor` = 0.5 +
|
||||||
|
* 0.5 × health, ∈ [0.5, 1.0], computed from ISV[LEARNING_HEALTH]). When
|
||||||
|
* the policy is confident, conviction wins; when training is healthy but
|
||||||
|
* Q-spread is tight, the health floor wins. Both are adaptive ISV-driven
|
||||||
|
* signals; no tuned constants. The floor only matters during cold start —
|
||||||
|
* once `maturity → 1` (≥10 completed trades) the term drops out entirely. */
|
||||||
|
float warmup_floor = fmaxf(
|
||||||
|
fminf(1.0f, fmaxf(0.0f, conviction)),
|
||||||
|
fminf(1.0f, fmaxf(0.0f, health_floor))
|
||||||
|
);
|
||||||
float effective_kelly = maturity * kelly_f + (1.0f - maturity) * warmup_floor;
|
float effective_kelly = maturity * kelly_f + (1.0f - maturity) * warmup_floor;
|
||||||
|
|
||||||
return effective_kelly * max_position * safety_multiplier;
|
return effective_kelly * max_position * safety_multiplier;
|
||||||
@@ -334,10 +350,12 @@ __device__ __forceinline__ float apply_kelly_cap(
|
|||||||
float sum_losses,
|
float sum_losses,
|
||||||
float max_position,
|
float max_position,
|
||||||
float safety_multiplier,
|
float safety_multiplier,
|
||||||
float conviction
|
float conviction,
|
||||||
|
float health_floor
|
||||||
) {
|
) {
|
||||||
float cap = kelly_position_cap(win_count, loss_count, sum_wins, sum_losses,
|
float cap = kelly_position_cap(win_count, loss_count, sum_wins, sum_losses,
|
||||||
max_position, safety_multiplier, conviction);
|
max_position, safety_multiplier, conviction,
|
||||||
|
health_floor);
|
||||||
return fmaxf(-cap, fminf(cap, target_position));
|
return fmaxf(-cap, fminf(cap, target_position));
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -648,7 +666,8 @@ __device__ __forceinline__ void unified_env_step_core(
|
|||||||
*win_count, *loss_count,
|
*win_count, *loss_count,
|
||||||
*sum_wins, *sum_losses,
|
*sum_wins, *sum_losses,
|
||||||
max_position_physics, safety,
|
max_position_physics, safety,
|
||||||
conviction
|
conviction,
|
||||||
|
health_safety
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -2,6 +2,22 @@
|
|||||||
|
|
||||||
**Status:** Populated during Plan 1 Task 6 (A.5 orphan audit). Updated on every commit per Invariant 7.
|
**Status:** Populated during Plan 1 Task 6 (A.5 orphan audit). Updated on every commit per Invariant 7.
|
||||||
|
|
||||||
|
Kelly cap warmup_floor health-coupled (2026-04-26): `trade_physics.cuh::kelly_position_cap`
|
||||||
|
previously computed `warmup_floor = clamp(conviction, 0, 1)`. At cold start
|
||||||
|
(maturity = 0) `effective_kelly = warmup_floor`, so a zero conviction collapsed
|
||||||
|
the cap to zero — eval-mode could never open a position to graduate Kelly stats,
|
||||||
|
the same bootstrap deadlock pattern the IQN trunk SAXPY exhibited. Confirmed by
|
||||||
|
cluster run `train-multi-seed-ddrpr` epoch 0: new `val_dir_dist [short=0.0000
|
||||||
|
hold=0.1953 long=0.0001 flat=0.8047]` shows Boltzmann fired correctly (hold +
|
||||||
|
flat ≈ 100% of bars) but every active-direction pick (Long/Short) was forced
|
||||||
|
to actual_dir = Flat by `target_position = 0`. Fix: `warmup_floor = max(conviction,
|
||||||
|
health_floor)` where `health_floor = 0.5 + 0.5 × health` is the same training-
|
||||||
|
stability floor already used by `safety_multiplier` (composed in
|
||||||
|
`unified_env_step_core` from `ISV[LEARNING_HEALTH]`). Both signals are adaptive
|
||||||
|
and ISV-driven; the floor only matters during cold start (maturity → 1 after
|
||||||
|
≥10 trades drops the term out entirely). Threaded through both `apply_kelly_cap`
|
||||||
|
and `kelly_position_cap` signatures.
|
||||||
|
|
||||||
Eval-mode action selection unified with Boltzmann softmax (2026-04-26):
|
Eval-mode action selection unified with Boltzmann softmax (2026-04-26):
|
||||||
`experience_kernels.cu` — the four factored action heads (direction, magnitude,
|
`experience_kernels.cu` — the four factored action heads (direction, magnitude,
|
||||||
order, urgency) all dropped their `else if (eval_mode)` strict-argmax + ISV-
|
order, urgency) all dropped their `else if (eval_mode)` strict-argmax + ISV-
|
||||||
|
|||||||
Reference in New Issue
Block a user