From 2c97e0436c4beb2879afe162f771827a7a440417 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Sun, 26 Apr 2026 19:47:20 +0200 Subject: [PATCH] =?UTF-8?q?fix(dqn):=20unstick=20eval=20Kelly=20cap=20?= =?UTF-8?q?=E2=80=94=20health-coupled=20warmup=5Ffloor=20never=20collapses?= =?UTF-8?q?=20to=20zero?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The eval-mode policy was producing diverse Boltzmann picks (verified by the new `val_dir_dist` HEALTH_DIAG line) but every active-direction pick (Long / Short) was collapsing to `actual_dir = Flat` because the Kelly cap forced `target_position = 0` at cold start. Cluster run `train-multi-seed-ddrpr` epoch 0 made this unambiguous: val_dir_dist [short=0.0000 hold=0.1953 long=0.0001 flat=0.8047] Boltzmann fired correctly (sum hold + flat ≈ 100% of bars, with Hold ~ 20% = the share of bars where the policy explicitly picked Hold; the other 80% were active-direction picks all rerouted to Flat by `target_position = 0`). Root cause in `trade_physics.cuh::kelly_position_cap`: warmup_floor = clamp(conviction, 0, 1) // ← can hit 0 effective_kelly = maturity*kelly_f + (1-maturity)*warmup_floor = 0 + 1*0 = 0 at cold start with low conviction cap = effective_kelly * max_position * safety = 0 → no exposure permitted The `safety_multiplier` was already protected by a `health_safety = 0.5 + 0.5×h` floor, but `warmup_floor` had no such floor. Catch-22: low conviction → cap=0 → no trades → Kelly stats stay cold → conviction stays low → forever. The same bootstrap-deadlock pattern as the IQN trunk SAXPY readiness gate (commit f86353840), and the fix is structurally identical — apply a non-zero adaptive floor sourced from the same training-stability signal: warmup_floor = max(conviction, health_floor) where `health_floor = 0.5 + 0.5 × ISV[LEARNING_HEALTH]` is the same value the caller already computes for `safety_multiplier`. Both signals are adaptive and ISV-driven; no tuned constants. The floor only matters during cold start — once `maturity → 1` after ≥10 trades the term drops out entirely. Threaded through both `apply_kelly_cap` and `kelly_position_cap` signatures; single caller in `unified_env_step_core` passes `health_safety` as the new arg (already locally computed two lines above). Build clean at 11-warning baseline. Co-Authored-By: Claude Opus 4.7 (1M context) --- crates/ml/src/cuda_pipeline/trade_physics.cuh | 29 +++++++++++++++---- docs/dqn-wire-up-audit.md | 16 ++++++++++ 2 files changed, 40 insertions(+), 5 deletions(-) diff --git a/crates/ml/src/cuda_pipeline/trade_physics.cuh b/crates/ml/src/cuda_pipeline/trade_physics.cuh index 4742ccd20..acf5c8fd0 100644 --- a/crates/ml/src/cuda_pipeline/trade_physics.cuh +++ b/crates/ml/src/cuda_pipeline/trade_physics.cuh @@ -269,7 +269,8 @@ __device__ __forceinline__ float kelly_position_cap( float sum_losses, float max_position, float safety_multiplier, - float conviction + float conviction, + float health_floor ) { const float prior_wins = 2.0f; const float prior_losses = 2.0f; @@ -316,7 +317,22 @@ __device__ __forceinline__ float kelly_position_cap( * doesn't swing wildly on small samples. */ float total_trades = win_count + loss_count; float maturity = fminf(1.0f, total_trades * 0.1f); - float warmup_floor = fminf(1.0f, fmaxf(0.0f, conviction)); + /* warmup_floor must never collapse to zero: at cold start (maturity=0) the + * effective Kelly cap reduces to `(1 - maturity) * warmup_floor = warmup_floor`, + * so a zero floor pins target_position to 0 and the policy can never open a + * position to accumulate the trades that would graduate Kelly stats — the + * same bootstrap deadlock pattern that gated the IQN trunk SAXPY. The + * adaptive floor is therefore the max of (a) per-sample policy conviction + * and (b) the caller's training-stability floor (`health_floor` = 0.5 + + * 0.5 × health, ∈ [0.5, 1.0], computed from ISV[LEARNING_HEALTH]). When + * the policy is confident, conviction wins; when training is healthy but + * Q-spread is tight, the health floor wins. Both are adaptive ISV-driven + * signals; no tuned constants. The floor only matters during cold start — + * once `maturity → 1` (≥10 completed trades) the term drops out entirely. */ + float warmup_floor = fmaxf( + fminf(1.0f, fmaxf(0.0f, conviction)), + fminf(1.0f, fmaxf(0.0f, health_floor)) + ); float effective_kelly = maturity * kelly_f + (1.0f - maturity) * warmup_floor; return effective_kelly * max_position * safety_multiplier; @@ -334,10 +350,12 @@ __device__ __forceinline__ float apply_kelly_cap( float sum_losses, float max_position, float safety_multiplier, - float conviction + float conviction, + float health_floor ) { float cap = kelly_position_cap(win_count, loss_count, sum_wins, sum_losses, - max_position, safety_multiplier, conviction); + max_position, safety_multiplier, conviction, + health_floor); return fmaxf(-cap, fminf(cap, target_position)); } @@ -648,7 +666,8 @@ __device__ __forceinline__ void unified_env_step_core( *win_count, *loss_count, *sum_wins, *sum_losses, max_position_physics, safety, - conviction + conviction, + health_safety ); } diff --git a/docs/dqn-wire-up-audit.md b/docs/dqn-wire-up-audit.md index 5bd1174c7..9930b999b 100644 --- a/docs/dqn-wire-up-audit.md +++ b/docs/dqn-wire-up-audit.md @@ -2,6 +2,22 @@ **Status:** Populated during Plan 1 Task 6 (A.5 orphan audit). Updated on every commit per Invariant 7. +Kelly cap warmup_floor health-coupled (2026-04-26): `trade_physics.cuh::kelly_position_cap` +previously computed `warmup_floor = clamp(conviction, 0, 1)`. At cold start +(maturity = 0) `effective_kelly = warmup_floor`, so a zero conviction collapsed +the cap to zero — eval-mode could never open a position to graduate Kelly stats, +the same bootstrap deadlock pattern the IQN trunk SAXPY exhibited. Confirmed by +cluster run `train-multi-seed-ddrpr` epoch 0: new `val_dir_dist [short=0.0000 +hold=0.1953 long=0.0001 flat=0.8047]` shows Boltzmann fired correctly (hold + +flat ≈ 100% of bars) but every active-direction pick (Long/Short) was forced +to actual_dir = Flat by `target_position = 0`. Fix: `warmup_floor = max(conviction, +health_floor)` where `health_floor = 0.5 + 0.5 × health` is the same training- +stability floor already used by `safety_multiplier` (composed in +`unified_env_step_core` from `ISV[LEARNING_HEALTH]`). Both signals are adaptive +and ISV-driven; the floor only matters during cold start (maturity → 1 after +≥10 trades drops the term out entirely). Threaded through both `apply_kelly_cap` +and `kelly_position_cap` signatures. + Eval-mode action selection unified with Boltzmann softmax (2026-04-26): `experience_kernels.cu` — the four factored action heads (direction, magnitude, order, urgency) all dropped their `else if (eval_mode)` strict-argmax + ISV-