fix(rl): anti-collapse probability floor + argo b_size=16 default

Two fixes for alpha-rl-9k9x6 (commit 3737feb66) findings:

(1) rl_pi_action_kernel.cu — per-action probability floor P_MIN=0.02
    with renorm after softmax. Structural guarantee H(π_sample)
    bounded below regardless of logits. 9k9x6 evidence:
      - π collapsed to action 1 (100% sample rate) by step 2500
      - action_entropy EMA → 0.001 by step 10000 (uniform=2.197)
      - entropy_coef controller pegged at COEF_MAX without effect
        because PPO update couldn't escape δ-function fixed point
      - 0 trades closed in 50000 steps; position monotonically
        accumulated to -45871 lots
    With P_MIN=0.02, worst-case H(π_sample) ≈ 0.77 nats — well
    above the observed collapse and well below uniform.
    Per `pearl_blend_formulas_must_have_permanent_floor`.

(2) argo template + script — N_BACKTESTS default 1 → 16. The CLI
    default lift in commit 3737feb66 was overridden by both the
    argo script and the wftmpl `value: "1"`. 9k9x6 actually ran
    b_size=1 despite being framed as the b_size=16 test.
    Per `pearl_b_size_1_signal_starvation_blocks_q_learning`.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-24 11:26:37 +02:00
parent a01a376bd2
commit 9c6c280bd8
3 changed files with 45 additions and 2 deletions

View File

@@ -32,7 +32,11 @@ BRANCH=$(git symbolic-ref --short HEAD 2>/dev/null || echo "ml-alpha-phase-a")
GPU_POOL=ci-training-l40s
N_STEPS=1000
SEQ_LEN=32
N_BACKTESTS=1
# Default 16 — fixes b_size=1 signal starvation per
# `pearl_b_size_1_signal_starvation_blocks_q_learning`. 16 parallel
# transitions per Adam step give Q proper gradient variance reduction.
# Override with `--n-backtests N` for sweeps.
N_BACKTESTS=16
PER_CAPACITY=4096
SEED=16962
INSTRUMENT_MODE=front-month