gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(class-a-p0a-downstream): DD penalty + MIN_HOLD_PENALTY_MAX scale to POS_CAP_ADAPTIVE

Browse Source 
Per Class A audit P0-A downstream batch — both constants were tuned for
fixed REWARD_POS_CAP=5.0f. P0-A made POS_CAP adaptive via isv[452]; this
commit propagates the ratio to keep the Kahneman 2:1 asymmetry coherent
across the reward shaping chain.

Items:
1. DD penalty -5.0f → -1.0f * isv[REWARD_POS_CAP_ADAPTIVE_INDEX]
   (helper signature change in trade_physics.cuh::compute_drawdown_penalty
   adds dd_penalty_scale parameter; sole call site in
   experience_kernels.cu:3775 resolves the ISV value with the same
   defensive guard as sp15_apply_sp12_cap)
2. MIN_HOLD_PENALTY_MAX 3.0f → 0.6f * isv[REWARD_POS_CAP_ADAPTIVE_INDEX]
   (existing 60% ratio from state_layout.cuh comment line 252-253
   preserved; resolved at the call site mirroring the
   effective_min_hold_target precedent for slot 451)

Cold-start fallbacks preserved:
- DD penalty: REWARD_POS_CAP=5.0f when ISV at sentinel/out-of-range
- MIN_HOLD_PENALTY_MAX: kernel-passed 3.0f from
  gpu_experience_collector.rs:399 (bit-identical pre-P0-A behavior)

Defensive guard at both consumer sites: ISV must be in
[REWARD_POS_CAP_MIN_BOUND=1.0, REWARD_POS_CAP_MAX_BOUND=50.0] AND not
within 1e-6f of SENTINEL_REWARD_POS_CAP=5.0f. Mirrors the existing
sp15_apply_sp12_cap and segment-complete cap fallback patterns.

Note: the SP15 quadratic DD penalty path (compute_sp15_final_reward_
kernel.cu::sp15_dd_penalty) is already fully ISV-driven via slots 420
(λ_dd) and 421 (dd_threshold) — only the legacy compute_drawdown_
penalty (linear ramp, slot-free) had the hardcoded -5.0f. The audit
recommendation suggested ratio = 1.0 for MIN_HOLD_PENALTY_MAX assuming
the value was 5.0f; actual is 3.0f and the existing tuning comment
locks the ratio at 60% — pure wiring uses 0.6.

Cumulative WR-plateau fix series:
- Class C bug 1 + P0-B (8f218cab2)
- P0-C (316db416b)
- P0-A (394de7d43) — adaptive POS_CAP/NEG_CAP producer
- P1 wiring (c4b6d6ef2) — var_floor only
- P0-A-downstream (this commit)

Per feedback_isv_for_adaptive_bounds + feedback_no_partial_refactor.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-08 09:18:22 +02:00
parent c4b6d6ef29
commit 657972a4b5
3 changed files with 113 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

53
crates/ml/src/cuda_pipeline/experience_kernels.cu
View File

@@ -3197,11 +3197,36 @@ extern "C" __global__ void experience_env_step(
(isv_hold_target > 0.0f && isv_hold_target < 240.0f)
? isv_hold_target
: min_hold_target; /* cold-start / NULL fallback */
/* Class A P0-A-downstream (2026-05-08): min_hold_penalty_max
* was tuned as a fixed 60% fraction of the pre-P0-A
* REWARD_POS_CAP=5.0f (= 3.0). With POS_CAP now adaptive via
* ISV[REWARD_POS_CAP_ADAPTIVE_INDEX=452], propagate the same
* 0.6 ratio so the penalty's relative size to the bounded
* reward stays constant ("60% of REWARD_POS_CAP — meaningful
* gradient pressure but not paralyzing" — state_layout.cuh
* line 252-253). Cold-start fallback: sentinel ISV (5.0f
* within EPS) OR ISV outside the [1.0, 50.0] defensive
* window → fall back to the kernel-passed `min_hold_penalty
* _max` (3.0f from gpu_experience_collector.rs:399), which
* is bit-identical to pre-P0-A behavior. Mirrors the
* effective_min_hold_target pattern above. Per
* feedback_isv_for_adaptive_bounds + feedback_no_partial_
* refactor. */
const float MIN_HOLD_PENALTY_RATIO = 0.6f;
float effective_min_hold_penalty_max = min_hold_penalty_max;
if (isv_signals_ptr != NULL) {
const float isv_pos = isv_signals_ptr[REWARD_POS_CAP_ADAPTIVE_INDEX];
if (isv_pos >= REWARD_POS_CAP_MIN_BOUND
&& isv_pos <= REWARD_POS_CAP_MAX_BOUND
&& fabsf(isv_pos - SENTINEL_REWARD_POS_CAP) > 1e-6f) {
effective_min_hold_penalty_max = MIN_HOLD_PENALTY_RATIO * isv_pos;
}
}
const float min_hold_penalty = compute_min_hold_penalty(
segment_hold_time,
effective_min_hold_target,
min_hold_temperature,
min_hold_penalty_max
effective_min_hold_penalty_max
);
float adjusted_pnl = capped_pnl - shaping_scale * min_hold_penalty;
r_popart = adjusted_pnl;
@@ -3770,9 +3795,31 @@ extern "C" __global__ void experience_env_step(
* Phase 3: gated by shaping_scale. The drawdown penalty is *behavioral*
* (steers the policy away from large equity drops); the *physics* of
* drawdown impact is already in the per-bar P&L. At shaping_scale=0
* (validation mode) we measure pure equity-change reward only. */
* (validation mode) we measure pure equity-change reward only.
*
* Class A P0-A-downstream (2026-05-08): the scale magnitude (was the
* hardcoded `-5.0f` inside compute_drawdown_penalty, sized to match
* the pre-P0-A REWARD_POS_CAP=5.0f) now scales proportionally with
* the adaptive REWARD_POS_CAP via ISV[REWARD_POS_CAP_ADAPTIVE_INDEX=
* 452]. The 1:1 ratio (penalty saturates at −1×POS_CAP × w_dd)
* preserves the original tuning relative to the cap. Cold-start
* fallback: sentinel ISV (5.0f within EPS) OR ISV outside
* [REWARD_POS_CAP_MIN_BOUND=1.0, REWARD_POS_CAP_MAX_BOUND=50.0] →
* fall back to the original 5.0f magnitude (bit-identical pre-P0-A
* cold-start behavior). Defensive guard mirrors `sp15_apply_sp12_cap`
* and the segment-complete asymmetric cap. Per
* feedback_isv_for_adaptive_bounds + feedback_no_partial_refactor. */
float dd_penalty_scale = REWARD_POS_CAP; /* fallback = 5.0f from state_layout.cuh */
if (isv_signals_ptr != NULL) {
const float isv_pos = isv_signals_ptr[REWARD_POS_CAP_ADAPTIVE_INDEX];
if (isv_pos >= REWARD_POS_CAP_MIN_BOUND
&& isv_pos <= REWARD_POS_CAP_MAX_BOUND
&& fabsf(isv_pos - SENTINEL_REWARD_POS_CAP) > 1e-6f) {
dd_penalty_scale = isv_pos;
}
}
float equity_now = cash + position * raw_close;
reward += shaping_scale * compute_drawdown_penalty(equity_now, peak_equity, dd_threshold, w_dd);
reward += shaping_scale * compute_drawdown_penalty(equity_now, peak_equity, dd_threshold, w_dd, dd_penalty_scale);
/* ---- Portfolio value for floor check ---- */
float new_portfolio_value_pre_floor = cash + position * raw_close;

17
crates/ml/src/cuda_pipeline/trade_physics.cuh
View File

@@ -535,19 +535,28 @@ __device__ __forceinline__ float compute_lump_sum_opp_cost(
}
/* ── Drawdown penalty: smooth ramp from threshold to floor ───────────── */
/* Returns a penalty in [-5*w_dd, 0]. Applied every step so the model */
/* learns to reduce position size DURING drawdown. */
/* Returns a penalty in [-dd_penalty_scale * w_dd, 0]. Applied every step */
/* so the model learns to reduce position size DURING drawdown. */
/* */
/* Class A P0-A-downstream (2026-05-08): `dd_penalty_scale` lifted from */
/* the hardcoded `-5.0f` (= 1× the pre-P0-A REWARD_POS_CAP=5.0f) to a */
/* caller-supplied scalar so it can scale proportionally with the now- */
/* adaptive REWARD_POS_CAP (ISV[REWARD_POS_CAP_ADAPTIVE_INDEX=452]). */
/* Cold-start fallback (sentinel ISV) lands at the original 5.0f at the */
/* call site, preserving bit-identical pre-P0-A behavior. Per */
/* feedback_isv_for_adaptive_bounds + feedback_no_partial_refactor. */
__device__ __forceinline__ float compute_drawdown_penalty(
float equity, float peak_equity,
float dd_threshold, float w_dd
float dd_threshold, float w_dd,
float dd_penalty_scale /* magnitude — caller passes pos_cap_eff (production: ISV[452] with sentinel-fallback) */
) {
if (w_dd <= 0.0f) return 0.0f;
float dd = compute_drawdown(equity, peak_equity);
if (dd <= dd_threshold) return 0.0f;
float floor_dd = 0.25f; // capital floor = 25% DD
float dd_excess = fminf((dd - dd_threshold) / (floor_dd - dd_threshold), 1.0f);
return -5.0f * dd_excess * w_dd;
return -dd_penalty_scale * dd_excess * w_dd;
}
/* ── Capital floor circuit breaker (PDT $25K rule) ───────────────────── */

50
docs/dqn-wire-up-audit.md
View File

@@ -7942,3 +7942,53 @@ Class A P1 audit identified four hardcoded constants that the Class A audit flag
1. The audit description for Item 1 conflates `floor_dd` (saturation point) and `dd_threshold` (trigger point) — these are distinct parameters in `compute_drawdown_penalty`. Future producer batch must add a separate slot for the saturation floor, not reuse slot 421.
2. The legacy `compute_drawdown_penalty` (called from `experience_kernels.cu:3769`) and the SP15 `sp15_dd_penalty` (called from `compute_sp15_final_reward_kernel.cu:154`) coexist with separate parameter flows. The SP15 path is fully ISV-driven (slots 420, 421); the legacy path takes config scalars. A future cleanup should retire the legacy path or migrate it to ISV.
## Class A P0-A-downstream: ratio scaling to REWARD_POS_CAP_ADAPTIVE (2026-05-08)
P0-A made `REWARD_POS_CAP` adaptive via `ISV[REWARD_POS_CAP_ADAPTIVE_INDEX=452]` (commit `394de7d43`). Two downstream penalty/cap constants were tuned for the pre-P0-A fixed `REWARD_POS_CAP=5.0f` and now scale **proportionally** to keep the reward-shaping chain coherent. Both are pure wiring (no new producers, no new ISV slots).
### Items
#### Item 1 — DONE: trade_physics.cuh DD penalty `-5.0f` scales to ISV[452]
- **Helper signature change**: `compute_drawdown_penalty` (in `crates/ml/src/cuda_pipeline/trade_physics.cuh:589-600`) gained a `float dd_penalty_scale` parameter. Body changed from `return -5.0f * dd_excess * w_dd;` to `return -dd_penalty_scale * dd_excess * w_dd;`. The hardcoded `-5.0f` was sized to match the pre-P0-A `REWARD_POS_CAP=5.0f` (1:1 ratio), so propagating the **adaptive** POS cap into the scale preserves the original tuning relative to the cap.
- **Consumer site migrated**: `crates/ml/src/cuda_pipeline/experience_kernels.cu:3775-3805` (sole call site — verified via `grep -nE "compute_drawdown_penalty\("`). Resolves `dd_penalty_scale` from `isv_signals_ptr[REWARD_POS_CAP_ADAPTIVE_INDEX]` with the same defensive guard as `sp15_apply_sp12_cap`: ISV value must be in `[REWARD_POS_CAP_MIN_BOUND=1.0f, REWARD_POS_CAP_MAX_BOUND=50.0f]` AND not within `1e-6f` of `SENTINEL_REWARD_POS_CAP=5.0f`. Outside that window → fall back to `REWARD_POS_CAP=5.0f` (state_layout.cuh macro), bit-identical pre-P0-A behavior.
- **No oracle test impact**: `compute_drawdown_penalty` has no SP12-trio oracle test (only `compute_asymmetric_capped_pnl` / `compute_min_hold_penalty` / `compute_lump_sum_opp_cost` are exercised via `sp12_reward_math_test_kernel.cu`). Signature change is internal to the production reward-composition path.
#### Item 2 — DONE: MIN_HOLD_PENALTY_MAX scales as 0.6 × ISV[452]
- **No helper signature change**: `compute_min_hold_penalty` (`trade_physics.cuh:542-552`) already takes `min_hold_penalty_max` as a parameter. Adaptive resolution lives at the **call site** in `crates/ml/src/cuda_pipeline/experience_kernels.cu:3191-3231`, mirroring the `effective_min_hold_target` precedent (Class A P0-C, slot 451) directly above it.
- **Ratio = 0.6, NOT 1.0**: per the existing `state_layout.cuh:251-253` comment ("MIN_HOLD_PENALTY_MAX = 3.0 = **60%** of REWARD_POS_CAP. Meaningful gradient pressure but not paralyzing"), the penalty was *deliberately* tuned at 60% of the cap, not 100%. Pure wiring preserves that 0.6 ratio: `effective_min_hold_penalty_max = MIN_HOLD_PENALTY_RATIO × isv_pos` with `MIN_HOLD_PENALTY_RATIO = 0.6f` declared at the call site. Cold-start fallback: kernel-passed `min_hold_penalty_max` (3.0f from `gpu_experience_collector.rs:399`), bit-identical pre-P0-A behavior. **Note**: the audit recommendation suggested ratio = 1.0 based on an assumed MIN_HOLD_PENALTY_MAX = 5.0f; actual is 3.0f. Choosing 1.0 would change tuning from 60% to 100% — that's not pure wiring. 0.6 is correct.
- **Defensive guard**: same as Item 1 — ISV must be in `[1.0, 50.0]` and not within `1e-6f` of sentinel 5.0f.
### Sites modified
| File | LOC delta | Change |
|------|-----------|--------|
| `crates/ml/src/cuda_pipeline/trade_physics.cuh` | +12 / −4 | `compute_drawdown_penalty` adds `dd_penalty_scale` param + docstring banner |
| `crates/ml/src/cuda_pipeline/experience_kernels.cu` | +35 / −3 | Two adaptive resolutions: `dd_penalty_scale` (line 3775) + `effective_min_hold_penalty_max` (line 3200) |
| `docs/dqn-wire-up-audit.md` | +30 | This entry |
### Architectural decisions
1. **No new ISV slots, no new producers**: pure consumer-side wiring of an existing P0-A producer (slot 452). Slot count `ISV_TOTAL_DIM=454` unchanged; checkpoint compatibility preserved.
2. **Defensive guard mirrors `sp15_apply_sp12_cap`**: same `[1.0, 50.0]` window + sentinel exclusion at every consumer of slot 452. Single source of truth for the adaptive POS cap; consumers never re-derive bounds.
3. **Helper signature change for Item 1**: `compute_drawdown_penalty` now takes `dd_penalty_scale` explicitly rather than reading ISV inside the device function. Matches the parametric convention noted in `trade_physics.cuh` lines 466-469 ("device functions take the bounds/parameters as arguments rather than reading the macros directly so the test wrapper can drive the same code path with arbitrary values"). No oracle test uses this helper, but the convention is preserved for future test wrappers.
4. **Ratio = 0.6 for Item 2 (NOT 1.0)**: the existing tuning comment locks the relative size at 60% of the cap. Pure wiring propagates the existing ratio, not a tuned new value.
### Verification
- `SQLX_OFFLINE=true CUDA_COMPUTE_CAP=86 cargo check -p ml --tests --all-targets` — clean (32 warnings, all pre-existing duplicates from earlier batches; 0 new).
- `SQLX_OFFLINE=true CUDA_COMPUTE_CAP=86 cargo test -p ml --test sp14_oracle_tests --release -- --ignored` — 8/8 pass (4 P0-A reward cap tests + 4 baseline).
- `SQLX_OFFLINE=true CUDA_COMPUTE_CAP=86 cargo test -p ml --test sp15_phase1_oracle_tests --release -- --ignored` — 36/36 pass (no regression from `compute_drawdown_penalty` signature change — SP15 reward axis path uses `sp15_dd_penalty`, not the legacy helper).
### Cumulative WR-plateau fix series
- Class C bug 1 + P0-B (`8f218cab2`): replay buffer intent→realized + Kelly warmup floor wiring.
- P0-C (`316db416b`): MIN_HOLD_TARGET adaptive from AVG_WIN_HOLD_TIME.
- P0-A (`394de7d43`): REWARD_POS/NEG_CAP adaptive producer (slots 452-453).
- P1 wiring (`c4b6d6ef2`): var_floor q_gap-only adaptive (1 of 4 wireable, 3 deferred for slot allocation).
- P0-A-downstream (this commit): DD penalty + MIN_HOLD_PENALTY_MAX scale to POS_CAP_ADAPTIVE.
### Concerns
1. **The legacy `compute_drawdown_penalty` is now partially adaptive but the SP15 `sp15_dd_penalty` quadratic path is fully ISV-driven via slots 420-421**. The two paths still coexist with different shapes (legacy: linear ramp scaled by POS_CAP; SP15: quadratic gated by λ_dd). The P1 batch already flagged this as future work. P0-A-downstream does not retire the legacy path — that's a separate refactor.
2. **`MIN_HOLD_PENALTY_RATIO = 0.6f` is a magic number at the call site** rather than a `#define` in `state_layout.cuh`. Kept local because it's strictly a relationship between two existing constants (3.0 / 5.0 = 0.6), not a new tunable signal. If a future batch lifts the ratio itself to ISV, the constant should move to `state_layout.cuh` first.