285d42aa7b
Layered risk stack per spec docs/superpowers/specs/2026-05-30-adaptive-risk-management-design.md:
Layer 1 (CMDP) hard session-DD, cooldown, max-open, inventory limits
Layer 2 (IQN τ) risk-averse action selection adapts to session drawdown
Layer 3 (Inventory) Avellaneda-Stoikov penalty β scales with reward magnitude
and inventory variance
Layer 4 (Kelly) half-Kelly fraction sizing from observed win-rate +
R-multiple, warmup-gated until cumulative_dones >= 1000
Layer D (Trail) wire dead a7/a8 (TrailTighten/Loosen) via independent
ISV factors, replacing the symmetric reciprocal
Architecture: every threshold ISV-driven (22 new slots, RL_SLOTS_END 662→684).
Every adaptive bound follows the canonical Wiener-α blend with floor 0.4,
sentinel-zero bootstrap, and asymmetric Schulman where applicable.
Kernels:
rl_cmdp_constraints_check session pnl + cooldown + consec-loss tracking
rl_iqn_action_tau_controller τ = clamp(0.5 - 5·dd_frac, τ_min, 1.0)
rl_inventory_beta_controller β_target = 0.01·E|reward| / (2·σ_inventory)
rl_kelly_fraction_controller f = clamp(safety · (p·b - q)/b, 0, 1)
rl_win_rate_ema_update closed-trade win-rate EMA from rewards + dones
rl_avg_win_loss_ema_update separate avg-win and avg-loss EMAs
rl_inventory_variance_update Welford variance of net-position-per-batch
Integration:
- All 7 new cubins loaded in IntegratedTrainer + launched per step in spec order
(CMDP after reward pipeline, before actions_to_market_targets reads override
flags; Layer 2/3/4 controllers in rl_fused_controllers.cu).
- actions_to_market_targets.cu: Layer 1 hard overrides (DD-triggered → 0 lots;
cooldown → 0 lots; max-open → block opening actions; inventory cap → block
one-sided expansion) and Layer 4 Kelly fraction scaling on target lots.
- rl_fused_reward_pipeline.cu: Layer 3 inventory penalty term in reward shaping.
- rl_trail_mutate.cu: a7 multiplies trail by RL_TRAIL_TIGHTEN_FACTOR_INDEX,
a8 by RL_TRAIL_LOOSEN_FACTOR_INDEX (was symmetric reciprocal — pearls
pearl_dead_trail_stop_actions_a7_a8).
Validation:
- 12 GPU-oracle invariants pass (tests/risk_stack_invariants.rs):
G1-G4 CMDP, G5-G7 IQN τ, G8-G9 inventory β, G10-G12 Kelly.
- 20/20 trade_management_kernels.rs tests pass (a7/a8 migrated).
- 5/5 controller_adaptive_floors.rs tests still pass.
- integrated_trainer_smoke passes (end-to-end pipeline launch).
- Local 1k smoke b=128: completes 1000/1000 steps, no NaN, controllers steady.
- compute-sanitizer memcheck (5 steps b=128): ERROR SUMMARY: 0 errors.
Plan: docs/superpowers/plans/2026-05-30-adaptive-risk-management-plan.md
58 lines
2.3 KiB
Plaintext
58 lines
2.3 KiB
Plaintext
// rl_inventory_beta_controller.cu — Layer 3: adaptive inventory penalty β.
|
||
//
|
||
// Spec: docs/superpowers/specs/2026-05-30-adaptive-risk-management-design.md
|
||
//
|
||
// Scales the inventory penalty `reward_shaped = reward_raw - β × |net_pos|`
|
||
// so that at ~2σ inventory the penalty is ~1% of typical reward magnitude.
|
||
//
|
||
// β_target = 0.01 × E[|reward|] / (2 × σ_inventory)
|
||
//
|
||
// Sentinel-zero bootstrap per `pearl_first_observation_bootstrap`: hold β
|
||
// at 0 until BOTH reward magnitude EMA and inventory variance EMA have
|
||
// real data. After bootstrap, Wiener-α blend with floor 0.4 per
|
||
// `pearl_wiener_alpha_floor_for_nonstationary`.
|
||
//
|
||
// Per `feedback_no_atomicadd`: single-thread single-block.
|
||
// Per `feedback_cpu_is_read_only`: pure device kernel.
|
||
// Per `feedback_isv_for_adaptive_bounds`: every threshold via ISV slot.
|
||
|
||
#include <stdint.h>
|
||
|
||
#define RL_INVENTORY_PENALTY_BETA_INDEX 674
|
||
#define RL_INVENTORY_VARIANCE_EMA_INDEX 675
|
||
#define RL_REWARD_MAGNITUDE_EMA_INDEX 614
|
||
#define RL_WIENER_ALPHA_FLOOR_INDEX 659
|
||
|
||
// 1% of typical reward magnitude at 2σ inventory — Avellaneda-Stoikov
|
||
// canonical "noticeable but not dominant" calibration. Structural ratio.
|
||
#define BETA_TARGET_FRAC_OF_REWARD 0.01f
|
||
#define BETA_SIGMA_MULTIPLIER 2.0f
|
||
|
||
extern "C" __global__ void rl_inventory_beta_controller(
|
||
float* __restrict__ isv
|
||
) {
|
||
if (threadIdx.x != 0 || threadIdx.y != 0 || threadIdx.z != 0) return;
|
||
if (blockIdx.x != 0 || blockIdx.y != 0 || blockIdx.z != 0) return;
|
||
|
||
const float reward_mag = isv[RL_REWARD_MAGNITUDE_EMA_INDEX];
|
||
const float inv_var = isv[RL_INVENTORY_VARIANCE_EMA_INDEX];
|
||
|
||
// Dead-signal hold: need real data on both inputs.
|
||
if (inv_var <= 0.0f || reward_mag <= 0.0f) return;
|
||
|
||
const float inv_std = sqrtf(inv_var);
|
||
const float beta_target = BETA_TARGET_FRAC_OF_REWARD * reward_mag
|
||
/ (BETA_SIGMA_MULTIPLIER * inv_std);
|
||
|
||
const float prev = isv[RL_INVENTORY_PENALTY_BETA_INDEX];
|
||
const float a_floor = isv[RL_WIENER_ALPHA_FLOOR_INDEX];
|
||
|
||
if (prev == 0.0f) {
|
||
// First-observation bootstrap.
|
||
isv[RL_INVENTORY_PENALTY_BETA_INDEX] = beta_target;
|
||
} else {
|
||
isv[RL_INVENTORY_PENALTY_BETA_INDEX] =
|
||
(1.0f - a_floor) * prev + a_floor * beta_target;
|
||
}
|
||
}
|