feat(rl): adaptive risk-management stack — 5 layers, all ISV-driven
Layered risk stack per spec docs/superpowers/specs/2026-05-30-adaptive-risk-management-design.md:
Layer 1 (CMDP) hard session-DD, cooldown, max-open, inventory limits
Layer 2 (IQN τ) risk-averse action selection adapts to session drawdown
Layer 3 (Inventory) Avellaneda-Stoikov penalty β scales with reward magnitude
and inventory variance
Layer 4 (Kelly) half-Kelly fraction sizing from observed win-rate +
R-multiple, warmup-gated until cumulative_dones >= 1000
Layer D (Trail) wire dead a7/a8 (TrailTighten/Loosen) via independent
ISV factors, replacing the symmetric reciprocal
Architecture: every threshold ISV-driven (22 new slots, RL_SLOTS_END 662→684).
Every adaptive bound follows the canonical Wiener-α blend with floor 0.4,
sentinel-zero bootstrap, and asymmetric Schulman where applicable.
Kernels:
rl_cmdp_constraints_check session pnl + cooldown + consec-loss tracking
rl_iqn_action_tau_controller τ = clamp(0.5 - 5·dd_frac, τ_min, 1.0)
rl_inventory_beta_controller β_target = 0.01·E|reward| / (2·σ_inventory)
rl_kelly_fraction_controller f = clamp(safety · (p·b - q)/b, 0, 1)
rl_win_rate_ema_update closed-trade win-rate EMA from rewards + dones
rl_avg_win_loss_ema_update separate avg-win and avg-loss EMAs
rl_inventory_variance_update Welford variance of net-position-per-batch
Integration:
- All 7 new cubins loaded in IntegratedTrainer + launched per step in spec order
(CMDP after reward pipeline, before actions_to_market_targets reads override
flags; Layer 2/3/4 controllers in rl_fused_controllers.cu).
- actions_to_market_targets.cu: Layer 1 hard overrides (DD-triggered → 0 lots;
cooldown → 0 lots; max-open → block opening actions; inventory cap → block
one-sided expansion) and Layer 4 Kelly fraction scaling on target lots.
- rl_fused_reward_pipeline.cu: Layer 3 inventory penalty term in reward shaping.
- rl_trail_mutate.cu: a7 multiplies trail by RL_TRAIL_TIGHTEN_FACTOR_INDEX,
a8 by RL_TRAIL_LOOSEN_FACTOR_INDEX (was symmetric reciprocal — pearls
pearl_dead_trail_stop_actions_a7_a8).
Validation:
- 12 GPU-oracle invariants pass (tests/risk_stack_invariants.rs):
G1-G4 CMDP, G5-G7 IQN τ, G8-G9 inventory β, G10-G12 Kelly.
- 20/20 trade_management_kernels.rs tests pass (a7/a8 migrated).
- 5/5 controller_adaptive_floors.rs tests still pass.
- integrated_trainer_smoke passes (end-to-end pipeline launch).
- Local 1k smoke b=128: completes 1000/1000 steps, no NaN, controllers steady.
- compute-sanitizer memcheck (5 steps b=128): ERROR SUMMARY: 0 errors.
Plan: docs/superpowers/plans/2026-05-30-adaptive-risk-management-plan.md
This commit is contained in:
jgrusewski
1185
docs/superpowers/plans/2026-05-30-adaptive-risk-management-plan.md
Normal file
1185
docs/superpowers/plans/2026-05-30-adaptive-risk-management-plan.md
Normal file
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,560 @@
|
||||
# Adaptive Risk Management — Design (Layered Stack)
|
||||
|
||||
**Goal**: close the train-eval distribution-shift gap surfaced by the 2026-05-30 adaptive-controller-floor refactor (commit `b1ef6664a`). The fix preserves the +$18M baseline (G9 PASS, +$15.81M) but fold 1 walk-forward eval came in marginally worse (-$616k vs -$500k OLD) despite massively better train pnl. **Eval JSONL revealed the load-bearing missing layers**: position commitment, action selection risk-awareness, and inventory exposure are all under-adapted relative to what the controller-floor architecture made possible.
|
||||
|
||||
**Why now**: the eval `win_rate=0.21 × R-multiple=1.8` Kelly fraction is negative — agent should not trade but does (1024 batches). Train win_rate 0.55 vs eval 0.21 = severe distribution shift. The controller-floor refactor made signal *interpretation* adaptive; commitment, action-selection, and reward shaping for inventory remained hardcoded.
|
||||
|
||||
**Out of scope**: changes to the perception encoder, Mamba2/CfC architecture, advantage normalization (Phase 4.5), the 12 adaptive controllers from b1ef6664a (those are now correct). Decision policy (the agent still chooses among the 11 discrete actions). Regime detection / meta-policy switching (deferred to follow-up per research caveat).
|
||||
|
||||
---
|
||||
|
||||
## Diagnosis — what the JSONL says
|
||||
|
||||
Per fold 1 eval (`/feature-cache/alpha-rl-runs/b1ef6664a/fold1/eval_summary.json`):
|
||||
|
||||
| Metric | Value | Interpretation |
|
||||
|---|---|---|
|
||||
| total_pnl_usd | -$616k | losing |
|
||||
| avg_win | $13,785 | wins are large |
|
||||
| avg_loss | $7,631 | R-multiple 1.8 (good) |
|
||||
| win_rate | 0.21 (vs train 0.55) | severe directional overfit |
|
||||
| max_drawdown_pct | 18.7% | controlled but real |
|
||||
| profit_factor | 0.83 | losing aggregate |
|
||||
|
||||
**Kelly at observed edge**: f* = (0.21 × 1.8 − 0.79)/1.8 = **−0.23** → don't trade. Agent trades anyway. **Position commitment never adapts to edge.**
|
||||
|
||||
**Internal evidence from in-run JSONL** (fold 1 step 16k, qpa = +0.68 = high agent confidence): **the IQN distribution would tell us the pessimistic Q (lower quantile) is significantly negative even when expected Q is positive**. The agent uses `argmax(E_τ[Q])` (mean over quantiles) — over-weights upside under noisy data. Risk-averse action selection (lower quantile τ) would naturally filter these losing trades.
|
||||
|
||||
**Bigger picture**: the system has no adaptive layer between "agent's decision" and "actual market commitment." Five distinct layers are absent or hardcoded:
|
||||
|
||||
| Layer | What it does | Status pre-spec |
|
||||
|---|---|---|
|
||||
| Hard constraints (max position, DD cap, etc.) | Override agent on safety violation | Mostly absent |
|
||||
| Risk-averse action selection | Bias decisions toward downside-aware Q | Uses E[Q] — risk-neutral |
|
||||
| Inventory exposure penalty | Discourage one-sided drift | Absent in reward shaping |
|
||||
| Position commitment / sizing | Scale lots by observed edge | Fixed at base lots |
|
||||
| Regime detection | Distribution-shift handling | Absent |
|
||||
|
||||
This spec addresses the first four. Regime detection is deferred per research evidence ("research-active, not production-validated").
|
||||
|
||||
---
|
||||
|
||||
## Scope — the layered risk stack
|
||||
|
||||
Apply `feedback_adaptive_not_tuned` consistently across the commitment layer:
|
||||
|
||||
1. **Layer 1 — Hard CMDP-style constraints (~120 LOC)**: 5 deterministic risk gates that override agent actions
|
||||
2. **Layer 2 — IQN risk-averse action selection (~30 LOC)**: use lower-quantile τ from existing IQN distribution
|
||||
3. **Layer 3 — Inventory penalty in reward shaping (~60 LOC)**: discourage one-sided net-position drift
|
||||
4. **Layer 4 — Kelly-fraction position sizing (~80 LOC)**: scale committed lots by observed edge
|
||||
5. **TrailTighten/Loosen wiring (~30 LOC)**: close dead-action gap per [[pearl_dead_trail_stop_actions_a7_a8]]
|
||||
|
||||
Single atomic commit per `feedback_no_partial_refactor`. Same pattern as the controller-floor refactor.
|
||||
|
||||
**The crucial reordering vs an earlier draft of this spec**: Kelly was originally the primary fix. Research (FineFT 2026, distributional-RL surveys) plus internal analysis showed **Layer 2 (IQN risk-averse τ) is the higher-leverage minimum-cost change** because we already compute the full IQN quantile distribution but throw away everything except the expectation. Kelly is now Layer 4 — fine-tuning after pessimistic action selection has filtered the trade set.
|
||||
|
||||
---
|
||||
|
||||
## Approaches considered
|
||||
|
||||
### Approach A — Kelly fraction only
|
||||
Original minimum-viable design. Single new controller computes f* from observed win_rate × R-multiple, multiplies lot size. ~80 LOC.
|
||||
|
||||
**Verdict**: Insufficient. Kelly is the *commitment* layer; the *selection* layer (which trades to even consider) is more impactful. The 21% eval win_rate means the agent shouldn't even be entering most trades, regardless of size.
|
||||
|
||||
### Approach B — Kelly + session DD circuit breaker
|
||||
Add a session-pnl-tracking ISV slot + override-to-Hold logic. Two-layer (Kelly per-trade + session-wide cap).
|
||||
|
||||
**Verdict**: Closer but still missing the action-selection layer. Session DD CB triggers AFTER damage is done.
|
||||
|
||||
### Approach C — B + confidence-to-size scaling
|
||||
Add Q-distribution magnitude as continuous size multiplier alongside Kelly.
|
||||
|
||||
**Verdict**: Three multiplicative layers (Kelly × confidence × base) become hard to reason about. Confidence is implicit in Layer 2's quantile selection.
|
||||
|
||||
### Approach D — Add TrailTighten/Loosen wiring to any of A/B/C
|
||||
Close the 10%-policy-mass-wasted gap by activating dead actions.
|
||||
|
||||
**Verdict**: Orthogonal to risk management. Worth doing but small impact.
|
||||
|
||||
### Approach E — Layered risk stack (RECOMMENDED)
|
||||
|
||||
Replace the Kelly-first design with five distinct layers, each addressing a different risk concern:
|
||||
|
||||
| Layer | Concern | ISV-driven? | Production validation |
|
||||
|---|---|---|---|
|
||||
| 1. Hard CMDP gates | Catastrophic-loss safety (session DD, max position, max inventory) | ✓ | Highest (standard in live HFT) |
|
||||
| 2. IQN risk-averse τ | Downside-aware action selection | ✓ (adaptive τ) | Research, but our IQN gives it for free |
|
||||
| 3. Inventory penalty | Adverse selection, one-sided drift | ✓ (adaptive β) | Highest (Avellaneda-Stoikov is canonical HFT) |
|
||||
| 4. Kelly sizing | Edge-aware commitment | ✓ | Lower (research stage in RL context) |
|
||||
| (D) TrailTighten/Loosen | Close dead-action gap | ✓ (factor slots) | N/A — wiring fix |
|
||||
|
||||
**Why E**: research evidence (FineFT KDD 2026, plus broader distributional-RL literature) supports the layered approach. Each layer addresses a distinct failure mode visible in the JSONL. Combined, they form the "deployment-grade risk stack" pattern.
|
||||
|
||||
**Total scope**: ~320 LOC across 5 new + 5 modified files. Same atomic-commit discipline as the prior refactor.
|
||||
|
||||
---
|
||||
|
||||
## Detailed design (Approach E)
|
||||
|
||||
### Layer 1 — Hard CMDP constraints
|
||||
|
||||
Deterministic safety gates that override agent actions in `actions_to_market_targets.cu`. NOT learned by the agent. Each constraint is an independent ISV slot, AND'd together — any single trigger forces Hold (or partial-flatten).
|
||||
|
||||
**New ISV slots** (allocated starting at `RL_SLOTS_END = 662`):
|
||||
|
||||
| Slot | Name | Purpose | Bootstrap |
|
||||
|---|---|---|---|
|
||||
| 662 | `RL_SESSION_PNL_USD_INDEX` | Cumulative pnl since session boundary | 0.0 |
|
||||
| 663 | `RL_SESSION_DD_LIMIT_USD_INDEX` | Drawdown cap (USD, negative) | -3500.0 (= 10% × $35k) |
|
||||
| 664 | `RL_SESSION_DD_TRIGGERED_INDEX` | Sticky flag, set when DD breached | 0.0 |
|
||||
| 665 | `RL_MAX_OPEN_UNITS_INDEX` | Max concurrent open positions per batch | 4.0 (= current `MAX_UNITS`) |
|
||||
| 666 | `RL_CONSEC_LOSS_LIMIT_INDEX` | Max consecutive losses before cool-down | 10.0 |
|
||||
| 667 | `RL_CONSEC_LOSS_COUNT_INDEX` | Running consecutive-loss count | 0.0 |
|
||||
| 668 | `RL_COOLDOWN_REMAINING_STEPS_INDEX` | Cool-down period after limit hit | 0.0 |
|
||||
| 669 | `RL_COOLDOWN_DURATION_INDEX` | Default cool-down length (steps) | 500.0 |
|
||||
| 670 | `RL_NET_INVENTORY_LIMIT_USD_INDEX` | Cap on net one-sided exposure | 10000.0 |
|
||||
|
||||
**Kernel: `rl_cmdp_constraints_check.cu`** (~100 LOC, single-thread single-block, runs once per step before `actions_to_market_targets`)
|
||||
|
||||
```c
|
||||
// Layer 1: hard CMDP gates. Output: writes booleans to ISV slots that
|
||||
// actions_to_market_targets reads as overrides.
|
||||
//
|
||||
// Five independent constraints, ALL must be satisfied (= no override) for
|
||||
// the agent's chosen action to be executed. Any violation → force Hold.
|
||||
|
||||
extern "C" __global__ void rl_cmdp_constraints_check(
|
||||
float* __restrict__ isv,
|
||||
const float* __restrict__ realized_pnl_per_batch, // [b_size]
|
||||
const float* __restrict__ net_position_per_batch, // [b_size] (signed contracts)
|
||||
const u8* __restrict__ unit_active_per_batch, // [b_size × MAX_UNITS]
|
||||
const int* __restrict__ trade_outcome_per_batch, // [b_size] −1/0/+1
|
||||
int b_size
|
||||
) {
|
||||
if (threadIdx.x != 0 || blockIdx.x != 0) return;
|
||||
|
||||
// 1. Session pnl accumulation + DD check.
|
||||
float step_pnl = 0.0f;
|
||||
for (int b = 0; b < b_size; b++) step_pnl += realized_pnl_per_batch[b];
|
||||
const float session_pnl = isv[RL_SESSION_PNL_USD_INDEX] + step_pnl;
|
||||
isv[RL_SESSION_PNL_USD_INDEX] = session_pnl;
|
||||
if (session_pnl < isv[RL_SESSION_DD_LIMIT_USD_INDEX]) {
|
||||
isv[RL_SESSION_DD_TRIGGERED_INDEX] = 1.0f;
|
||||
}
|
||||
|
||||
// 2. Cool-down decrement.
|
||||
float cooldown = isv[RL_COOLDOWN_REMAINING_STEPS_INDEX];
|
||||
if (cooldown > 0.0f) {
|
||||
isv[RL_COOLDOWN_REMAINING_STEPS_INDEX] = cooldown - 1.0f;
|
||||
}
|
||||
|
||||
// 3. Consecutive loss tracking. Outcomes: -1 = loss, 0 = open/hold, +1 = win.
|
||||
float consec = isv[RL_CONSEC_LOSS_COUNT_INDEX];
|
||||
for (int b = 0; b < b_size; b++) {
|
||||
const int outcome = trade_outcome_per_batch[b];
|
||||
if (outcome == -1) consec += 1.0f;
|
||||
else if (outcome == 1) consec = 0.0f;
|
||||
// outcome == 0 (no change): consec unchanged
|
||||
}
|
||||
isv[RL_CONSEC_LOSS_COUNT_INDEX] = consec;
|
||||
if (consec >= isv[RL_CONSEC_LOSS_LIMIT_INDEX]) {
|
||||
isv[RL_COOLDOWN_REMAINING_STEPS_INDEX] = isv[RL_COOLDOWN_DURATION_INDEX];
|
||||
isv[RL_CONSEC_LOSS_COUNT_INDEX] = 0.0f; // reset after triggering
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Override logic in `actions_to_market_targets.cu`:
|
||||
|
||||
```c
|
||||
const bool dd_triggered = isv[RL_SESSION_DD_TRIGGERED_INDEX] >= 0.5f;
|
||||
const bool in_cooldown = isv[RL_COOLDOWN_REMAINING_STEPS_INDEX] > 0.0f;
|
||||
const bool override_hold = dd_triggered || in_cooldown;
|
||||
|
||||
if (override_hold) {
|
||||
actions[b] = ACTION_HOLD;
|
||||
target_lots[b] = 0;
|
||||
return; // skip rest of action translation
|
||||
}
|
||||
|
||||
// Max-open-units check (per-batch, dynamic):
|
||||
int active_units = 0;
|
||||
for (int u = 0; u < MAX_UNITS; u++) {
|
||||
if (unit_active[b * MAX_UNITS + u]) active_units++;
|
||||
}
|
||||
if (active_units >= (int)isv[RL_MAX_OPEN_UNITS_INDEX] &&
|
||||
is_opening_action(actions[b])) {
|
||||
actions[b] = ACTION_HOLD; // refuse new opens, allow closes
|
||||
}
|
||||
|
||||
// Inventory cap check:
|
||||
const float net_pos_usd = compute_net_position_usd(b);
|
||||
if (fabsf(net_pos_usd) > isv[RL_NET_INVENTORY_LIMIT_USD_INDEX] &&
|
||||
increases_one_sided_exposure(actions[b], net_pos_usd)) {
|
||||
actions[b] = ACTION_HOLD;
|
||||
}
|
||||
```
|
||||
|
||||
**Session boundary**: trainer resets `RL_SESSION_PNL_USD_INDEX`, `RL_SESSION_DD_TRIGGERED_INDEX`, `RL_CONSEC_LOSS_COUNT_INDEX` to 0 on each fold boundary (start of train phase, start of eval phase).
|
||||
|
||||
### Layer 2 — IQN risk-averse action selection
|
||||
|
||||
The genuinely-novel insight from research: **we already compute the IQN quantile distribution**. The standard action selector picks `argmax(E_τ[Q(s, a, τ)])` — discards the distribution shape. Instead, pick `argmax(Q(s, a, τ_action))` with adaptive `τ_action ∈ (0, 0.5]` — lower τ = more pessimistic = downside-aware decisions.
|
||||
|
||||
**New ISV slots**:
|
||||
|
||||
| Slot | Name | Purpose | Bootstrap |
|
||||
|---|---|---|---|
|
||||
| 671 | `RL_IQN_ACTION_TAU_INDEX` | Quantile used for action selection ∈ (0, 1] | 0.5 (= median, near current E[Q] behavior) |
|
||||
| 672 | `RL_IQN_ACTION_TAU_MIN_INDEX` | Lower bound | 0.1 |
|
||||
| 673 | `RL_IQN_ACTION_TAU_DD_SENSITIVITY_INDEX` | How fast τ drops with drawdown | 5.0 |
|
||||
|
||||
**Adaptive controller `rl_iqn_action_tau_controller.cu`** (~50 LOC):
|
||||
|
||||
```c
|
||||
// Adapt action-selection quantile based on recent drawdown.
|
||||
// Normal trading: τ = 0.5 (median ≈ expected-Q-like behavior).
|
||||
// In drawdown: τ → MIN (pessimistic — only take trades where even
|
||||
// the worst-case Q is positive).
|
||||
//
|
||||
// Recent drawdown signal: (session_pnl - max_session_pnl) / |starting_capital|
|
||||
// Bounded in [0, 1]: 0 = at peak, 1 = total wipeout.
|
||||
// τ_action = max(τ_min, 0.5 - sensitivity × drawdown_frac)
|
||||
|
||||
extern "C" __global__ void rl_iqn_action_tau_controller(
|
||||
float* __restrict__ isv
|
||||
) {
|
||||
if (threadIdx.x != 0 || blockIdx.x != 0) return;
|
||||
|
||||
const float session_pnl = isv[RL_SESSION_PNL_USD_INDEX];
|
||||
const float max_pnl = isv[RL_SESSION_MAX_PNL_USD_INDEX]; // tracked separately
|
||||
const float capital = isv[RL_STARTING_CAPITAL_USD_INDEX];
|
||||
|
||||
const float drawdown_usd = fmaxf(0.0f, max_pnl - session_pnl);
|
||||
const float drawdown_frac = drawdown_usd / fmaxf(capital, 1.0f);
|
||||
|
||||
const float tau_min = isv[RL_IQN_ACTION_TAU_MIN_INDEX];
|
||||
const float sensitivity = isv[RL_IQN_ACTION_TAU_DD_SENSITIVITY_INDEX];
|
||||
const float tau_target = fmaxf(tau_min, 0.5f - sensitivity * drawdown_frac);
|
||||
isv[RL_IQN_ACTION_TAU_INDEX] = tau_target;
|
||||
}
|
||||
```
|
||||
|
||||
**Modification to existing action selection kernel** (e.g., `rl_ensemble_action_value.cu` or wherever the argmax over Q happens):
|
||||
|
||||
```c
|
||||
// BEFORE:
|
||||
const float exp_q = compute_expected_q(s, a); // mean over sampled τ
|
||||
// argmax over actions on exp_q
|
||||
|
||||
// AFTER:
|
||||
const float tau_a = isv[RL_IQN_ACTION_TAU_INDEX]; // adaptive, default 0.5
|
||||
const float q_at_tau = compute_q_at_quantile(s, a, tau_a);
|
||||
// argmax over actions on q_at_tau
|
||||
```
|
||||
|
||||
The IQN kernel already computes `Q(s, a, τ)` for sampled `τ`. Selecting a specific quantile is a single-line indexing change.
|
||||
|
||||
**Critical interaction with C51 ensemble**: the agent uses C51+IQN ensemble. The ensemble blend (α) is already ISV-driven. We apply the risk-averse τ to the IQN side only, then blend as usual. The C51 side continues to use expected value because C51 doesn't naturally expose quantiles.
|
||||
|
||||
### Layer 3 — Inventory penalty (Avellaneda-Stoikov)
|
||||
|
||||
Add a reward shaping term proportional to `|net_position_per_batch|`. Discourages systematic one-sided exposure that bleeds via adverse selection.
|
||||
|
||||
**New ISV slots**:
|
||||
|
||||
| Slot | Name | Purpose | Bootstrap |
|
||||
|---|---|---|---|
|
||||
| 674 | `RL_INVENTORY_PENALTY_BETA_INDEX` | Penalty coefficient per |net_position| unit | 0.0 (sentinel — adaptive on first observation) |
|
||||
| 675 | `RL_INVENTORY_VARIANCE_EMA_INDEX` | Observed inventory variance for β scaling | 0.0 sentinel |
|
||||
|
||||
**Modification to `rl_fused_reward_pipeline.cu`** (~30 LOC):
|
||||
|
||||
```c
|
||||
// After raw reward computation, before clamps:
|
||||
const float net_pos = net_position_per_batch[b]; // signed contracts
|
||||
const float beta = isv[RL_INVENTORY_PENALTY_BETA_INDEX];
|
||||
const float penalty = beta * fabsf(net_pos); // always reduces reward
|
||||
shaped_reward = raw_reward - penalty;
|
||||
```
|
||||
|
||||
**Adaptive controller `rl_inventory_beta_controller.cu`** (~50 LOC) — derives β from observed inventory variance:
|
||||
|
||||
```c
|
||||
// β should scale so penalty is meaningful when inventory drifts but
|
||||
// negligible during balanced trading.
|
||||
// Target: penalty ≈ 1% of typical absolute reward when inventory at ~2σ.
|
||||
//
|
||||
// β = 0.01 × E[|reward|] / (2 × sqrt(var(inventory)))
|
||||
//
|
||||
// Sentinel-zero: hold β = 0 until variance EMA accumulates real signal.
|
||||
|
||||
const float reward_mag = isv[RL_REWARD_MAGNITUDE_EMA_INDEX]; // existing slot 614
|
||||
const float inv_var = isv[RL_INVENTORY_VARIANCE_EMA_INDEX];
|
||||
if (inv_var <= 0.0f || reward_mag <= 0.0f) return;
|
||||
|
||||
const float inv_std = sqrtf(inv_var);
|
||||
const float beta_target = 0.01f * reward_mag / (2.0f * inv_std);
|
||||
// Wiener blend with floor 0.4 (existing pearl pattern)
|
||||
isv[RL_INVENTORY_PENALTY_BETA_INDEX] = wiener_blend(
|
||||
isv[RL_INVENTORY_PENALTY_BETA_INDEX],
|
||||
beta_target,
|
||||
isv[RL_WIENER_ALPHA_FLOOR_INDEX]
|
||||
);
|
||||
```
|
||||
|
||||
### Layer 4 — Kelly-fraction position sizing
|
||||
|
||||
The original spec's contribution, demoted to fine-tuning layer. Operates *after* Layer 2 has filtered actions and Layer 1 has applied safety gates. Sizes the trades that survive.
|
||||
|
||||
**New ISV slots**:
|
||||
|
||||
| Slot | Name | Purpose | Bootstrap |
|
||||
|---|---|---|---|
|
||||
| 676 | `RL_KELLY_FRACTION_INDEX` | Position-size multiplier ∈ [0, 1] | 1.0 (full size pre-warmup) |
|
||||
| 677 | `RL_WIN_RATE_EMA_INDEX` | Observed win_rate EMA | 0.5 (neutral sentinel) |
|
||||
| 678 | `RL_AVG_WIN_USD_EMA_INDEX` | Observed avg_win | 0.0 sentinel |
|
||||
| 679 | `RL_AVG_LOSS_USD_EMA_INDEX` | Observed avg_loss | 0.0 sentinel |
|
||||
| 680 | `RL_KELLY_SAFETY_FRAC_INDEX` | Half-Kelly multiplier (Thorp) | 0.5 |
|
||||
| 681 | `RL_KELLY_MIN_TRADES_FOR_RELEASE_INDEX` | Warmup gate | 1000.0 |
|
||||
|
||||
**Kernel `rl_kelly_fraction_controller.cu`** (~80 LOC) — same design as original spec, repeated here for completeness:
|
||||
|
||||
```c
|
||||
extern "C" __global__ void rl_kelly_fraction_controller(float* __restrict__ isv) {
|
||||
if (threadIdx.x != 0 || blockIdx.x != 0) return;
|
||||
|
||||
// Warmup: hold at bootstrap until enough closed trades.
|
||||
if (isv[RL_CUMULATIVE_DONES_INDEX] < isv[RL_KELLY_MIN_TRADES_FOR_RELEASE_INDEX]) {
|
||||
isv[RL_KELLY_FRACTION_INDEX] = 1.0f;
|
||||
return;
|
||||
}
|
||||
|
||||
const float p = isv[RL_WIN_RATE_EMA_INDEX];
|
||||
const float avg_win = isv[RL_AVG_WIN_USD_EMA_INDEX];
|
||||
const float avg_loss = isv[RL_AVG_LOSS_USD_EMA_INDEX];
|
||||
if (avg_loss <= 0.0f) { isv[RL_KELLY_FRACTION_INDEX] = 1.0f; return; }
|
||||
|
||||
const float b = avg_win / avg_loss;
|
||||
const float f_kelly = (p * b - (1.0f - p)) / b;
|
||||
const float safety = isv[RL_KELLY_SAFETY_FRAC_INDEX];
|
||||
float f = f_kelly * safety;
|
||||
f = fmaxf(0.0f, fminf(f, 1.0f));
|
||||
isv[RL_KELLY_FRACTION_INDEX] = f;
|
||||
}
|
||||
```
|
||||
|
||||
**Modification in `actions_to_market_targets.cu`** (~3 LOC):
|
||||
|
||||
```c
|
||||
// After Layer 1 overrides have been applied (DD CB, cooldown, etc.):
|
||||
const float kelly = isv[RL_KELLY_FRACTION_INDEX];
|
||||
target_lots[b] = (int)ceilf((float)target_lots[b] * kelly);
|
||||
// If kelly = 0, target_lots = 0 = no commitment.
|
||||
```
|
||||
|
||||
### Layer D — TrailTighten/Loosen wiring
|
||||
|
||||
Resolve the 10%-policy-mass-wasted gap per [[pearl_dead_trail_stop_actions_a7_a8]]:
|
||||
|
||||
**New ISV slots**:
|
||||
|
||||
| Slot | Name | Purpose | Bootstrap |
|
||||
|---|---|---|---|
|
||||
| 682 | `RL_TRAIL_TIGHTEN_FACTOR_INDEX` | Multiplier on trail distance for a7 | 0.9 (tighten by 10%) |
|
||||
| 683 | `RL_TRAIL_LOOSEN_FACTOR_INDEX` | Multiplier on trail distance for a8 | 1.1 (loosen by 10%) |
|
||||
|
||||
**Modification to `actions_to_market_targets.cu`** wires a7/a8 to actually mutate `unit_trail_distance_d`:
|
||||
|
||||
```c
|
||||
case ACTION_TRAIL_TIGHTEN: {
|
||||
const float factor = isv[RL_TRAIL_TIGHTEN_FACTOR_INDEX];
|
||||
for (int u = 0; u < MAX_UNITS; u++) {
|
||||
if (unit_active_d[b * MAX_UNITS + u]) {
|
||||
unit_trail_distance_d[b * MAX_UNITS + u] *= factor;
|
||||
}
|
||||
}
|
||||
break;
|
||||
}
|
||||
case ACTION_TRAIL_LOOSEN: {
|
||||
const float factor = isv[RL_TRAIL_LOOSEN_FACTOR_INDEX];
|
||||
for (int u = 0; u < MAX_UNITS; u++) {
|
||||
if (unit_active_d[b * MAX_UNITS + u]) {
|
||||
unit_trail_distance_d[b * MAX_UNITS + u] *= factor;
|
||||
}
|
||||
}
|
||||
break;
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Slot allocation summary
|
||||
|
||||
22 new ISV slots, `RL_SLOTS_END` 662 → 684.
|
||||
|
||||
| Range | Layer | Purpose |
|
||||
|---|---|---|
|
||||
| 662–670 | 1 (CMDP gates) | Session pnl, DD limit, max open, consec loss, cooldown, inventory cap (9 slots) |
|
||||
| 671–673 | 2 (IQN τ) | Action τ, τ_min, DD sensitivity (3 slots) |
|
||||
| 674–675 | 3 (inventory penalty) | β + inventory variance EMA (2 slots) |
|
||||
| 676–681 | 4 (Kelly) | Kelly fraction + EMAs + safety + warmup (6 slots) |
|
||||
| 682–683 | D (trail) | Tighten/loosen factors (2 slots) |
|
||||
|
||||
Memory impact: 88 bytes additional (negligible).
|
||||
|
||||
---
|
||||
|
||||
## Per-step pipeline order
|
||||
|
||||
The per-step trainer pipeline must execute Layers in this order:
|
||||
|
||||
1. EMA producers (existing — kl_pi_ema, q_divergence_ema, etc., PLUS new: win_rate_ema, avg_win_ema, avg_loss_ema, inventory_variance_ema)
|
||||
2. Welford variance updates (existing kernel from b1ef6664a)
|
||||
3. Adaptive controllers (existing 12 — fused kernel)
|
||||
4. **Layer 1 — `rl_cmdp_constraints_check`** (new)
|
||||
5. **Layer 2 — `rl_iqn_action_tau_controller`** (new — adapts τ from drawdown)
|
||||
6. **Layer 3 — `rl_inventory_beta_controller`** (new — adapts β)
|
||||
7. **Layer 4 — `rl_kelly_fraction_controller`** (new)
|
||||
8. **Layer 5 — TrailTighten/Loosen ISV slots** (no per-step kernel; ISV slots read during action translation)
|
||||
9. Action selection — IQN ensemble uses adaptive τ for argmax
|
||||
10. `actions_to_market_targets` — applies Layer 1 overrides + Layer 4 Kelly scaling + Layer D trail mutations
|
||||
11. Reward computation — `rl_fused_reward_pipeline` applies Layer 3 inventory penalty
|
||||
12. PPO/DQN updates as usual
|
||||
|
||||
---
|
||||
|
||||
## Files
|
||||
|
||||
### New
|
||||
|
||||
- `crates/ml-alpha/cuda/rl_cmdp_constraints_check.cu` (~100 LOC)
|
||||
- `crates/ml-alpha/cuda/rl_iqn_action_tau_controller.cu` (~50 LOC)
|
||||
- `crates/ml-alpha/cuda/rl_inventory_beta_controller.cu` (~50 LOC)
|
||||
- `crates/ml-alpha/cuda/rl_kelly_fraction_controller.cu` (~80 LOC)
|
||||
- `crates/ml-alpha/cuda/rl_win_rate_ema_update.cu` (~40 LOC if not derivable)
|
||||
- `crates/ml-alpha/cuda/rl_avg_win_loss_ema_update.cu` (~50 LOC)
|
||||
- `crates/ml-alpha/cuda/rl_inventory_variance_update.cu` (~40 LOC)
|
||||
- `crates/ml-alpha/tests/risk_stack_invariants.rs` (~10 GPU-oracle gates)
|
||||
|
||||
### Modified
|
||||
|
||||
- `crates/ml-alpha/src/rl/isv_slots.rs` — +22 slots, `RL_SLOTS_END` 662 → 684
|
||||
- `crates/ml-alpha/build.rs` — register new cubins
|
||||
- `crates/ml-alpha/cuda/actions_to_market_targets.cu` — Layer 1 overrides + Layer 4 Kelly scaling + Layer D trail mutations
|
||||
- `crates/ml-alpha/cuda/rl_fused_reward_pipeline.cu` — Layer 3 inventory penalty term
|
||||
- `crates/ml-alpha/cuda/rl_ensemble_action_value.cu` (or wherever IQN action selection happens) — Layer 2 quantile-τ argmax
|
||||
- `crates/ml-alpha/cuda/rl_fused_controllers.cu` — add new branches for τ, β, Kelly (matching the fused-kernel pattern from b1ef6664a)
|
||||
- `crates/ml-alpha/src/trainer/integrated.rs` — load new cubins, wire per-step launches, bootstrap entries, session reset on fold boundary
|
||||
|
||||
### Not touched
|
||||
|
||||
- 12 adaptive controllers from b1ef6664a (orthogonal layer)
|
||||
- Welford-variance kernel infrastructure (reused)
|
||||
- Phase 4.5 advantage normalization
|
||||
- Perception encoder, Mamba2, CfC heads
|
||||
- Decision policy gradient (the agent still picks discrete actions)
|
||||
|
||||
---
|
||||
|
||||
## Validation gates
|
||||
|
||||
### Local unit-level (RTX 3050 Ti)
|
||||
|
||||
**G1 — CMDP DD trigger**: Write `session_pnl = -4000` (below limit -3500) → assert triggered=1, agent actions overridden to Hold.
|
||||
|
||||
**G2 — CMDP cooldown**: Trigger 10 consecutive losses → assert cooldown counter set to 500, agent actions overridden until counter reaches 0.
|
||||
|
||||
**G3 — CMDP max-open**: Pre-set 4 active units, agent picks opening action → assert action forced to Hold.
|
||||
|
||||
**G4 — CMDP inventory cap**: Set `net_position = $11k` (above $10k limit), agent picks opening-same-side action → assert Hold.
|
||||
|
||||
**G5 — IQN τ adapts to drawdown**: Set `session_pnl = -1500` (≈4% drawdown of $35k capital) → assert `τ_action ≈ max(0.1, 0.5 - 5 × 0.043) ≈ 0.285` (mid-pessimistic).
|
||||
|
||||
**G6 — IQN τ at no-drawdown**: Set `session_pnl = max_pnl` → assert `τ_action = 0.5`.
|
||||
|
||||
**G7 — Inventory penalty β adapts**: Set `reward_mag_ema = 0.1`, `inv_variance_ema = 4.0` (σ=2) → assert `β = 0.01 × 0.1 / (2 × 2) = 0.00025`. Negligible at balanced inventory, real at high.
|
||||
|
||||
**G8 — Kelly math at known edge**: Feed `win_rate=0.55, avg_win=1.8, avg_loss=1.0, trades=2000` → expect `f = 0.5 × ((0.55×1.8 - 0.45)/1.8) = 0.5 × 0.30 = 0.15`.
|
||||
|
||||
**G9 — Kelly returns 0 at negative edge**: Feed `win_rate=0.21, avg_win=1.8, avg_loss=1.0, trades=2000` → expect `f = 0` (Kelly negative → clamped).
|
||||
|
||||
**G10 — TrailTighten reduces distance**: Pre-set `unit_trail_distance = 100`, agent picks a7 → assert distance becomes 90.
|
||||
|
||||
**G11 — integrated_trainer_smoke** still passes end-to-end.
|
||||
|
||||
**G12 — 1k local smoke at b=128**: completed_clean, no NaN, all 5 layers' ISV slots populated and tracking sensibly.
|
||||
|
||||
### Cluster
|
||||
|
||||
**G13 — Walk-forward fold 1 re-run** (same params as today's 62x7p, new SHA). Combined layers should close the eval gap.
|
||||
|
||||
Targets (each is a soft pass; combination of 3+ passes = G13 PASS):
|
||||
- eval pnl > -$200k (better than today's -$616k, **PRIMARY**)
|
||||
- profit_factor > 0.90 (vs 0.83)
|
||||
- max_drawdown_pct < 15% (vs 18.7%)
|
||||
- Kelly fraction: ends < 1.0 if observed edge negative (shows Layer 4 active)
|
||||
- IQN τ: ends < 0.5 if any drawdown occurred (shows Layer 2 adaptive)
|
||||
- DD breaker: fires when expected (or never if pnl stays positive)
|
||||
- Inventory penalty β: > 0 throughout (shows Layer 3 adapted off bootstrap)
|
||||
|
||||
**G14 — ksll2 regression test**: same params as today's rwk9t. Final pnl ≥ +$13M (within 20% of +$15.81M). Confirms layered stack doesn't regress large-data regime.
|
||||
|
||||
---
|
||||
|
||||
## Open questions
|
||||
|
||||
1. **Starting capital ($35k) per [[project_ml_alpha_starting_capital]]** — should we add `RL_STARTING_CAPITAL_USD_INDEX` as an ISV slot? Yes, makes session DD cap and Layer 2 drawdown ratio both adaptive to capital changes. Add to Layer 1.
|
||||
|
||||
2. **Layer 2 sensitivity (5.0)** — at session DD 10%, τ drops from 0.5 to 0. At 4% drawdown, τ ≈ 0.3. Reasonable? Tunable via ISV. May need empirical sweep.
|
||||
|
||||
3. **Inventory variance EMA — measured over what window?** Need to choose Wiener-α. Default `RL_WIENER_ALPHA_FLOOR_INDEX = 0.4` from existing pearl. Adaptive could come later.
|
||||
|
||||
4. **Layer 3 β bootstrap (0.0 = sentinel)** — agent gets ZERO inventory penalty until variance EMA accumulates. Acceptable cold-start.
|
||||
|
||||
5. **Should `RL_MAX_OPEN_UNITS_INDEX` = 4 (current `MAX_UNITS`) be redundant?** The action space already enforces this via unit_active tracking. Yes, redundant but cheap; treats it as belt-and-braces.
|
||||
|
||||
6. **Approach E vs the original B+D — anything we lose?** The `rl_session_drawdown_check.cu` kernel from the prior design is now folded into the broader `rl_cmdp_constraints_check.cu`. Same functionality, more layered.
|
||||
|
||||
7. **Multi-seed validation?** A single fold-1 result is noisy (1024 trades, high variance). Worth running 3 seeds to confirm. Defer to post-validation cluster sweep if G13 single-seed result is marginal.
|
||||
|
||||
8. **Regime detection (Approach E+1)?** FineFT (KDD 2026) demonstrates risk-aware ensemble routing on regime changes. Heavyweight implementation (online clustering + meta-controller). Defer to follow-up spec if Approach E doesn't close the eval gap.
|
||||
|
||||
---
|
||||
|
||||
## Risks and mitigations
|
||||
|
||||
- **Layer 2 τ-quantile path may not be exposed by current IQN kernel signature** → audit `rl_ensemble_action_value.cu` first; if `Q(s, a, τ)` isn't readily indexable, may need kernel restructure (small change).
|
||||
|
||||
- **Kelly EMA estimates unreliable in early training** → warmup gate (1000 trades) holds Kelly at bootstrap. Same pattern as b1ef6664a's reward_floor cumulative-dones gate.
|
||||
|
||||
- **Five layers compounding may make agent ultra-conservative — eval might be 0 with no trades** → that's actually fine for G13 (eval pnl > -$200k bar). But for G14 (ksll2 +$13M), need to verify layers don't over-restrict on full-data regime. Mitigation: warmup gates default to "behave like the prior code" until enough signal accumulates.
|
||||
|
||||
- **Session boundary resets must be wired by trainer** → must call ISV resets on train→eval transition. Test G11 catches this. Plan task explicitly enumerates the reset points.
|
||||
|
||||
- **Approach D wiring changes action distribution dynamics** → trainable but worth a regression check on entropy. 1k smoke catches it.
|
||||
|
||||
- **CMDP "force Hold" creates discontinuity in policy gradient** → standard technique in CMDP-RL literature, well-understood. Override doesn't update gradient through the constraint, only through the Hold action's natural Q.
|
||||
|
||||
---
|
||||
|
||||
## Next step after approval
|
||||
|
||||
Write the plan: `docs/superpowers/plans/2026-05-30-adaptive-risk-management-plan.md`. Task-level decomposition:
|
||||
|
||||
1. ISV slot allocation (22 slots)
|
||||
2. Layer 1 (CMDP constraints kernel + actions_to_market_targets overrides)
|
||||
3. Layer 2 (IQN τ controller + action-selection kernel modification)
|
||||
4. Layer 3 (inventory penalty in reward + β controller)
|
||||
5. Layer 4 (Kelly controller + actions_to_market_targets sizing)
|
||||
6. Layer D (TrailTighten/Loosen wiring)
|
||||
7. New EMA producers (win_rate, avg_win, avg_loss, inventory variance)
|
||||
8. Fused-kernel integration of new controllers
|
||||
9. Trainer integration (per-step launches + bootstrap + session reset)
|
||||
10. GPU-oracle invariant tests (G1-G11)
|
||||
11. 1k local smoke + sanitizer (G12)
|
||||
12. Atomic commit
|
||||
13. Cluster G13 (fold 1 re-run) + G14 (ksll2 regression)
|
||||
|
||||
Expected scope: ~320 LOC across 8 new + 7 modified files, ~13 tasks, single atomic commit. Same `feedback_no_partial_refactor` discipline.
|
||||
Reference in New Issue
Block a user