spec(ml-backtesting): CBSW review pass — 11 fixes (perf + correctness)
Critical review surfaced 11 actionable issues in the v1 spec; 10 fixed inline (#11 — legacy-comparison test — dropped per user decision since empirical legacy behavior is already known: n_trades=0). Performance: 1. expf → piecewise-linear ramp (~5× cheaper on GPU, no transcendental in the hot path). Operationally equivalent: monotonic, saturating, midpoint at K. Side-benefit: pure max-confidence at cold-start (sq=0 instead of sigmoid's 11.9% leakage). 2. Single fused per-horizon pass replaces the two-loop sketch. 3. Tier 2 scope narrowed to decision_policy_default ONLY. The bytecode VM (decision_policy_program) stays unchanged — runs only for custom strategy experiments, never in production policy. Halves Tier 2's surface area + test cost. 4. __device__ helpers cbsw_signal_quality + cbsw_weight introduced for single source of truth. Correctness bugs (silently present in v1 sketch, would have shipped): 5. strong_h and sq_min_active initializers added (were referenced before init in the per-horizon loop). 6. Attribution mask at cold-start was setting all 5 bits because every w[h] >= floor > 1e-9; trades would pollute ALL horizons' recent_sharpe. Fix: binary split — at sq_min_active < 0.5 attribute only to strong_h; at mature, attribute per weights > floor + epsilon. 7. sq_min was "min over h", which permanently locked the aggregator into cold-start mode if any horizon never gets attributed (e.g., h6000-only-trading regime). Fix: sq_min_active = min over h with n_trades_seen > 0; cold horizons don't gate maturity. 8. Opposing-horizons case now correctly fires on the strongest single horizon at cold-start (piecewise-linear sq=0 → pure max-conf), with explicit design-choice note explaining why this conservatism trade- off favors firing. Spec hygiene: 9. Rate validation committed to a measurement gate (Q2 ev/s ≥ 95% of Q1 ev/s) instead of back-of-envelope estimate. 10. Kernel ABI explicitly stated as unchanged → feedback_no_partial_ refactor compliance is trivial at the kernel boundary. 12. Tier 1's use_cold_start_stopgap field is DROPPED in Q2 atomically (not orphaned), and DoD checklist includes a grep-zero gate for feedback_no_legacy_aliases compliance. Risks updated: removed the sigmoid-narrowing risk (irrelevant now); added register-pressure risk with the rate-gate mitigation; added explicit "cold-start may lose on noisy trades" risk with empirical escalation path (bump kelly_floor 0.20 → 0.40 if Q2 smoke shows large negative PnL). Math walk-throughs rewritten for piecewise-linear (different numbers at cold-start): cold = pure max-conf, mature = pure weighted-sharpe, clean separation at sq_min_active threshold. Awaiting review of the revised spec before writing-plans dispatch. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
@@ -69,19 +69,21 @@ CBSW changes both the **weight formula** AND the **aggregation type**, transitio
|
||||
|
||||
```
|
||||
n_h = isv[h].n_trades_seen
|
||||
K = MIN_TRADES_FOR_VAR_CAP = 10 (already exists, sample-size threshold)
|
||||
sigmoid_scale = 5.0 (half-transition width)
|
||||
signal_quality[h] = sigmoid((n_h - K) / sigmoid_scale)
|
||||
K = MIN_TRADES_FOR_VAR_CAP = 10 (already exists, sample-size threshold)
|
||||
RAMP_HALF_WIDTH = 5.0 (half-width of linear transition zone)
|
||||
signal_quality[h] = clamp((n_h − K + RAMP_HALF_WIDTH) / (2 × RAMP_HALF_WIDTH), 0, 1)
|
||||
```
|
||||
|
||||
Properties:
|
||||
- `n=0` → `sq = sigmoid(-2) ≈ 0.119` (cold)
|
||||
- `n=K-5=5` → `sq = sigmoid(-1) ≈ 0.269` (transitioning)
|
||||
- `n=K=10` → `sq = sigmoid(0) = 0.500` (mid-transition)
|
||||
- `n=K+5=15` → `sq = sigmoid(+1) ≈ 0.731` (transitioning)
|
||||
- `n=K+10=20` → `sq = sigmoid(+2) ≈ 0.881` (nearly mature)
|
||||
**Piecewise-linear ramp, not sigmoid.** Operationally identical (monotonic + saturating + same midpoint), but ~5× cheaper on GPU: 2 `fmaxf/fminf` + 1 division per horizon vs 1 `expf` + arithmetic. Operational smoothness of sigmoid is mathematically pleasing but irrelevant here — Kelly aggregation doesn't differentiate the weight.
|
||||
|
||||
Smooth, monotonic, signal-driven by `n_trades_seen` (the ISV-side sample count). Reuses the existing `MIN_TRADES_FOR_VAR_CAP` threshold so there's no new tuning knob.
|
||||
Properties:
|
||||
- `n=0` → `sq = clamp(−0.5,0,1) = 0.000` (pure cold)
|
||||
- `n=K−5=5` → `sq = 0.000` (start of ramp)
|
||||
- `n=K=10` → `sq = 0.500` (mid-transition)
|
||||
- `n=K+5=15` → `sq = 1.000` (fully mature)
|
||||
- `n=K+10=20` → `sq = 1.000` (saturated)
|
||||
|
||||
Cold-start is now **pure max-confidence** (sq=0 → conviction weight only, ss_strong only in aggregator). This is cleaner than the sigmoid's 11.9% leakage at n=0. Reuses the existing `MIN_TRADES_FOR_VAR_CAP` threshold so there's no new tuning knob.
|
||||
|
||||
#### Weight formula (blended per-horizon)
|
||||
|
||||
@@ -100,69 +102,103 @@ At cold-start the weight = `max(floor, sig_mag[h])` — strong-conviction horizo
|
||||
Weighted-mean alone can't escape the dilution bound. We need max-confidence at cold-start. Hybrid:
|
||||
|
||||
```
|
||||
strong_h = argmax_h |ss[h]| // max-confidence pick
|
||||
ss_strong = ss[strong_h]
|
||||
ss_weighted = Σ w_norm[h] × ss[h]
|
||||
sq_min = min over h of signal_quality[h] // transition gate
|
||||
final_size = (1 - sq_min) * ss_strong + sq_min * ss_weighted
|
||||
strong_h = argmax_h |ss[h]| // max-confidence pick
|
||||
ss_strong = ss[strong_h]
|
||||
ss_weighted = Σ (w[h] / w_sum) × ss[h] // weighted-mean piece
|
||||
sq_min_active = min over h with n_trades_seen[h] > 0 of signal_quality[h]
|
||||
(defaults to 0.0 when NO horizon has any trades)
|
||||
final_size = (1 − sq_min_active) × ss_strong + sq_min_active × ss_weighted
|
||||
```
|
||||
|
||||
At cold-start: 100% max-confidence → strong-horizon-only trades fire.
|
||||
At mature: 100% weighted-sharpe → existing behavior.
|
||||
Transition: smooth blend over the same K=10-trade threshold.
|
||||
**Why `sq_min_active` and not `sq_min`**: if only one horizon ever contributes to closes (e.g., h6000 = the AUC-best horizon does all the trading), `n_trades_seen` for other horizons stays at 0 forever. With a naive `sq_min`, the aggregator would be **permanently** stuck in cold-start mode despite h6000 being fully mature. Restricting the min to horizons that have *actually contributed* fixes this: only horizons with real sample history gate the transition. Horizons with n=0 are silently excluded — they don't get to veto maturity.
|
||||
|
||||
At cold-start (no closes anywhere): `sq_min_active = 0.0` → 100% max-confidence → strong-horizon-only trades fire.
|
||||
At fully mature (all h have closes): `sq_min_active = min sq_h` → weighted-mean takes over.
|
||||
At partial mature (only h6000 has closes): `sq_min_active = sq_{h6000}` → if it's mature, aggregator is mature.
|
||||
|
||||
**Attribution mask** (which horizons get credited on trade close, drives `isv_kelly_update_on_close`):
|
||||
|
||||
```
|
||||
if (sq_min_active < 0.5):
|
||||
attribution_mask = (1u << strong_h) // cold-start: blame the strong horizon only
|
||||
else:
|
||||
attribution_mask = OR over h with w[h] > sharpe_weight_floor + ε of (1u << h)
|
||||
// mature: credit horizons that contributed signal beyond the floor
|
||||
```
|
||||
|
||||
The cold-start branch fixes a silent bug in the v1 sketch: previously `attribution_mask` was set whenever `w[h] > 1e-9`, which was always true (floor = 0.10 ≫ 1e-9), so cold-start trades polluted ALL 5 horizons' `recent_sharpe`. With the binary split, the strong horizon takes the credit it deserves; other horizons are unaffected by trades they didn't drive.
|
||||
|
||||
**Design choice — opposing-horizon trades at cold-start**: with piecewise-linear `sq=0`, the aggregator at cold-start is pure max-confidence. For `alpha = [0.75, 0.30, 0.50, 0.50, 0.50]` (one bullish, one bearish, rest neutral), CBSW fires a long on h0's signal — h1's opposing bearish signal is ignored. This is **deliberate**: at cold-start the model has no historical performance data to use as a tiebreaker, so the strongest single-horizon conviction wins. Once `recent_sharpe[h]` is populated, the mature-regime weighted aggregator will properly penalize horizons that have historically lost on these patterns.
|
||||
|
||||
The alternative (suppress trades on opposing horizons) would echo the consensus-requirement bug. We explicitly choose to fire on single-horizon strong signals at cold-start, with the understanding that some of these trades will lose and `recent_sharpe[h]` will adjust accordingly.
|
||||
|
||||
### Math walk-through (cold-start, single-horizon strong)
|
||||
|
||||
`alpha = [0.85, 0.50, 0.50, 0.50, 0.50]`, n=0 everywhere, sq_min ≈ 0.119:
|
||||
`alpha = [0.85, 0.50, 0.50, 0.50, 0.50]`, n=0 everywhere → `sq_min_active = 0.0`:
|
||||
|
||||
| Quantity | Value | Note |
|
||||
|---|---|---|
|
||||
| `sig_mag` | `[0.70, 0, 0, 0, 0]` | |
|
||||
| `dir` | `[+1, +1, +1, +1, +1]` | doesn't matter for sig_mag=0 |
|
||||
| `ss = dir × sig_mag × 0.20 × 5` | `[0.70, 0, 0, 0, 0]` | kelly_floor=0.20, cap=max_lots=5 |
|
||||
| `w` (with floor) | `[max(0.10, 0.70), 0.10, 0.10, 0.10, 0.10]` = `[0.70, 0.10, 0.10, 0.10, 0.10]` | sig_mag → high weight on strong h |
|
||||
| `w_sum` | 1.10 | |
|
||||
| `w_norm` | `[0.636, 0.091, 0.091, 0.091, 0.091]` | |
|
||||
| `ss_weighted` | `0.636 × 0.70 + 0 = 0.445` | linear-mean piece |
|
||||
| `ss = dir × sig_mag × 0.20 × 5` | `[+0.70, 0, 0, 0, 0]` | kelly_floor=0.20, cap=max_lots=5 |
|
||||
| `w` (with floor) at sq=0 | `[max(0.10, 0.70), 0.10, 0.10, 0.10, 0.10]` = `[0.70, 0.10, 0.10, 0.10, 0.10]` | pure conviction → strong h gets full sig_mag |
|
||||
| `ss_strong` | `0.70` | max-conf pick |
|
||||
| `final_size` | `(1 − 0.119) × 0.70 + 0.119 × 0.445 = 0.616 + 0.053 = 0.669` | |
|
||||
| `lots = truncf(0.669 + 0.5)` | **1** | trade fires |
|
||||
| `final_size` | `(1 − 0) × 0.70 + 0 × ss_weighted = 0.70` | pure max-confidence at cold-start |
|
||||
| `lots = truncf(0.70 + 0.5)` | **1** | trade fires |
|
||||
| `attribution_mask` (sq < 0.5) | `(1u << 0) = 0b00001` | only h0 gets credited on close |
|
||||
|
||||
Single-horizon strong signal at cold-start now fires a 1-lot trade.
|
||||
Single-horizon strong signal at cold-start fires a 1-lot trade, attributed solely to the convicted horizon.
|
||||
|
||||
### Math walk-through (weak conflict)
|
||||
### Math walk-through (opposing horizons at cold-start)
|
||||
|
||||
`alpha = [0.55, 0.45, 0.55, 0.45, 0.55]`, n=0, sq_min ≈ 0.119:
|
||||
`alpha = [0.75, 0.30, 0.50, 0.50, 0.50]` (h0 bullish, h1 bearish, rest neutral), n=0 → `sq_min_active = 0.0`:
|
||||
|
||||
| Quantity | Value |
|
||||
|---|---|
|
||||
| `sig_mag` | `[0.50, 0.40, 0, 0, 0]` |
|
||||
| `dir` | `[+1, −1, +1, +1, +1]` |
|
||||
| `ss` | `[+0.50, −0.40, 0, 0, 0]` |
|
||||
| `strong_h` (argmax \|ss\|) | h=0, `ss_strong = +0.50` |
|
||||
| `final_size` (pure max-conf) | `0.50` |
|
||||
| `lots` | **1** (long h0's signal) |
|
||||
| `attribution_mask` | `(1u << 0)` |
|
||||
|
||||
Cold-start fires on the strongest single-horizon signal even when other horizons disagree. Per the design-choice note: subsequent close attributes loss/win exclusively to h0, letting `recent_sharpe[0]` adjust.
|
||||
|
||||
### Math walk-through (weak conflict at cold-start)
|
||||
|
||||
`alpha = [0.55, 0.45, 0.55, 0.45, 0.55]`, n=0 → sq=0:
|
||||
|
||||
| Quantity | Value |
|
||||
|---|---|
|
||||
| `sig_mag` | `[0.10, 0.10, 0.10, 0.10, 0.10]` |
|
||||
| `dir` | `[+1, −1, +1, −1, +1]` |
|
||||
| `ss` | `[+0.10, −0.10, +0.10, −0.10, +0.10]` |
|
||||
| `strong_h` (argmax of \|ss\|) | any (ties); pick h=0; `ss_strong = +0.10` |
|
||||
| `w` | `[max(0.10, 0.10)] × 5 = [0.10] × 5` (all equal) |
|
||||
| `w_norm` | `[0.20] × 5` |
|
||||
| `ss_weighted` | `0.20 × (0.10 − 0.10 + 0.10 − 0.10 + 0.10) = 0.02` |
|
||||
| `final_size` | `0.881 × 0.10 + 0.119 × 0.02 = 0.091` |
|
||||
| `lots = truncf(0.091 + 0.5)` | **0** → noop |
|
||||
| `strong_h` (first tie) | h=0; `ss_strong = +0.10` |
|
||||
| `final_size` | `0.10` |
|
||||
| `lots = truncf(0.10 + 0.5)` | **0** → noop |
|
||||
|
||||
Weak conflicting signals correctly don't trade.
|
||||
Weak signals correctly fail to clear the lots≥1 threshold. The model needs at least one horizon with conviction ≥ ~0.5 to fire 1 lot at cold-start (sig_mag × kelly_floor × cap_lots > 0.5 → sig_mag > 0.5 when kelly_floor=0.20, max_lots=5).
|
||||
|
||||
### Math walk-through (mature regime, n=20 everywhere)
|
||||
### Math walk-through (h6000-only mature regime)
|
||||
|
||||
`alpha = [0.85, 0.50, 0.50, 0.50, 0.50]`, sq_min ≈ 0.881, `recent_sharpe = [1.5, 0.1, 0.1, 0.1, 0.1]` (model converged: h0 is the best horizon):
|
||||
After 50 trades, all attributed to h6000 (the AUC-best horizon at training). `n_trades_seen = [0, 0, 0, 0, 50]`, `recent_sharpe = [0, 0, 0, 0, 1.5]`. `sq[h] = [0, 0, 0, 0, 1.0]`. `sq_min_active = min over active h = sq[4] = 1.0` (only h=4 is active).
|
||||
|
||||
`alpha = [0.85, 0.50, 0.50, 0.50, 0.85]` (h0 and h6000 both strong-long this decision):
|
||||
|
||||
| Quantity | Value |
|
||||
|---|---|
|
||||
| `w_blend` | `0.119 × sig_mag + 0.881 × sharpe = [0.083+1.322, 0.0+0.088, ...] = [1.405, 0.088, 0.088, 0.088, 0.088]` |
|
||||
| `w` (with floor) | `[1.405, 0.10, 0.10, 0.10, 0.10]` |
|
||||
| `w_norm` | `[0.778, 0.055, 0.055, 0.055, 0.055]` |
|
||||
| `ss_weighted` | `0.778 × 0.70 = 0.544` |
|
||||
| `ss_strong` | `0.70` |
|
||||
| `final_size` | `0.119 × 0.70 + 0.881 × 0.544 = 0.083 + 0.479 = 0.562` |
|
||||
| `lots` | **1** |
|
||||
| `sig_mag` | `[0.70, 0, 0, 0, 0.70]` |
|
||||
| `ss` | `[+0.70, 0, 0, 0, +0.70]` |
|
||||
| `w` (sq[h] varies) | `[max(0.10, 1.0 × 0.70 + 0.0 × 0), 0.10, 0.10, 0.10, max(0.10, 0.0 × 0.70 + 1.0 × 1.5)]` = `[0.70, 0.10, 0.10, 0.10, 1.50]` |
|
||||
| `w_sum` | 2.50 |
|
||||
| `w_norm` | `[0.28, 0.04, 0.04, 0.04, 0.60]` |
|
||||
| `ss_weighted` | `0.28 × 0.70 + 0.60 × 0.70 = 0.196 + 0.420 = 0.616` |
|
||||
| `ss_strong` (tie at \|ss\|=0.70, picks first) | h=0; `ss_strong = +0.70` |
|
||||
| `final_size` | `(1 − 1.0) × 0.70 + 1.0 × 0.616 = 0.616` | pure weighted-sharpe |
|
||||
| `lots = truncf(0.616 + 0.5)` | **1** |
|
||||
| `attribution_mask` (sq ≥ 0.5) | `(1u << 0) \| (1u << 4) = 0b10001` (both w[0] and w[4] > floor) |
|
||||
|
||||
Mature regime preserves the asymmetric Sharpe-driven weighting. Strong-Sharpe horizon dominates correctly.
|
||||
Mature regime correctly emphasizes h6000 (1.5 sharpe weight dominates h0's cold conviction weight of 0.70). Both horizons get attributed on close because both contributed signal beyond the floor — this lets h0's `recent_sharpe` start tracking from this trade, gradually accelerating its transition out of cold-start.
|
||||
|
||||
---
|
||||
|
||||
@@ -196,75 +232,123 @@ When `program_lens[b] > 0`, the bytecode kernel handles the backtest and the def
|
||||
|
||||
## 4. Tier 2 — Kernel CBSW (proper architecture)
|
||||
|
||||
**Goal:** Replace both decision kernels' weight + aggregation logic with the CBSW formula from §2. Tier 1's bytecode stopgap becomes unnecessary and is reverted in the same commit.
|
||||
**Goal:** Replace `decision_policy_default`'s weight + aggregation logic with the CBSW formula from §2. Tier 1's bytecode stopgap becomes unnecessary and is reverted in the same commit (`use_cold_start_stopgap` field dropped entirely per `feedback_no_legacy_aliases`).
|
||||
|
||||
**Kernel signature additions** (both `decision_policy_default` and `decision_policy_program`):
|
||||
**Scope: `decision_policy_default` ONLY.** `decision_policy_program` is left unchanged because it runs only when `program_lens[b] > 0` (custom bytecode strategies), which the production deployment never sets. Modifying both kernels doubles surface area + test cost for zero production benefit. The bytecode VM remains a pluggable experiment surface — including Tier 1's max-confidence pattern, which can stay in `decision_policy_program` indefinitely as a documented strategy preset.
|
||||
|
||||
**Kernel ABI unchanged.** No new args added. The two new constants are file-scope `#define`s:
|
||||
|
||||
```cuda
|
||||
// New per-backtest scalar args (kept symmetric with existing P4 floors)
|
||||
const float* sigmoid_scale_per_b, // = 5.0 default
|
||||
int min_trades_for_trust_per_b // existing MIN_TRADES_FOR_VAR_CAP, now per-b
|
||||
// File-scope, decision_policy.cu top:
|
||||
#define MIN_TRADES_FOR_TRUST 10u // matches MIN_TRADES_FOR_VAR_CAP — sample-size threshold for trusting historical recent_sharpe
|
||||
#define RAMP_HALF_WIDTH 5.0f // half-width of CBSW linear transition; 0 below K-half, 1 above K+half
|
||||
```
|
||||
|
||||
Actually — to minimize per-call upload overhead and keep the kernel ABI tight, use file-scope `#define` for both:
|
||||
Both are sample-size constants, not adaptive bounds — they satisfy `pearl_isv_for_adaptive_bounds.md` (the pearl flags adaptive bounds, not statistical thresholds). Per `feedback_no_partial_refactor`: kernel ABI unchanged means no consumer migration; Tier 2 is a pure kernel-body change.
|
||||
|
||||
**`__device__` helper for CBSW weight** (avoids duplication if/when other kernels need the same logic):
|
||||
|
||||
```cuda
|
||||
#define MIN_TRADES_FOR_TRUST 10u
|
||||
#define SIGMOID_SCALE 5.0f
|
||||
```
|
||||
|
||||
(Both are sample-size thresholds — not adaptive bounds — so they satisfy `pearl_isv_for_adaptive_bounds.md`. Constants reused across the codebase: same K=10 as `MIN_TRADES_FOR_VAR_CAP`.)
|
||||
|
||||
**Kernel body changes:**
|
||||
|
||||
```cuda
|
||||
// Inside the per-horizon loop, compute sig_mag and signal_quality:
|
||||
const float n_h = (float)isv[h].n_trades_seen;
|
||||
const float sq_h = 1.0f / (1.0f + expf(-(n_h - (float)MIN_TRADES_FOR_TRUST) / SIGMOID_SCALE));
|
||||
|
||||
// Weight: blend sig_mag (cold) and recent_sharpe (mature), with permanent floor.
|
||||
const float sharpe_w = fmaxf(0.0f, isv[h].recent_sharpe);
|
||||
const float conv_w = sig_mag; // already computed for ss[h]
|
||||
const float w_blend = (1.0f - sq_h) * conv_w + sq_h * sharpe_w;
|
||||
weights[h] = fmaxf(sharpe_weight_floor, w_blend);
|
||||
w_sum += weights[h];
|
||||
|
||||
// Track strong_h for max-confidence aggregation
|
||||
if (fabsf(signed_sizes[h]) > fabsf(signed_sizes[strong_h])) {
|
||||
strong_h = h;
|
||||
__device__ static inline float cbsw_signal_quality(unsigned int n_trades_seen) {
|
||||
const float n = (float)n_trades_seen;
|
||||
const float lo = (float)MIN_TRADES_FOR_TRUST - RAMP_HALF_WIDTH; // 5.0
|
||||
const float span = 2.0f * RAMP_HALF_WIDTH; // 10.0
|
||||
return fmaxf(0.0f, fminf(1.0f, (n - lo) / span));
|
||||
}
|
||||
|
||||
__device__ static inline float cbsw_weight(float sig_mag, float recent_sharpe,
|
||||
unsigned int n_trades_seen, float floor) {
|
||||
const float sq = cbsw_signal_quality(n_trades_seen);
|
||||
const float sharpe_w = fmaxf(0.0f, recent_sharpe);
|
||||
const float w_blend = (1.0f - sq) * sig_mag + sq * sharpe_w;
|
||||
return fmaxf(floor, w_blend);
|
||||
}
|
||||
sq_min = fminf(sq_min, sq_h); // gate for the hybrid aggregator
|
||||
```
|
||||
|
||||
**Kernel body — single fused per-horizon pass** (replaces the existing weight + aggregation block in `decision_policy_default`):
|
||||
|
||||
```cuda
|
||||
// After the per-horizon loop, hybrid aggregation:
|
||||
float final_size = 0.0f;
|
||||
float signed_sizes[N_HORIZONS];
|
||||
float weights[N_HORIZONS];
|
||||
int strong_h = 0; // init explicit (was implicit-bug in v1 sketch)
|
||||
float strong_abs = 0.0f;
|
||||
float w_sum = 0.0f;
|
||||
float sq_min_active = 1.0f; // init at max; only horizons with n>0 reduce this
|
||||
int n_active_horizons = 0;
|
||||
|
||||
#pragma unroll
|
||||
for (int h = 0; h < N_HORIZONS; ++h) {
|
||||
const float p_h = alpha_probs[h];
|
||||
const IsvKellyState& s = isv[h];
|
||||
|
||||
const float sig_mag = fabsf(p_h - 0.5f) * 2.0f;
|
||||
const float dir = (p_h > 0.5f) ? 1.0f : -1.0f;
|
||||
const float kelly_frac = compute_kelly_frac(s, kelly_frac_floor); // existing logic
|
||||
const float cap_lots = compute_cap_lots(s, target_annual_vol_units,
|
||||
annualisation_factor, max_lots); // existing logic
|
||||
signed_sizes[h] = dir * sig_mag * kelly_frac * cap_lots;
|
||||
|
||||
weights[h] = cbsw_weight(sig_mag, s.recent_sharpe, s.n_trades_seen, sharpe_weight_floor);
|
||||
w_sum += weights[h];
|
||||
|
||||
// max-confidence pick: track strong_h by |ss|
|
||||
const float abs_ss = fabsf(signed_sizes[h]);
|
||||
if (abs_ss > strong_abs) { strong_abs = abs_ss; strong_h = h; }
|
||||
|
||||
// sq_min over ACTIVE horizons only (n_trades_seen > 0); cold horizons don't gate maturity
|
||||
if (s.n_trades_seen > 0u) {
|
||||
sq_min_active = fminf(sq_min_active, cbsw_signal_quality(s.n_trades_seen));
|
||||
n_active_horizons += 1;
|
||||
}
|
||||
}
|
||||
// If no horizon has any closes yet, the aggregator is pure max-confidence.
|
||||
if (n_active_horizons == 0) sq_min_active = 0.0f;
|
||||
|
||||
// Hybrid aggregator
|
||||
float ss_weighted = 0.0f;
|
||||
if (w_sum > 1e-9f) {
|
||||
float ss_weighted = 0.0f;
|
||||
#pragma unroll
|
||||
for (int h = 0; h < N_HORIZONS; ++h) {
|
||||
ss_weighted += (weights[h] / w_sum) * signed_sizes[h];
|
||||
if (weights[h] > 1e-9f) attribution_mask |= (1u << h);
|
||||
}
|
||||
const float ss_strong = signed_sizes[strong_h];
|
||||
final_size = (1.0f - sq_min) * ss_strong + sq_min * ss_weighted;
|
||||
}
|
||||
const float ss_strong = signed_sizes[strong_h];
|
||||
const float final_size = (1.0f - sq_min_active) * ss_strong
|
||||
+ sq_min_active * ss_weighted;
|
||||
|
||||
// Attribution: cold → strong_h only; mature → all horizons above floor + ε
|
||||
unsigned int attribution_mask;
|
||||
if (sq_min_active < 0.5f) {
|
||||
attribution_mask = (1u << strong_h);
|
||||
} else {
|
||||
attribution_mask = 0u;
|
||||
#pragma unroll
|
||||
for (int h = 0; h < N_HORIZONS; ++h) {
|
||||
if (weights[h] > sharpe_weight_floor + 1e-6f) attribution_mask |= (1u << h);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Single forward pass + one short aggregation loop (5 elements, fully unrolled by the compiler). No `expf`. The `cbsw_signal_quality` is called twice per horizon (once in `cbsw_weight`, once in the sq_min reduce) but both calls inline trivially and the result is shared in registers by the compiler.
|
||||
|
||||
**Files:**
|
||||
- `crates/ml-backtesting/cuda/decision_policy.cu` — both kernels (default + program). ~30 LOC each.
|
||||
- `crates/ml-backtesting/src/harness.rs` + sim — revert the Tier 1 stopgap bytecode upload.
|
||||
- `crates/ml-backtesting/cuda/decision_policy.cu` — `decision_policy_default` body change + 2 `__device__` helpers + `#define`s. ~50 LOC.
|
||||
- `crates/ml-backtesting/src/harness.rs` — revert the Tier 1 stopgap bytecode upload, **drop** the `use_cold_start_stopgap` field entirely from `BacktestHarnessConfig` + `UniformSimParams`.
|
||||
- `bin/fxt-backtest/src/main.rs` — drop the `--cold-start-stopgap` CLI flag.
|
||||
- `config/ml/sweep_smoke.yaml` — drop `use_cold_start_stopgap: true` from the variant.
|
||||
- `crates/ml-backtesting/tests/cbsw_aggregator.rs` (NEW) — unit tests per §2 math walk-throughs.
|
||||
|
||||
**Tests:**
|
||||
|
||||
| Test | Setup | Assertion |
|
||||
|---|---|---|
|
||||
| `cbsw_cold_start_single_horizon_strong_fires_trade` | n=0 all h, alpha=[0.85,0.50,0.50,0.50,0.50] | side=0, size ≥ 1 |
|
||||
| `cbsw_cold_start_single_horizon_strong_fires_trade` | n=0 all h, alpha=[0.85,0.50,0.50,0.50,0.50] | side=0, size≥1, attribution_mask = 0b00001 |
|
||||
| `cbsw_cold_start_opposing_horizons_fires_on_strongest` | n=0, alpha=[0.75,0.30,0.50,0.50,0.50] | side=0, size≥1, attribution_mask = 0b00001 |
|
||||
| `cbsw_cold_start_weak_conflict_does_not_trade` | n=0, alpha=[0.55,0.45,0.55,0.45,0.55] | side=2, size=0 |
|
||||
| `cbsw_cold_start_consensus_strong_fires_trade` | n=0, alpha=[0.85]×5 | side=0, size ≥ 1 |
|
||||
| `cbsw_mature_recovers_weighted_sharpe` | n=20 all h, recent_sharpe=[1.5,0.1,0.1,0.1,0.1], alpha=[0.85,0.50,0.50,0.50,0.50] | side=0, size ≥ 1, ss_weighted contribution > 0.5 of final |
|
||||
| `cbsw_transition_smooth` | n=K all h, alpha=[0.70,0.50,0.50,0.50,0.50] | size monotone in n_trades_seen sweep |
|
||||
| `cbsw_cold_start_consensus_strong_fires_trade` | n=0, alpha=[0.85]×5 | side=0, size≥1 |
|
||||
| `cbsw_mature_recovers_weighted_sharpe` | n=20 all h, recent_sharpe=[1.5,0.1,0.1,0.1,0.1], alpha=[0.85,0.50,0.50,0.50,0.50] | side=0, size≥1, attribution_mask has all 5 bits set, ss_weighted contribution > 90% of final_size |
|
||||
| `cbsw_h6000_only_active_does_not_block_maturity` | n_trades_seen=[0,0,0,0,50], recent_sharpe=[0,0,0,0,1.5], alpha=[0.85,0.50,0.50,0.50,0.85] | sq_min_active = 1.0 (only h=4 active), final_size ≈ ss_weighted (mature regime), trade fires |
|
||||
| `cbsw_transition_smooth` | sweep n_trades_seen=[0..20], alpha=[0.70,0.50,0.50,0.50,0.50] | size monotone non-decreasing in n_trades_seen, no discontinuity at n=K=10 |
|
||||
|
||||
**Validation gate:** Re-run threshold-tuning smoke at threshold=0.0, cost=0.125. Expected `n_trades > 100`, `total_pnl_usd != 0`. Cross-check that Tier 1 and Tier 2 produce similar n_trades counts at cold-start (both should fire on single-horizon strong signals). Tier 2 should be *better* than Tier 1 once trades start closing (recent_sharpe takes over, max-confidence over-trading subsides).
|
||||
|
||||
@@ -306,16 +390,16 @@ If found, log as separate spec follow-ups. Don't expand THIS spec's scope.
|
||||
|
||||
## 6. Migration plan (atomic commits)
|
||||
|
||||
Per `feedback_no_partial_refactor`: each commit is internally consistent and ships its own tests.
|
||||
Per `feedback_no_partial_refactor`: each commit is internally consistent and ships its own tests. **Kernel ABI is unchanged across all four commits** — Tier 1 uses the existing `decision_policy_program` kernel as-is (bytecode upload only); Tier 2 changes `decision_policy_default`'s body but not its signature. No consumer migration required at the kernel boundary.
|
||||
|
||||
| Commit | Subject | Files | Test gate |
|
||||
|---|---|---|---|
|
||||
| Q1 | `feat(ml-backtesting): cold-start stopgap — max-confidence bytecode policy (Tier 1)` | sim/batched_config.rs (+ use_cold_start_stopgap flag), harness.rs (upload bytecode program when flag set), main.rs (CLI flag), sweep_smoke.yaml (set flag) | Local: harness builds bytecode program of correct shape. Cluster: smoke fires `n_trades > 100`. |
|
||||
| Q2 | `feat(ml-backtesting): CBSW kernel aggregator (Tier 2)` | decision_policy.cu (both kernels), tests/cbsw_aggregator.rs (5 new tests), harness.rs (revert Tier 1 stopgap upload, drop flag) | Local: all 5 cbsw_* tests pass. Cluster: smoke fires `n_trades > 100`. |
|
||||
| Q1 | `feat(ml-backtesting): cold-start stopgap — max-confidence bytecode policy (Tier 1)` | sim/batched_config.rs (+ `use_cold_start_stopgap: bool` field on `UniformSimParams` and `BatchedSimConfig`), harness.rs (when flag set, build 7-instr bytecode program via `Strategy::flatten()` and upload via existing `sim.upload_program()` API), main.rs (CLI flag), sweep_smoke.yaml (set flag) | Local: existing `parallel_sim_correctness::*` + `threshold_and_cost::*` continue to pass (smoke-flag default false, no regression). Cluster: smoke fires `n_trades > 100`, `total_pnl_usd != 0`. |
|
||||
| Q2 | `feat(ml-backtesting): CBSW kernel aggregator (Tier 2)` | decision_policy.cu (`decision_policy_default` body + 2 new `__device__` helpers + 2 `#define`s; `decision_policy_program` UNCHANGED), tests/cbsw_aggregator.rs (7 new tests), sim/batched_config.rs (drop `use_cold_start_stopgap` field entirely), harness.rs (drop the upload-when-flag-set branch), main.rs (drop the CLI flag), sweep_smoke.yaml (drop the flag setting) | Local: all 7 cbsw_* tests pass; pre-existing `decision_floor_coldstart`, `parallel_sim_correctness`, `threshold_and_cost`, `forward_graph_capture`, `save_load_roundtrip_preserves_all_weights` all still pass. Cluster: smoke fires `n_trades > 100`, `total_pnl_usd != 0`. |
|
||||
| Q3 | `docs(memory): pearl_conviction_bootstrap_for_kelly_aggregation` | new memory pearl file + MEMORY.md index update | Manual review. |
|
||||
| Q4 | `spec(ml-alpha): deployability sweep depends on CBSW` | append §3.4 cross-reference to the parallelism spec | n/a (doc-only) |
|
||||
|
||||
Q1 and Q2 land **before** running the deployability sweep on the cluster. Q3 and Q4 can land anytime after Q2 (documentation, no functional impact).
|
||||
Q1 and Q2 land **before** running the deployability sweep on the cluster. Q3 and Q4 can land anytime after Q2 (documentation, no functional impact). The Tier 1 → Tier 2 transition is **atomic**: Q2 deletes the stopgap field and config-side wiring in the same commit that lands the proper kernel fix. No half-state where both paths exist.
|
||||
|
||||
---
|
||||
|
||||
@@ -346,9 +430,11 @@ Pre-existing tests that MUST continue to pass:
|
||||
|
||||
## 8. Rate target validation
|
||||
|
||||
Tier 2 adds a sigmoid per horizon per decision = 5 extra `expf()` calls per decision. At the existing 2100 ev/s rate and stride=4, this is ~500k decisions/quarter × 5 expf ≈ 2.5M expf operations per quarter — negligible (~0.01% of forward cost on L40S).
|
||||
Tier 2 replaces the `max(floor, recent_sharpe)` weight with a CBSW blend. Per-horizon per-decision cost added: ~6 arithmetic ops (subtraction, division, two `fmaxf/fminf`, one FMA, one comparison + branch) — no transcendentals, no memory reads beyond what was already done. At stride=4 with N_HORIZONS=5, that's ~30 ops × 500k decisions/quarter = 15M ops/quarter per backtest. Effectively free on L40S relative to the ~2 ms/decision forward cost.
|
||||
|
||||
No rate regression expected. Re-bench at Q2's smoke if curious; do not gate on it.
|
||||
**Validation is a measurement, not an estimate.** Q2 smoke must produce `ev/s` rate within ±5% of the Q1 smoke's rate at matching `n_parallel` and stride. If the post-CBSW rate drops by more than 5%, something is wrong (likely the compiler didn't unroll the loop, or there's a register-spill from the extra locals). Gate Q2's commit on the measured rate ratio, not on the back-of-envelope estimate.
|
||||
|
||||
Procedure: run a 1M-event Q1-mode smoke (Tier 1 active) on cluster; record ev/s. Then run the same 1M-event smoke at Q2's SHA (Tier 1 reverted, CBSW kernel live); record ev/s. Ratio must be ≥ 0.95.
|
||||
|
||||
---
|
||||
|
||||
@@ -357,10 +443,10 @@ No rate regression expected. Re-bench at Q2's smoke if curious; do not gate on i
|
||||
| Risk | Likelihood | Mitigation |
|
||||
|---|---|---|
|
||||
| Tier 1 stopgap fires too many trades, blowing through max_lots constraints | Medium — max-confidence at MATURE regime is over-aggressive | Tier 1 is brief (one validation run); Tier 2 lands immediately after and reverts the stopgap. Tier 1's max_events stays capped at 2M to keep blast radius small. |
|
||||
| Tier 2's sigmoid transition introduces non-monotonic sizing during the 5-15 trade window | Low | The transition is monotone in n_trades_seen by construction (sigmoid is monotone, both regimes' weights are monotonic in their drivers). The `cbsw_transition_smooth` test gates this. |
|
||||
| Tier 2's piecewise-linear transition has a non-differentiable kink at n=K±half_width | Low | Mathematically clean: the transition is monotone, saturating, and continuous (linear pieces meet at the endpoints). Kelly aggregation doesn't differentiate weights, so non-differentiability is irrelevant. `cbsw_transition_smooth` test confirms monotonicity by sweeping n_trades_seen. |
|
||||
| Tier 2's tests pass but real-data behavior still has 0 trades for some unrelated reason | Medium | If Q2 smoke shows `n_trades = 0`, do NOT proceed to deployability. Diagnose first — likely a downstream sim-side bug (cap_lots collapse, fill failure, OCO trap). |
|
||||
| The Tier 2 changes to `decision_policy_program` break the bytecode VM's other op-codes (OP_AGG_MEAN, OP_AGG_WEIGHTED_SHARPE) | Low | The CBSW logic only modifies how OP_AGG_WEIGHTED_SHARPE's final size is computed; other op-codes (EMIT_PER_HORIZON_SIZE, OP_AGG_MEAN, OP_AGG_MAX_CONFIDENCE) unchanged. Add a regression smoke for each op-code if confidence is low. |
|
||||
| CBSW's max-confidence component over-emphasizes one horizon mature too | Medium — at sq=0.881 we still blend in 11.9% max-conf | Acceptable; the 11.9% blend ensures the model can fire on extreme single-horizon signals even after maturity. If problematic, narrow the sigmoid scale (5.0 → 2.0) to make the transition sharper. |
|
||||
| Register pressure from the new locals (signed_sizes, weights, strong_h, strong_abs, sq_min_active, n_active_horizons) spills to local memory and tanks the kernel rate | Low — N_HORIZONS=5, total new locals are ~16 floats + 2 ints = 72 bytes, well within per-thread register budget on sm_89 (255 32-bit regs) | The §8 measurement gate (rate ratio ≥ 0.95) catches this. If it fires, refactor to reduce live ranges (compute strong_h via separate scan, drop ss_weighted accumulator and recompute post-loop). |
|
||||
| Cold-start fires trades on noisy single-horizon convictions, losing on most | Medium — fundamental trade-off | This IS the design choice (§2 design-choice note). The empirical question is whether the model has enough alpha to break even on noisy cold-start trades while `recent_sharpe` is bootstrapping. The Q2 smoke's `total_pnl_usd` answers this empirically. If negative + large, escalate to a higher-conviction floor (`kelly_frac_floor` 0.20 → 0.40) to reduce trade size during the cold-start window. |
|
||||
|
||||
---
|
||||
|
||||
@@ -379,7 +465,10 @@ No rate regression expected. Re-bench at Q2's smoke if curious; do not gate on i
|
||||
- Commits Q1–Q4 landed with all gating tests green.
|
||||
- Threshold-tuning smoke at threshold=0.0, cost=0.125 produces `n_trades > 100` and finite Sharpe/max-dd.
|
||||
- A second smoke at threshold=0.55 (≈ p60 from the threshold-tuning distribution), cost=0.125 produces a non-trivial trade count — proves the threshold gate + CBSW interact correctly.
|
||||
- **Rate gate**: post-Q2 cluster smoke ev/s rate ≥ 95% of pre-Q2 (Tier 1 active) ev/s rate at matching 1M-event setup. Confirms CBSW's per-decision overhead is negligible.
|
||||
- **All 7 cbsw_* unit tests pass on local RTX 3050 Ti** (per §4 test table); pre-existing CUDA tests still pass.
|
||||
- **`feedback_no_legacy_aliases` compliance**: Q2 deletes `use_cold_start_stopgap` and `--cold-start-stopgap` entirely; grep for both terms returns zero hits post-Q2.
|
||||
- Pearl `pearl_conviction_bootstrap_for_kelly_aggregation.md` committed under memory/ and linked from MEMORY.md.
|
||||
- Parallelism spec (`2026-05-19-deployability-sweep-parallelism-design.md`) cross-references this spec.
|
||||
- All pearls hold: permanent floor preserved, ISV-driven bootstrap, no version suffixes, no legacy aliases.
|
||||
- All pearls hold: permanent floor preserved, ISV-driven bootstrap, no version suffixes, no legacy aliases, no host branches in captured graph (untouched), no `expf` in the new aggregator hot path.
|
||||
- Out-of-scope items are explicitly listed (above), not silently deferred.
|
||||
|
||||
Reference in New Issue
Block a user