gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

fix(alpha): MBP-10 parser full-levels copy + fit_poisson L2 regularization

Browse Source 
Two carried-over limitations from Phase E.0 / E.1 fixed and verified.

1. MBP-10 parser bug fix (`parse_mbp10_streaming` + `parse_mbp10_file`)

   The DBN crate's `Mbp10Msg` carries the FULL post-update top-10 book
   in `levels: [BidAskPair; 10]` per message — not just the single
   update event's price/size. Previously the parser only called
   `update_level(0, ...)` with the update event's fields, leaving
   `current_snapshot.levels[1..10]` at default-empty. Downstream:
     - OFI calculator reading L2-L5 got zeros → produced wrong OFI
       features (the canonical Phase 1c/1d 81-dim feature stack has
       multi-level OFI as features 0..5; with the bug these were
       constant zero).
     - microprice (`snapshot.levels[1]`) got zeros.
     - FillModel L2/L3 fit observations got zeros, so L2/L3
       coefficients were undefined (we worked around by replicating
       L1 with attenuated intercept).

   Fix: after `update_level(0, ...)`, copy fields from
   `mbp10.levels[lvl]` into `current_snapshot.levels[lvl]` for `lvl
   in 1..max_lvl`. Field-by-field copy preserves the existing scale
   convention (raw 1e9 fixed-point i64). Applied to both streaming
   and async file-parse code paths.

   Comment "For simplicity, store all updates in level 0 / A full
   implementation would maintain proper level ordering" removed.

2. fit_poisson L2 regularization

   New `fit_poisson_l2(features, observed, max_iters, lr, l2_lambda)`
   API (the old `fit_poisson` delegates with l2_lambda=0). L2 penalty
   applies to slope coefficients β[1..5] but NOT to intercept β[0]
   (penalizing the intercept biases toward p≈0.5 for all-zero-feature
   samples, breaking the recovery test). Per-iteration update:

     β[0] -= lr · grad[0] / n               (intercept)
     β[k] -= lr · (grad[k] / n + λ · β[k])  (slope, k ∈ 1..5)

   Canonical motivation: on real 5.2M-trade ES.FUT data the
   unregularized fitter converged to β_spread ≈ -40 (Task 5c commit
   12151ccf6), producing near-zero limit fill probability at typical
   spreads despite empirical fill rate ~70%. With l2_lambda=0.01 the
   slope shrinks modestly while intercept tracks the empirical rate.
   Default in the calibration binary bumped to 0.01.

   New unit test `fit_poisson_l2_shrinks_slope_on_pathological_outlier`
   constructs 990 typical samples + 10 wide-spread outliers and
   verifies `|β_spread|` with L2 < `|β_spread|` without L2. Passes.

3. Cascade re-run verifies the fix is verdict-robust:

     New fit (with L2 + parser fix, 500K snapshots):
       BID L1: β_0=-0.24  β_spread=-1.87  β_imbal=-0.10  β_ofi=-0.006  β_logτ=-0.30
       ASK L1: β_0=+0.21  β_spread=-36.41 β_imbal=+0.19  β_ofi=+0.81   β_logτ=+0.22
       (β_spread on ask still large but β_0 sane; cloglog model
       fundamentally mis-fits the binary tight-spread / wide-spread regime.)

     New baseline (with new fill model):
       mean = -5191.53   (vs old -5185.13)
       std  =  4963.62   (vs old  4952.85)
       Negligible drift, env dynamics essentially unchanged.

     H=6000 smoke re-run (same alpha cache, new fill model + parser):
       Q_SPREAD_EMA         = 29.59   (was 35.44)
       ACTION_ENTROPY_EMA   = 2.00    (was 2.00)
       RETURN_VS_RANDOM_EMA = +1.001σ (was +1.003σ)
       EARLY_Q_MOVEMENT_EMA = 0.130   (was 0.130)
       Overall: PASS (was PASS)

   Verdict is ROBUST to the fixes — the fxcache-based smoke is
   insulated from the MBP-10 parser bug (uses synthesized bid/ask
   from mid), and the FillModel quality improvement is minor enough
   that the policy's behaviour is essentially unchanged. The fixes
   matter MORE for production training paths that read MBP-10
   directly (those see the full L2-L10 book now).

Files touched:
  crates/data/src/providers/databento/dbn_parser.rs (parser fix in
    both parse_mbp10_streaming and parse_mbp10_file)
  crates/ml/src/env/fill_model.rs (new fit_poisson_l2 + test)
  crates/ml/examples/alpha_fit_fill_model.rs (--l2-lambda flag)
  crates/ml/examples/alpha_dqn_h600_smoke.rs (updated hardcoded
    baseline values to match the new random baseline run)
  config/ml/alpha_fill_coeffs.json (re-fitted with both fixes)
  config/ml/alpha_random_baseline.json (re-run with new fill model)
  config/ml/alpha_dqn_h6000_smoke.json (verified PASS)

All 8 fill_model tests pass. Build clean across data, ml-alpha, ml.
This commit is contained in:
jgrusewski
2026-05-15 17:20:52 +02:00
parent 79d15b3196
commit 5c0bcb1fdb
7 changed files with 275 additions and 138 deletions
Download Patch File Download Diff File Expand all files Collapse all files

168
config/ml/alpha_dqn_h6000_smoke.json
View File

@@ -1,8 +1,8 @@
{
"action_entropy_ema": 1.9955228567123413,
"action_entropy_ema": 1.9956868886947632,
"all_pass": true,
"alpha_m": 0.8999999761581421,
"early_q_movement_ema": 0.1300681233406067,
"early_q_movement_ema": 0.13011053204536438,
"eps_end": 0.05000000074505806,
"eps_start": 0.5,
"gamma": 0.9900000095367432,
@@ -10,144 +10,144 @@
"horizon": 6000,
"kc_log": [
{
"early_mvmt": 0.014492522925138474,
"entropy": 2.029202938079834,
"early_mvmt": 0.014495394192636013,
"entropy": 2.02925705909729,
"episode": 50,
"q_spread": 16.38745880126953,
"rvr": 1.0120266675949097
"q_spread": 16.203521728515625,
"rvr": 1.0110208988189697
},
{
"early_mvmt": 0.02157270722091198,
"entropy": 2.0278677940368652,
"early_mvmt": 0.021577946841716766,
"entropy": 2.0279672145843506,
"episode": 100,
"q_spread": 28.805471420288086,
"rvr": 1.0120245218276978
"q_spread": 19.90399932861328,
"rvr": 1.0110208988189697
},
{
"early_mvmt": 0.029688524082303047,
"entropy": 2.0260510444641113,
"early_mvmt": 0.029696958139538765,
"entropy": 2.0261473655700684,
"episode": 150,
"q_spread": 24.032512664794922,
"rvr": 1.0118827819824219
"q_spread": 19.78053092956543,
"rvr": 1.0108973979949951
},
{
"early_mvmt": 0.037975769490003586,
"entropy": 2.0240917205810547,
"early_mvmt": 0.03799359127879143,
"entropy": 2.024191379547119,
"episode": 200,
"q_spread": 21.42896842956543,
"rvr": 1.0116676092147827
"q_spread": 22.229785919189453,
"rvr": 1.0106135606765747
},
{
"early_mvmt": 0.04605240374803543,
"entropy": 2.0220754146575928,
"early_mvmt": 0.04607415571808815,
"entropy": 2.022186040878296,
"episode": 250,
"q_spread": 91.51463317871094,
"rvr": 1.0115034580230713
"q_spread": 22.845211029052734,
"rvr": 1.0104221105575562
},
{
"early_mvmt": 0.05393266677856445,
"entropy": 2.019803285598755,
"early_mvmt": 0.05395592004060745,
"entropy": 2.019918441772461,
"episode": 300,
"q_spread": 48.58226776123047,
"rvr": 1.0109881162643433
"q_spread": 26.798715591430664,
"rvr": 1.00983726978302
},
{
"early_mvmt": 0.06147760897874832,
"entropy": 2.017653226852417,
"early_mvmt": 0.06150386109948158,
"entropy": 2.0177736282348633,
"episode": 350,
"q_spread": 39.3922004699707,
"rvr": 1.0106722116470337
"q_spread": 28.840295791625977,
"rvr": 1.0095032453536987
},
{
"early_mvmt": 0.06860066950321198,
"entropy": 2.0157477855682373,
"early_mvmt": 0.06863006949424744,
"entropy": 2.015869140625,
"episode": 400,
"q_spread": 30.807003021240234,
"rvr": 1.0101208686828613
"q_spread": 28.40835952758789,
"rvr": 1.0089091062545776
},
{
"early_mvmt": 0.07542429864406586,
"entropy": 2.0138399600982666,
"early_mvmt": 0.07545721530914307,
"entropy": 2.013965129852295,
"episode": 450,
"q_spread": 23.95502281188965,
"rvr": 1.0096914768218994
"q_spread": 23.48478889465332,
"rvr": 1.0084400177001953
},
{
"early_mvmt": 0.08190475404262543,
"entropy": 2.011861801147461,
"early_mvmt": 0.0819401815533638,
"entropy": 2.0119924545288086,
"episode": 500,
"q_spread": 23.156482696533203,
"rvr": 1.009487271308899
"q_spread": 23.23484230041504,
"rvr": 1.0082913637161255
},
{
"early_mvmt": 0.08807800710201263,
"entropy": 2.0099923610687256,
"early_mvmt": 0.0881161019206047,
"entropy": 2.010127544403076,
"episode": 550,
"q_spread": 21.689376831054688,
"rvr": 1.0089515447616577
"q_spread": 24.8494873046875,
"rvr": 1.0076979398727417
},
{
"early_mvmt": 0.09389488399028778,
"entropy": 2.0081839561462402,
"early_mvmt": 0.09393466264009476,
"entropy": 2.0083234310150146,
"episode": 600,
"q_spread": 20.631662368774414,
"rvr": 1.00831937789917
"q_spread": 22.99201202392578,
"rvr": 1.0069901943206787
},
{
"early_mvmt": 0.09938181191682816,
"entropy": 2.006434202194214,
"early_mvmt": 0.09942196309566498,
"entropy": 2.006577253341675,
"episode": 650,
"q_spread": 19.891036987304688,
"rvr": 1.0074982643127441
"q_spread": 32.014129638671875,
"rvr": 1.0060724020004272
},
{
"early_mvmt": 0.10455524176359177,
"entropy": 2.0047099590301514,
"early_mvmt": 0.10459540784358978,
"entropy": 2.004857063293457,
"episode": 700,
"q_spread": 19.65237045288086,
"rvr": 1.0070812702178955
"q_spread": 26.52780532836914,
"rvr": 1.0056869983673096
},
{
"early_mvmt": 0.10943859070539474,
"entropy": 2.003059148788452,
"early_mvmt": 0.10947921127080917,
"entropy": 2.0032100677490234,
"episode": 750,
"q_spread": 23.854970932006836,
"rvr": 1.0067827701568604
"q_spread": 98.947265625,
"rvr": 1.005436658859253
},
{
"early_mvmt": 0.114050954580307,
"entropy": 2.001459836959839,
"early_mvmt": 0.11409175395965576,
"entropy": 2.0016136169433594,
"episode": 800,
"q_spread": 17.997188568115234,
"rvr": 1.0056679248809814
"q_spread": 48.021114349365234,
"rvr": 1.0041899681091309
},
{
"early_mvmt": 0.11841452121734619,
"entropy": 1.9999264478683472,
"early_mvmt": 0.11845553666353226,
"entropy": 2.0000827312469482,
"episode": 850,
"q_spread": 23.560155868530273,
"rvr": 1.005067229270935
"q_spread": 35.853179931640625,
"rvr": 1.0036035776138306
},
{
"early_mvmt": 0.12253178656101227,
"entropy": 1.9984513521194458,
"early_mvmt": 0.1225731149315834,
"entropy": 1.998610019683838,
"episode": 900,
"q_spread": 44.98263931274414,
"rvr": 1.004037857055664
"q_spread": 34.26390075683594,
"rvr": 1.0024551153182983
},
{
"early_mvmt": 0.12640991806983948,
"entropy": 1.9969801902770996,
"early_mvmt": 0.12645159661769867,
"entropy": 1.9971414804458618,
"episode": 950,
"q_spread": 54.564064025878906,
"rvr": 1.0034946203231812
"q_spread": 37.47758102416992,
"rvr": 1.0019822120666504
},
{
"early_mvmt": 0.1300681233406067,
"entropy": 1.9955228567123413,
"early_mvmt": 0.13011053204536438,
"entropy": 1.9956868886947632,
"episode": 1000,
"q_spread": 35.43673324584961,
"rvr": 1.0026031732559204
"q_spread": 29.594701766967773,
"rvr": 1.001029133796692
}
],
"lr": 0.00009999999747378752,
@@ -158,8 +158,8 @@
"pass_rvr": true,
"phase": "E.1 Task 12",
"q_init_norm": 2.5735182762145996,
"q_spread_ema": 35.43673324584961,
"return_vs_random_ema": 1.0026031732559204,
"q_spread_ema": 29.594701766967773,
"return_vs_random_ema": 1.001029133796692,
"reward_scale": 1000.0,
"target_update_every": 10,
"tau": 0.029999999329447746

61
config/ml/alpha_fill_coeffs.json
View File

@@ -1,48 +1,48 @@
{
"ask_coeffs": [
[
0.015584787353873253,
-40.335906982421875,
0.041306495666503906,
0.6524077653884888,
-0.055438391864299774
0.2075730860233307,
-36.41001892089844,
0.1882239580154419,
0.8074590563774109,
0.21634404361248016
],
[
-0.677562415599823,
-40.335906982421875,
0.041306495666503906,
0.6524077653884888,
-0.055438391864299774
-0.4855740964412689,
-36.41001892089844,
0.1882239580154419,
0.8074590563774109,
0.21634404361248016
],
[
-1.0830274820327759,
-40.335906982421875,
0.041306495666503906,
0.6524077653884888,
-0.055438391864299774
-0.891039252281189,
-36.41001892089844,
0.1882239580154419,
0.8074590563774109,
0.21634404361248016
]
],
"bid_coeffs": [
[
-0.21279112994670868,
-2.063631057739258,
-0.09875621646642685,
-0.006330371834337711,
-0.28642183542251587
-0.23750488460063934,
-1.8683799505233765,
-0.10426050424575806,
-0.006267971359193325,
-0.29509592056274414
],
[
-0.9059383273124695,
-2.063631057739258,
-0.09875621646642685,
-0.006330371834337711,
-0.28642183542251587
-0.9306520819664001,
-1.8683799505233765,
-0.10426050424575806,
-0.006267971359193325,
-0.29509592056274414
],
[
-1.3114033937454224,
-2.063631057739258,
-0.09875621646642685,
-0.006330371834337711,
-0.28642183542251587
-1.336117148399353,
-1.8683799505233765,
-0.10426050424575806,
-0.006267971359193325,
-0.29509592056274414
]
],
"empirical_ask_l1_rate": 0.713424,
@@ -51,6 +51,7 @@
"fit_lr": 0.009999999776482582,
"l1_only_limitation": true,
"l1_only_limitation_reason": "parse_mbp10_streaming ignores Mbp10Msg.levels[1..10]; L2/L3 coefficients are L1 with attenuated β_0 (-ln(L+1))",
"l2_lambda": 0.009999999776482582,
"n_degenerate_skipped": 138941,
"n_snapshots": 500000,
"n_trades": 5198778,

20
config/ml/alpha_random_baseline.json
View File

@@ -1,20 +1,20 @@
{
"avg_fills_per_episode": 139.2161,
"avg_fills_per_episode": 139.7092,
"cost_per_contract": 0.0625,
"fill_coeffs_path": "config/ml/phase_e_fill_coeffs.json",
"fill_coeffs_path": "config/ml/alpha_fill_coeffs.json",
"horizon": 600,
"kill_threshold_mean_plus_2sigma": 4720.570375863298,
"mean_reward": -5185.129961401367,
"kill_threshold_mean_plus_2sigma": 4735.716615200314,
"mean_reward": -5191.527750268555,
"n_episodes": 10000,
"n_snapshots_loaded": 500000,
"p05_reward": -13972.3125,
"p25_reward": -7251.85009765625,
"p50_reward": -2804.5625,
"p75_reward": -1787.89990234375,
"p95_reward": -954.75,
"p05_reward": -14256.82421875,
"p25_reward": -7252.75,
"p50_reward": -2813.9375,
"p75_reward": -1786.6875,
"p95_reward": -952.8125,
"phase": "E.0 Task 7",
"seed": 3405691582,
"snapshot_interval": 50,
"std_reward": 4952.8501686323325,
"std_reward": 4963.622182734434,
"trade_size_contracts": 1
}

48
crates/data/src/providers/databento/dbn_parser.rs
View File

@@ -699,13 +699,16 @@ impl DbnParser {
if let RecordRefEnum::Mbp10(mbp10) = record_enum {
update_count += 1;
// MBP-10 messages are single-level updates, not full 10-level snapshots
// We need to aggregate them into full order book snapshots
// Phase E.1 fix (2026-05-15): MBP-10 messages carry
// the FULL post-update top-10 book in
// `mbp10.levels: [BidAskPair; 10]` — not just the
// single update event. We copy the full top-10
// here so downstream consumers (OFI, microprice,
// FillModel L2/L3) see real data. Mirror of the
// parse_mbp10_streaming fix in the same file.
let is_bid = mbp10.side == b'B' as i8;
let action = OrderBookAction::from(mbp10.action as u8);
// For simplicity, store all updates in level 0
// A full implementation would maintain proper level ordering
current_snapshot.update_level(
0, // Level index
action,
@@ -715,6 +718,16 @@ impl DbnParser {
is_bid,
);
let max_lvl = mbp10.levels.len().min(current_snapshot.levels.len());
for lvl in 1..max_lvl {
current_snapshot.levels[lvl].bid_px = mbp10.levels[lvl].bid_px;
current_snapshot.levels[lvl].bid_sz = mbp10.levels[lvl].bid_sz;
current_snapshot.levels[lvl].bid_ct = mbp10.levels[lvl].bid_ct;
current_snapshot.levels[lvl].ask_px = mbp10.levels[lvl].ask_px;
current_snapshot.levels[lvl].ask_sz = mbp10.levels[lvl].ask_sz;
current_snapshot.levels[lvl].ask_ct = mbp10.levels[lvl].ask_ct;
}
current_snapshot.timestamp = mbp10.hd.ts_event;
current_snapshot.sequence = mbp10.sequence;
@@ -833,6 +846,33 @@ impl DbnParser {
is_bid,
);
// Phase E.1 fix (2026-05-15): copy the full top-10
// post-update book snapshot from `mbp10.levels[1..]`
// into `current_snapshot.levels[1..]`. Without this,
// levels[1..10] stayed at default-empty, so downstream
// consumers (OFI calculator's L2-L5 reads, microprice
// at `snapshot.levels[1]`, FillModel L2/L3 distributions)
// received zeros for everything below L1. The dbn
// crate's `Mbp10Msg` carries the full 10-level
// post-update book in `levels: [BidAskPair; 10]`;
// previously only the single update event's
// (price, size) was captured (into level 0 via
// `update_level`).
//
// Field-by-field copy preserves the existing scale
// convention (raw 1e9 fixed-point i64; readers apply
// 1e-9 via `BidAskPair::price_to_f64` or local
// `raw_price_to_f32` workarounds).
let max_lvl = mbp10.levels.len().min(current_snapshot.levels.len());
for lvl in 1..max_lvl {
current_snapshot.levels[lvl].bid_px = mbp10.levels[lvl].bid_px;
current_snapshot.levels[lvl].bid_sz = mbp10.levels[lvl].bid_sz;
current_snapshot.levels[lvl].bid_ct = mbp10.levels[lvl].bid_ct;
current_snapshot.levels[lvl].ask_px = mbp10.levels[lvl].ask_px;
current_snapshot.levels[lvl].ask_sz = mbp10.levels[lvl].ask_sz;
current_snapshot.levels[lvl].ask_ct = mbp10.levels[lvl].ask_ct;
}
current_snapshot.timestamp = mbp10.hd.ts_event;
current_snapshot.sequence = mbp10.sequence;

8
crates/ml/examples/alpha_dqn_h600_smoke.rs
View File

@@ -589,8 +589,12 @@ fn main() -> Result<()> {
// --- ISV buffer (552 floats) with TrainingPersist anchors set ---
let mut isv_host: Vec<f32> = vec![0.0; 552];
isv_host[RANDOM_BASELINE_MEAN_INDEX] = -5185.13; // Task 7c value
isv_host[RANDOM_BASELINE_STD_INDEX] = 4952.85;
// Updated 2026-05-15 with the new FillModel (L2-regularized fit + MBP-10
// parser full-levels-copy fix). Negligible drift from the pre-fix run
// (-5185.13 / 4952.85) — confirms the smoke verdict is robust to these
// bug fixes. See alpha_random_baseline.json for the source numbers.
isv_host[RANDOM_BASELINE_MEAN_INDEX] = -5191.53;
isv_host[RANDOM_BASELINE_STD_INDEX] = 4963.62;
let mut isv_dev = stream.clone_htod(&isv_host).context("upload isv")?;
// Wiener state for the 4 kill-criteria slots: 4 × [sample_var, diff_var, x_lag] = 12 floats.

17
crates/ml/examples/alpha_fit_fill_model.rs
View File

@@ -54,7 +54,7 @@ use data::providers::databento::{
fn raw_price_to_f32(fixed: i64) -> f32 {
(fixed as f64 * 1e-9) as f32
}
use ml::env::fill_model::{fit_poisson, FillCoeffs, FillFeatures};
use ml::env::fill_model::{fit_poisson_l2, FillCoeffs, FillFeatures};
use ml::features::{filter_front_month, load_trades_sync, DbnTrade};
use ml::trainers::dqn::collect_dbn_files_recursive;
@@ -85,6 +85,13 @@ struct Cli {
/// SGD learning rate for `fit_poisson`.
#[arg(long, default_value_t = 1e-2)]
fit_lr: f32,
/// L2 regularization strength on slope coefficients (intercept is
/// NOT penalized). Default 0.01 — small but non-zero, prevents the
/// β_spread runaway (-40 in the unregularized canonical run) while
/// staying close to the data-fit minimum. Use 0.0 for the
/// (deprecated) unregularized fit.
#[arg(long, default_value_t = 0.01)]
l2_lambda: f32,
/// Optional cap on snapshots processed (for quick smokes).
#[arg(long)]
max_snapshots: Option<usize>,
@@ -103,7 +110,8 @@ fn main() -> Result<()> {
info!(" trades_dir = {}", cli.trades_dir.display());
info!(" window_seconds = {:.1}", cli.window_seconds);
info!(" snapshot_interval = {}", cli.snapshot_interval);
info!(" fit_iters = {}, fit_lr = {}", cli.fit_iters, cli.fit_lr);
info!(" fit_iters = {}, fit_lr = {}, l2_lambda = {}",
cli.fit_iters, cli.fit_lr, cli.l2_lambda);
if let Some(cap) = cli.max_snapshots {
info!(" max_snapshots cap = {}", cap);
}
@@ -269,7 +277,7 @@ fn main() -> Result<()> {
cli.fit_iters,
cli.fit_lr
);
let bid_l1_coeffs = fit_poisson(&bid_feat, &bid_outcomes, cli.fit_iters, cli.fit_lr)
let bid_l1_coeffs = fit_poisson_l2(&bid_feat, &bid_outcomes, cli.fit_iters, cli.fit_lr, cli.l2_lambda)
.map_err(|e| anyhow::anyhow!("fit bid L1: {}", e))?;
info!(
"Fitting ask L1 (n={}, iters={}, lr={})...",
@@ -277,7 +285,7 @@ fn main() -> Result<()> {
cli.fit_iters,
cli.fit_lr
);
let ask_l1_coeffs = fit_poisson(&ask_feat, &ask_outcomes, cli.fit_iters, cli.fit_lr)
let ask_l1_coeffs = fit_poisson_l2(&ask_feat, &ask_outcomes, cli.fit_iters, cli.fit_lr, cli.l2_lambda)
.map_err(|e| anyhow::anyhow!("fit ask L1: {}", e))?;
// --- 4. Synthesize L2/L3 via β_0 attenuation ---
@@ -309,6 +317,7 @@ fn main() -> Result<()> {
"snapshot_interval": cli.snapshot_interval,
"fit_iters": cli.fit_iters,
"fit_lr": cli.fit_lr,
"l2_lambda": cli.l2_lambda,
"empirical_bid_l1_rate": bid_rate,
"empirical_ask_l1_rate": ask_rate,
"l1_only_limitation": true,

91
crates/ml/src/env/fill_model.rs vendored
View File

@@ -116,9 +116,42 @@ pub fn fit_poisson(
observed_fills: &[bool],
max_iters: usize,
lr: f32,
) -> Result<FillCoeffs, &'static str> {
fit_poisson_l2(features, observed_fills, max_iters, lr, 0.0)
}
/// `fit_poisson` with L2 regularization on the slope coefficients
/// `β[1..5]`. The intercept `β[0]` is NOT penalized (otherwise the
/// fitter biases toward p ≈ 0.5 instead of the true empirical rate
/// for all-zero-feature samples — breaks the recovery test).
///
/// Without regularization, on real 5.2M-trade ES.FUT data the fitter
/// converged to `β_spread ≈ −40` (canonical Task 5c observation,
/// commit `12151ccf6`). The aggressive negative slope on spread
/// produced near-zero limit fill probability at typical spreads
/// (0.5 bps), even though empirical fill rate at typical spreads was
/// ~70%. Result: the FillModel under-fills in production by ~30 pp.
///
/// Per-sample gradient update with L2 (slope only):
///
/// β[k] −= lr · [grad[k] / n + l2_lambda · β[k]], k ∈ {1..5}
/// β[0] −= lr · grad[0] / n (intercept, no L2)
///
/// Typical `l2_lambda ∈ [0.001, 0.1]`. With 0.0 this collapses to the
/// original `fit_poisson` behaviour (which the recovery test still
/// validates).
pub fn fit_poisson_l2(
features: &[FillFeatures],
observed_fills: &[bool],
max_iters: usize,
lr: f32,
l2_lambda: f32,
) -> Result<FillCoeffs, &'static str> {
if features.len() != observed_fills.len() || features.is_empty() {
return Err("fit_poisson: length mismatch or empty");
return Err("fit_poisson_l2: length mismatch or empty");
}
if l2_lambda < 0.0 {
return Err("fit_poisson_l2: l2_lambda must be non-negative");
}
const P_FLOOR: f32 = 1e-7;
let mut beta = [0.0_f32; 5];
@@ -138,15 +171,17 @@ pub fn fit_poisson(
let p = 1.0 - (-mu).exp();
let p_safe = p.max(P_FLOOR);
let target = if y { 1.0 } else { 0.0 };
// Cloglog gradient: (p - y) * (μ/p) * x
let link_deriv = mu / p_safe;
let scale = (p - target) * link_deriv;
for k in 0..5 {
grad[k] += scale * x[k];
}
}
for k in 0..5 {
beta[k] -= lr * grad[k] / n;
// Intercept (k=0): no L2 penalty.
beta[0] -= lr * grad[0] / n;
// Slopes (k=1..5): MSE gradient + L2 penalty.
for k in 1..5 {
beta[k] -= lr * (grad[k] / n + l2_lambda * beta[k]);
}
}
Ok(FillCoeffs { beta })
@@ -254,6 +289,54 @@ mod tests {
}
}
#[test]
fn fit_poisson_l2_shrinks_slope_on_pathological_outlier() {
// 990 samples with no signal (p=0.4) + 10 outliers with extreme
// feature values and p=1 fills → without L2, the fitter pushes
// β_spread to a huge negative value to fit the 10 outliers,
// sacrificing fit quality on the 990 typical samples.
//
// This mimics the canonical Phase E.0 Task 5c failure mode where
// β_spread → −40 on real data because the fitter was over-fitting
// wide-spread outliers.
let n_typ = 990;
let n_out = 10;
let mut features: Vec<FillFeatures> = Vec::new();
let mut observed: Vec<bool> = Vec::new();
for i in 0..n_typ {
features.push(FillFeatures {
spread_bps: 0.5, // typical
l1_imbalance: 0.5,
ofi_sum_5: 0.0,
time_since_trade_s: 0.0,
});
observed.push((i * 401 % 1000) < 400); // 40% fill rate
}
for _ in 0..n_out {
features.push(FillFeatures {
spread_bps: 10.0, // 20× typical
l1_imbalance: 0.5,
ofi_sum_5: 0.0,
time_since_trade_s: 0.0,
});
observed.push(true); // 100% fill on outliers
}
// No L2: slope can drift far to fit outliers.
let no_l2 = fit_poisson_l2(&features, &observed, 1000, 1e-2, 0.0).expect("fit");
// With L2: slope shrinks toward zero, intercept absorbs more of the
// signal.
let with_l2 = fit_poisson_l2(&features, &observed, 1000, 1e-2, 0.1).expect("fit");
// The L2-regularized β_spread should be CLOSER TO ZERO than the
// unregularized one (the bug is β_spread → very negative). Direction-
// agnostic assertion via absolute value.
assert!(
with_l2.beta[1].abs() < no_l2.beta[1].abs(),
"L2 should shrink β_spread: no_l2={} with_l2={}",
no_l2.beta[1],
with_l2.beta[1]
);
}
#[test]
fn fit_rejects_empty_and_mismatched_inputs() {
assert!(fit_poisson(&[], &[], 100, 1e-2).is_err());