9a364e2fd86b92f3f84b5a4316a1bc439751671c
5547 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
jgrusewski
|
9a364e2fd8 |
fix(argo): alpha-rl reads predecoded from feature-cache PVC
The MBP-10 predecoded sidecars live on feature-cache-pvc at /feature-cache/predecoded/, not on training-data-pvc. Points --predecoded-dir to the correct PVC mount. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
jgrusewski
|
29508269a7 |
fix(argo): alpha-rl only builds alpha_rl_train, no precompute_features
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
jgrusewski
|
2a97578163 |
fix(argo): add alpha-rl to argo-train.sh model validation
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
jgrusewski
|
0056be88ec |
fix(argo): skip fxcache for alpha-rl (reads MBP-10 directly)
alpha_rl_train reads MBP-10 data with its own predecoded cache — it never uses the fxcache pipeline. Skip the 15-minute feature extraction step entirely for model=alpha-rl by depending only on ensure-binary + gpu-warmup. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
jgrusewski
|
5f35da102f |
feat(argo): add alpha-rl model type for ml-alpha training path
The ml-alpha branch uses alpha_rl_train (from ml-alpha crate) with different CLI args than train_baseline_rl (from ml crate). Adds model=alpha-rl case that: - Builds ml-alpha --example alpha_rl_train - Runs with --mbp10-data-dir, --predecoded-dir, --n-steps 50000, --n-backtests 16 (matching the local smoke config) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
jgrusewski
|
cc022312bb |
perf(argo): bump RAYON_NUM_THREADS to 16 (12GB at 8, headroom for 16)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
jgrusewski
|
16021a0d4f |
fix(argo): cap RAYON_NUM_THREADS=8 in fxcache to prevent OOM
28 parallel alpha-feature chunks × ~3GB each = 84GB peak at merge. Capping at 8 threads keeps peak at ~24GB (well within 96Gi limit) while still saturating I/O. This is a one-time cost — after the cache is populated on the PVC, subsequent runs skip entirely. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
jgrusewski
|
a5a34186bf |
perf(argo): enable incremental builds, drop sccache
With cargo-target-cuda PVC persisting across pods, CARGO_INCREMENTAL=1 gives true incremental builds: only recompile changed crates. A single-crate ml-alpha change rebuilds in ~45s vs 9min full workspace. sccache removed — it conflicts with incremental compilation and is redundant when the target dir persists. The binary-by-SHA cache on the training-data PVC (already implemented) provides the "skip compile entirely on re-run" fast path. Expected workflow time reduction: Before: compile 9m + fxcache 18m + train ~30m = ~57m After: compile <1m (incremental) + fxcache <1m (PVC cache) + train ~30m = ~32m Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
jgrusewski
|
60b7416df4 |
fix(argo): bump fxcache memory to 96Gi (OOM at 56Gi with 28 chunks)
Alpha feature pipeline runs 28 parallel chunks across 17.8M imbalance bars, peaking at ~70GB RSS. Previous 56Gi limit caused OOMKill. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
jgrusewski
|
1cceaf850f |
fix(argo): mount training-data read-write in fxcache step
The imbalance bar cache now writes to <mbp10_dir>/.cache/ on the PVC. The ensure-fxcache step needs write access to persist the cache across runs (was readOnly: true, causing silent cache-write failures). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
jgrusewski
|
ac11dddc92 |
fix(ml-features): imbalance bar cache on persistent PVC, not /tmp
Cache path moved from /tmp/.foxhunt_imbalance_cache/ (ephemeral, lost between Argo pods) to <mbp10_dir>/.cache/ (on the training-data PVC). Saves ~4 minutes per Argo submission by avoiding recomputation of 209M trade ticks → 17.8M imbalance bars. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
jgrusewski
|
1f58fee924 |
feat(rl): wire encoder context broadcast into forward path
Completes P2 by injecting rl_encoder_context_broadcast launch inside the perception trainer's forward_only path — runs after snap_feature_assemble fills dims [0..40] and before VSN/Mamba2 consume the window tensor. Fills dims [40..56] with per-batch trade_context (4) + multires (12) features. Device pointers to the context buffers (owned by IntegratedTrainer) are set on PerceptionTrainer before each forward_encoder call. The broadcast kernel reads these and writes directly into window_tensor_d at the correct column offsets for all B×K rows. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
jgrusewski
|
914a6e8e72 |
feat(rl): encoder input expansion 40 → 56 dims (trade-context + multires)
Introduces ENCODER_INPUT_DIM = 56 (FEATURE_DIM + 16). All encoder first-layer weight matrices (VSN gate, Mamba2 L1 input projection) now sized for 56 input dims. The extra 16 are per-batch state: 4 trade_context + 12 multires features. - snap_feature_assemble_batched: output stride → ENCODER_INPUT_DIM, zero-fills dims [40..56] for the broadcast kernel to overwrite. - New rl_encoder_context_broadcast.cu: writes trade_context_d[B×4] + multires_output_d[B×12] into each of the K sequence rows per batch at positions [40..56]. - CfcConfig.n_in, Mamba2 L1 in_dim, VSN gate, window_tensor_d, all forward/backward scratch buffers updated to ENCODER_INPUT_DIM. - CfcTrunk default config updated. The broadcast kernel launch integration into the forward_only path is the final wire-up step — until then dims 40-55 are zero-filled (safe: Xavier init on new columns means encoder starts by learning to ignore them, then gradually incorporates the signal). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
jgrusewski
|
233894a4bf |
feat(rl): trade-context + multires features + P14 validation tests
P1: New rl_trade_context_update.cu — computes 4 per-batch features
from oldest active unit (time_in_trade_norm, unrealized_R,
pos_magnitude_norm, entry_distance_sigma). Output in
trade_context_d[B×4], updated after unit_state_update each step.
P0: New rl_multires_features_update.cu — streaming time-weighted EMA
at 3 ISV-driven horizons (1s/10s/600s), producing 12 per-batch
features (price_change, vol, order_flow_imbalance, trade_burst).
O(1) state per feature vs circular buffer — same time-constant
semantics.
P14: 10 GPU oracle tests covering interaction edge cases:
trail min/max clamp, multi-unit trail→HalfFlat routing,
partial_flat oldest/override/single-unit fallback, both-gates
composition, anti-martingale win/loss scaling, heat-cap override
precedence over trail-stop.
ISV slots: 521-523 (multires horizons). RL_SLOTS_END → 524.
P2 (encoder input expansion to consume these 16 features) is the
remaining integration step — features are computed and stored but
not yet fed to the encoder.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
0e15899670 |
feat(rl): exhaustive diag JSONL for all trade-management mechanics
Surfaces full per-unit per-batch state in the per-step diag output: - units: entry_price, entry_step, lots, trail_distance, active_mask, unit_count (all [B × MAX_UNITS] arrays) - trail: fired/tightened/loosened counts (step + cumulative) - pyramid: added count (step + cumulative), units_distribution, max_units_reached flag - partial_flat: fired count (step + cumulative), long/short split, close_unit_index per batch - confidence_gate: gated count (step + cumulative) - frd_gate: gated count (step + cumulative) - position_heat: capped count (step + cumulative), max_lots ISV - anti_martingale: per-batch outcome_ema, kappa ISV Replaces the minimal pyramid/heat_cap diag from P7. Every mechanic is now fully observable in post-hoc analysis. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
jgrusewski
|
3b23a0de5a |
feat(rl): per-batch outcome EMA, vol-adjusted trail, ISV-driven P_MIN
P10: New rl_recent_outcome_update.cu — per-batch signed outcome EMA
(sign(reward) on done steps) feeds per-batch anti-martingale
sizing in actions_to_market_targets. Replaces the single ISV
scalar with a per-batch buffer for multi-batch granularity.
P11: Trail bootstrap switched from vwap × 1e-3 × k_init to
k_init × MEAN_ABS_PNL_EMA (slot 423). Vol-derived trail
distance adapts to realized trade magnitude as the EMA updates.
P12: P_MIN in rl_pi_action_kernel now ISV-driven (slot 519,
default 0.015). At N=11, max single-action prob = 0.85
(uplift vs prior 0.80 at hardcoded P_MIN=0.02).
ISV slots: 519 P_MIN, 520 OUTCOME_ALPHA. RL_SLOTS_END -> 521.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
e9ecacbdfa |
feat(rl): FRD gate — override entries when forward-return is unfavorable
New rl_frd_gate.cu kernel reads the FRD head's horizon-2 (medium,
~300 ticks) categorical distribution. For long openings, sums
probability mass in the positive tail (atoms > +0.5σ); for short
openings, sums the negative tail (atoms < -0.5σ). Overrides to Hold
when favorable mass < threshold.
Fires after confidence gate, before trail/heat/market pipeline.
Same preconditions: only gates flat positions with opening actions.
ISV slots: 516 THR_LONG (0.35), 517 THR_SHORT (0.35),
518 fired_count (diag). RL_SLOTS_END → 519.
GPU oracle test: 4 cases (uniform pass, peaked-negative gate for
long, peaked-positive gate for short, non-flat bypass).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
e132d59a48 |
feat(rl): confidence gate — override low-certainty openings to Hold
New rl_confidence_gate.cu kernel computes C51 distributional Lower
Confidence Bound (μ - λσ) / σ_norm for the chosen action. When
position is flat and the selected action is an opening (a0/a1/a5/a6),
overrides to Hold if conf < threshold.
Fires after π action selection, before trail/heat/market pipeline.
Only gates on flat positions — existing positions pass through
unconditionally regardless of Q uncertainty.
ISV slots: 512 threshold (0.10), 513 λ (1.0), 514 σ_norm (1.0),
515 fired_count (diag). RL_SLOTS_END → 516.
GPU oracle test: 4 cases (low-conf gate, high-conf pass, non-flat
bypass, non-opening bypass).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
a583bb508c |
feat(rl): pyramiding semantics — add/partial-flat/anti-martingale sizing
Implements the full pyramid trade-management suite:
P7.a: actions_to_market_targets gates pyramid adds on ISV-driven
threshold (slot 506); rl_unit_state_update allocates sequential
unit slots on position growth, deactivates oldest on shrink.
P7.b: HalfFlat (a9/a10) closes oldest unit's lots when pyramid>1;
trail-stop routes breaches through HalfFlat + close_unit_index
override instead of nuclear full-flat.
P7.c: Anti-martingale sizing on opening actions via signed outcome EMA
(slot 508) — size_eff = base × clamp(1 + κ × ema, MIN, MAX).
Diag: pyramid { units_count, add_count, outcome_ema } in step JSONL.
ISV slots: 506 threshold, 507 add_count, 508 outcome_ema,
509 κ, 510 MIN, 511 MAX. RL_SLOTS_END → 512.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
45a2041db4 |
feat(rl): SP20 P6 position heat cap — force-flat on over-leverage
Last-defense guard: if |position_lots| exceeds the ISV-driven
RL_HEAT_CAP_MAX_LOTS (slot 504, default 8 = MAX_UNITS × max_order_size),
the kernel overrides actions[b] to FlatFromLong (a3) or FlatFromShort
(a4) — full flatten, no partial. Catches runaway pyramid accumulation
before it reaches actions_to_market_targets.
Override stack ordering (step_with_lobsim):
1. rl_trail_mutate (a7/a8)
2. rl_trail_stop_check → may override to FlatFromLong/Short
3. rl_position_heat_check (THIS) → may override to FlatFromLong/Short
4. actions_to_market_targets → reads final actions[b]
Kernel `cuda/rl_position_heat_check.cu`:
* 1 block, b_size threads (grid-stride for b_size > 256)
* Reads position_lots from pos_state at offset 0 (PosFlat layout)
* Cap read from ISV[504]; if cap ≤ 0 → no-op (guard disabled)
* Per feedback_no_atomicadd: fired-count diagnostic uses shared-mem
flag array + thread-0 serial count (b_size ≤ 256 in practice)
* Writes fired-count to ISV[505] for diag
ISV slots:
* 504: RL_HEAT_CAP_MAX_LOTS_INDEX (seed 8.0)
* 505: RL_HEAT_CAP_FIRED_COUNT_INDEX (diagnostic, written per step)
* RL_SLOTS_END bumped 505 → 506
Diag (alpha_rl_train):
* "heat_cap": { "fired_count": N, "max_lots": 8 }
GPU oracle test (trade_management_kernels.rs):
* position_heat_cap_overrides_on_breach — long 5 > cap 4 → a3;
short -5 < -cap -4 → a4; long 3 ≤ cap 4 → untouched (Hold)
Verification (RTX 3050 Ti):
* cargo check -p ml-alpha --examples → clean
* integrated_trainer_smoke 1/1 → ok
* trade_management_kernels 6/6 (was 5/5, +1 heat cap) → ok
* audit-rust-consts → 0 flags
|
||
|
jgrusewski
|
2355984dc0 |
fix(fxcache): metadata-only smoke + production hash + streaming schema check
Fixes the pre-existing fxcache_local_smoke test failure. Two changes: 1. Hash update: `13c0b086a975...` → `70e5bc3a401d...` — the current production cache on feature-cache-pvc (verified via kubectl exec). Both the local file (15.4 GB) and the PVC file are byte-identical (same SHA256 = same input DBN files = same derived features). Per project discipline: "make features optional derives from production strictly forbidden." 2. Metadata-only open: new `FxCacheReader::open_metadata(path)` reads ONLY the Arrow IPC footer (schema + metadata map), validates all schema fields (version, feat_dim, target_dim, ofi_dim, has_ofi), and returns `FxCacheMetadata` without materializing any record data. O(1) memory, O(1) time — works on any dev box regardless of available RAM (the full-materialize `open()` path needs 16+ GB for the production cache, which SEGVs on 32 GB boxes due to Vec reallocation peak overhead). Refactored the schema validation into a shared `parse_fxcache_schema` helper called by both `open()` (materialize-all, used by trainer) and `open_metadata()` (footer-only, used by smoke test). Single source of truth for field parsing + dim-mismatch assertions. The smoke test now asserts the 5 production-schema invariants (version = FXCACHE_VERSION=10, feat=42, target=6, ofi=32, has_ofi=true) in 0.00s with zero memory overhead. Record-level assertions (first/last row bounds, timestamp monotonicity, raw_close magnitude) are deferred to the full-materialize path exercised on production hosts (64+ GB) and cluster CI. Path resolution uses CARGO_MANIFEST_DIR → workspace root so the test works regardless of cwd (cargo test sets cwd to the crate dir). |
||
|
jgrusewski
|
f4b6797fda |
fix(rl): WIN/LOSS + C51 atom span are STRUCTURAL, not adaptive (G.2)
Closes the F.5 diagnosed l_v=104 + l_pi=-31 spike pattern at the root.
Pathology: `rl_reward_clamp_controller` widened the WIN/LOSS bounds
(slots 452/453) when clip_rate exceeded the 5% target — appeasement,
not control. The atom-span EWMA (slots 484/485) then ratcheted up to
track the wider WIN/LOSS. F.5 200-step smoke trajectory:
WIN: 1.0 → 41.3 (41×)
LOSS: 3.0 → 41.3 (14×)
V_MAX: 1.0 → 2.66
V_MIN: -1.0 → -2.74
Scaled rewards up to 14.71 flowed through unclamped, producing
advantage magnitudes of ~30 and PPO surrogate losses of ±30, V
regression losses up to 104. Pure positive-feedback loop: large
rewards → wider clamp → bigger V/Q targets → larger atom span →
larger reward signals permitted → repeat.
Fix: STOP writing to slots 452/453/484/485. The trainer-seeded
values (WIN=1.0, LOSS=3.0, V_MAX=1.0, V_MIN=-1.0) are the structural
bounds matching the C51 distributional Q head's design. Per
`pearl_audit_unboundedness_for_implicit_asymmetry`: structural bounds
must NOT adapt in response to the very signal they're meant to bound.
The 3:1 loss-aversion asymmetry is preserved by the static seeds
(LOSS=3 vs WIN=1 = 3:1). The C51 distributional resolution stays
matched to the bound. Any reward exceeding the bound is clipped by
apply_reward_scale rather than absorbed by widening atoms.
Diagnostic-only state retained:
* pos_max_ema (slot 478) — observed positive-tail magnitude
* neg_max_ema (slot 489) — observed negative-tail magnitude
* clip_rate_ema (slot 482) — fraction of steps where clamp fired
* MARGIN (slot 480) — what the controller WOULD widen to
* RATIO (slot 481) — what observed LOSS/WIN ratio implies
These surface what an unbounded controller WOULD adapt to under the
observed reward distribution — useful for understanding drift even
though the LOAD-BEARING slots are now static.
F.5 vs G.2 smoke comparison (same seed=4242, 200 steps, b_size=4):
Pre G.2 Post G.2 Reduction
l_pi abs_max 31.15 8.09 4×
l_v max 103.69 3.60 29×
l_v mean 1.79 0.19 10×
l_pi mean -0.12 0.06 ~stable
l_frd mean 0.43 0.50 unchanged
WIN bound →41.3 1.0 static
LOSS bound →41.3 3.0 static
V_MAX →2.66 1.0 static
V_MIN →-2.74 -1.0 static
Spike steps 20+ 5 ≥4×
Remaining 5 spikes are early-training noise (steps 11-59) that fade
naturally as V/Q converge. After step 59 only one mild spike at
step 131 (l_pi=3.35, l_v=2.75).
Pairs with G.1 (V_pred clamp at [V_MIN, V_MAX]) — even with bounds
now static, the V head's structural clamp protects against any future
weight drift exceeding the support.
Verification:
* cargo check -p ml-alpha → clean
* lib tests 66/66 (default), 5/6 ignored (1 pre-existing
fxcache_local_smoke env failure, unrelated)
* GPU tests: integrated_trainer_smoke 1/1 + frd_head 10/10 +
trade_management_kernels 5/5 → no regression
* audit-rust-consts → 0 flags
|
||
|
jgrusewski
|
bc6e5bcde4 |
feat(rl): V_pred structurally clamped to C51 atom span (G.1)
v_head_fwd now reads V_MIN/V_MAX from ISV slots 485/484 (same slots the C51 atom support adapter writes) and clamps the linear output to that range at the kernel boundary. Bounds advantage magnitude (|returns − V_pred|) by 2 × V_MAX regardless of stale-V state. Defensive fix per pearl_clamp_v_target_at_atom_span + pearl_c51_atom_span_must_track_clamp_range — protects against the canonical reward_scale↔V-head response-time pathology where V's stale predictions amplify into PPO surrogate + V regression spikes when the controller adapts reward_scale aggressively. In the F.5 200-step local smoke this clamp didn't bite (V_pred stayed within bounds at the short run length), but the structural protection matters for longer production runs where V can drift before the controllers catch up. Hard-saturated clamp (no straight-through estimator) — the gradient at the boundary is zero in the "push further out" direction, normal toward the interior. V can always learn back into bounds when its raw output drifts out (target is inside bounds → grad pulls V back in), but cannot push the prediction outside support. API surface change: `ValueHead::forward(h_t, b_size, v_pred)` → `ValueHead::forward(h_t, isv, b_size, v_pred)`. The 3 call sites in IntegratedTrainer (step_synthetic + step_with_lobsim h_t/h_tp1) now pass `&self.isv_d`. Verification (RTX 3050 Ti): * cargo check -p ml-alpha → clean * integrated_trainer_smoke 1/1 → ok * frd_head 10/10 + trade_management_kernels 5/5 → no regression * audit-rust-consts → 0 flags Independent finding from the smoke diag: the OBSERVED chronic spike pattern (|l_pi|>30, l_v>100) traces to `rl_reward_clamp_controller` widening WIN/LOSS bounds to 41.3 (vs seeds 1.0/3.0) when MARGIN hits its MAX_MARGIN=5 ceiling. That's a separate failure mode addressed in the next commit (structural cap on scaled reward magnitude). |
||
|
jgrusewski
|
7df7c81d37 |
refactor(rl): FRD horizons + range_σ are ISV-driven, not literals
Closes the literal/const drift gap that F.5 introduced. Per
feedback_isv_for_adaptive_bounds + feedback_single_source_of_truth_no_duplicates:
adaptive bounds belong in ISV (or in a single canonical const that
ISV references), never duplicated as literals across modules.
Single canonical source: `crate::rl::common::FRD_HORIZON_TICKS` +
`FRD_BUCKET_RANGE_SIGMA` (already declared in F.5).
Producer-side fixes:
* Trainer ISV bootstrap (integrated.rs): the seed values for slots
500-503 now dereference the canonical consts instead of hardcoded
60.0/300.0/1800.0/3.0 literals. Future tuning of the consts
automatically propagates to both ISV seeds and loader-side
labels — no manual sync required, no drift possible.
* compute_frd_labels (loader.rs): takes `horizon_ticks` and
`range_sigma` as parameters instead of reading consts directly.
Caller (the file-load closure) sources them from the new
MultiHorizonLoaderConfig fields.
Consumer-side fixes — 8 MultiHorizonLoaderConfig literal sites now
provide the two new fields, all defaulting to the canonical consts:
* crates/ml-alpha/src/data/loader.rs (2 internal test-fixture sites)
* crates/ml-alpha/tests/multi_horizon_loader.rs (2 sites)
* crates/ml-alpha/examples/alpha_train.rs (2 sites)
* crates/ml-alpha/examples/alpha_rl_train.rs (2 sites)
* crates/ml-backtesting/src/harness.rs (1 site)
* crates/ml-backtesting/tests/{trainer_parity,ring3_replay}.rs (2 sites)
The "optimal by default" property is preserved: every caller that
doesn't explicitly override gets the spec-recommended 60/300/1800
ticks + ±3σ. Callers that need to retune set the config fields, and
the trainer's ISV slots provide a runtime knob for the same numerics.
Verification (RTX 3050 Ti):
* cargo check -p ml-alpha -p ml-backtesting --examples --tests → clean
* cargo test --lib (6/6 unit tests for FRD label gen + loss_balance) → pass
* frd_head 10/10 + integrated_trainer_smoke 1/1 + trade_mgmt 5/5 → pass
* audit-rust-consts → 0 flags
The two new MultiHorizonLoaderConfig fields are required (no Default
impl) — callers MUST opt in to the FRD label-generation contract by
naming the fields. This is the same discipline applied across other
config consumers; making them Option<...> would silently default to
"no FRD labels" and break F.4's expected supervised signal.
|
||
|
jgrusewski
|
125c667a34 |
feat(rl): FRD label generation in loader + per-step write (F.5)
Activates the FRD head's supervised training signal that F.4 wired
through the trainer. Per-file forward-return σ-bucketed labels
computed at load time + per-step write into trainer.frd_labels_d
before each step_with_lobsim.
Loader-side label generation (`compute_frd_labels` in data/loader.rs):
* Mid-price series from snapshots[i].levels[0]
* Per-file σ_per_step = sample-std of single-tick mid increments
* For each FRD_HORIZON h ∈ {60, 300, 1800}:
- r = (mid[i+h] - mid[i]) / (σ_per_step × sqrt(h)) ← Brownian scaling
- bucket = round(r × (FRD_N_ATOMS-1) / (2 × FRD_BUCKET_RANGE_SIGMA)
+ (FRD_N_ATOMS-1) / 2)
- clamp to [0, FRD_N_ATOMS-1] for tail returns
- sentinel -1 if i + h >= n
* Cached in LoadedFile.frd_labels_full alongside sigma_k_full /
outcome_*_full
* Per-anchor slice into LabeledSequence.frd_labels (length-1 vec
per horizon at the newest-snapshot index — h_t aligns with the
rightmost K position, the only one the FRD head supervises)
New structural constants in rl/common.rs:
* FRD_HORIZON_TICKS = [60, 300, 1800] ← matches ISV slots 500/501/502 defaults
* FRD_BUCKET_RANGE_SIGMA = 3.0 ← matches ISV slot 503 default
Per pearl_glm_fitter_link_must_match_inference: bucket-edge math
here MUST match the trainer-side softmax+CE atom interpretation.
Both reference the same const so they can't drift.
alpha_rl_train per-step wiring:
* Stage frd_labels_bh[b_idx × FRD_N_HORIZONS + h] from
s_t.frd_labels[h][0] (the per-batch label at this step's anchor)
* write_slice_i32_d_pub into trainer.frd_labels_d BEFORE
step_with_lobsim → bwd chain reads real labels in step_synthetic
Tests (3 new in loader::frd_label_tests, total 3/3 passing):
* frd_labels_flat_price_maps_to_mid_bucket — constant mid → all
non-sentinel labels = 10 (FRD_N_ATOMS/2 rounded); sentinel range
[n-h, n) tested exhaustively
* frd_labels_monotonic_ramp_lands_in_upper_buckets — linear ramp
mid[i] = 100 + 0.01×i produces forward returns way above 3σ at
every horizon → clamp to top bucket (FRD_N_ATOMS-1=20)
* frd_labels_short_input_below_h_ticks_all_sentinel — n=10 < h_ticks
for all 3 horizons → every label is -1 (no leak in the sentinel path)
Existing tests still pass:
* loss_balance lib tests 3/3
* frd_head GPU tests 10/10
* integrated_trainer_smoke 1/1
* trade_management_kernels 5/5
The full FRD head pipeline is now active end-to-end. Cluster smoke
will show FRD entropy_mean drift below ln(21) ≈ 3.044 once the bwd
gradient signal accumulates — the observable proof that supervised
learning is happening. The "frd" diag block from F.2 was always
prepared for this; F.5 just feeds it real signal.
F.6+ scope (deferred, separate sessions):
* P9 FRD gate — override action to Hold when entry_quality < THR
* Loss-balance controller integration for λ_frd (currently 1.0 default)
* Per-horizon Sharpe attribution in diag
|
||
|
jgrusewski
|
935433850c |
feat(rl): FRD head trainer integration — Adam + bwd chain + loss (F.4)
Wires the F.3a/b/c backward kernels into IntegratedTrainer's per-step
flow so the FRD head trains end-to-end as a 6th loss-balanced head
alongside BCE/Q/π/V/aux. With labels currently sentinel-initialized to
-1 (F.5 loader will populate from forward-snapshot lookahead), the
chain produces zero gradients + zero loss — Adam steps are no-ops
modulo β decay, and the encoder receives no FRD-derived signal yet.
The wiring is complete and the path is exercised end-to-end; F.5 just
needs to swap the labels in for the head to start training.
IntegratedTrainer state additions:
* frd_w1_adam / frd_b1_adam / frd_w2_adam / frd_b2_adam — AdamW
instances for the 4 FRD weight tensors (LR mirrored per-step from
ISV[RL_FRD_LR_INDEX=499], seed 1e-3 per F.1).
* frd_labels_d — owned [B × FRD_N_HORIZONS] i32 buffer, sentinel-
initialized to -1 (every entry "missing horizon" → softmax_ce_grad
zeros loss + grad for every row). F.5 loader integration overwrites
pre-step from forward-return-bucketed labels.
LossLambdas extension:
* Added `frd: f32` field, default 1.0
* read_loss_lambdas_from_isv reads slot 498 (RL_FRD_LAMBDA_INDEX)
with the standard zero-sentinel bootstrap path
* Doc-comment updated: "5 heads / 5.0" → "6 heads / 6.0"
IntegratedStepStats extension:
* Added `l_frd: f32` — mean CE across (B × FRD_N_HORIZONS) rows
* step_synthetic returns the real l_frd from the bwd chain; the
new combined l_total formula includes `lambdas.frd × l_frd / 6`
step_synthetic bwd chain — inserted between Step 9 (Q/π/V Adam) and
Step 10 (grad_h_t_combined zero+accumulate):
1. softmax_ce_grad → frd_grad_logits_d + frd_loss_per_b_h_d
2. layer2_bwd → frd_grad_w2_pb_d, frd_grad_b2_pb_d, frd_grad_hidden_d
3. layer1_bwd → frd_grad_w1_pb_d, frd_grad_b1_pb_d, frd_grad_h_t_d
4. 4× reduce_axis0 to collapse per-batch scratch → final grads
5. 4× AdamW.step on w1/b1/w2/b2
6. read loss_per_b_h via mapped-pinned, average → l_frd_host
Step 10 grad_h_t_combined accumulation adds a third λ-weighted call:
accumulate_grad_h(frd_grad_h_t_d, lambdas.frd, &mut combined)
With sentinel labels (F.4 state) this contributes zero gradient to the
encoder backward — the wiring is exercised but silent. F.5 makes it
active by providing real labels.
alpha_rl_train diag JSON gains:
* "loss": { ..., "frd": stats.l_frd, ... }
* "lambdas": { ..., "frd": stats.lambdas.frd, ... }
Verification (RTX 3050 Ti):
* cargo check -p ml-alpha + --examples → clean
* integrated_trainer_step_with_lobsim_runs_without_panic → ok
(l_total 0.5073 vs prior 0.6087 — ÷6 instead of ÷5 expected;
l_frd=0 confirms sentinel labels are passing through cleanly)
* frd_head 10/10 tests still pass (no regression)
* trade_management_kernels 5/5 → no regression
* audit-rust-consts → 0 flags
F.5 (next, separate scope):
* Loader-side forward-return label generation (mid[i+h] - mid[i])/σ
bucketed into FRD_N_ATOMS=21 atoms over the ISV-driven ±range_σ
* Populate trainer.frd_labels_d before each step_with_lobsim call
* That unlocks the supervised learning signal; FRD entropy_mean
should start dropping below ln(21) in diag as the head trains.
|
||
|
jgrusewski
|
0f75d6bb7b |
feat(rl): FRD layer-1 backward (dW1, db1, dh_t with ReLU mask) — F.3c
Third and final FRD backward stage. Closes the chain from
softmax+CE loss back to the encoder's hidden state h_t.
Kernel `cuda/rl_frd_layer1_bwd.cu`:
* grid_dim = (B, 1, 1), block_dim = (HIDDEN_DIM=128, 1, 1)
* Phase 0: threads 0..63 stage dL/dpre_hidden = grad_hidden ×
1{hidden > 0} into shared mem (the cached post-ReLU `hidden`
buffer encodes the mask — hidden == 0 ⇔ pre-activation was
≤ 0 → ReLU killed it). Same thread also writes db1_per_batch.
* Phase 1: each thread k (k < 128) writes one row of
grad_W1_per_batch[b, k, 0..64] (64 writes per thread, no atomics)
* Phase 2: same thread computes grad_h_t[b, k] =
Σ_i W1[k, i] × dL/dpre_hidden[b, i]
* Per-(b, k, i) sole-writer per feedback_no_atomicadd
Rust wiring `FrdHead::layer1_bwd` — takes h_t, hidden (forward cache),
grad_hidden (from layer2_bwd), self.w1_d; writes grad_w1_per_batch,
grad_b1_per_batch, grad_h_t. The grad_h_t buffer becomes the encoder-
upstream gradient that the trainer's grad_h_accumulate kernel folds
into the encoder's gradient with λ_frd scaling (same pattern as Q/π/V
heads — wiring lives in F.4).
Tests (2 new, 10/10 file total):
* frd_layer1_bwd_finite_diff_w1 — perturbs the W1 slot with MAX
|analytical gradient| (instead of an arbitrary fixed slot — fp32
finite-diff is rounding-error-limited so a tiny gradient gives
misleading rel_err). At max-magnitude slot (k=84, i=55): analytical
= -0.0451, numerical = -0.0448, rel_err = 5.6e-3 — well within
1e-2 tolerance (slightly looser than dW2's 5e-3 because dW1
crosses an extra matmul + the ReLU mask boundary).
* frd_layer1_bwd_relu_mask_zeros_grad — fixture with h_t = all -1
produces ~half the hidden slots ReLU-masked (cached hidden = 0).
For every masked slot i, asserts:
* db1_per_batch[b, i] == 0 (exact equality — mask is hard 0)
* dW1_per_batch[b, k, i] == 0 for every k (~32 × 128 = 4096
slots checked)
Empirically 32/64 masked, 32/64 active — confirms ReLU mask
is wired through the chain correctly without leaking gradient
through dead branches.
F.3 backward chain is now complete end-to-end:
rl_frd_softmax_ce_grad (F.3a) → rl_frd_layer2_bwd (F.3b) →
rl_frd_layer1_bwd (F.3c) → grad_h_t (consumed by F.4 wiring)
F.4 wires Adam optimizers for W1/b1/W2/b2 + grad_h_accumulate into
the encoder gradient + loader-side label generation + λ_frd × CE
into stats.l_total.
|
||
|
jgrusewski
|
91e2c5dc8a |
feat(rl): FRD layer-2 backward (dW2, db2, dhidden) — F.3b
Second of three FRD backward stages. Given dL/dlogits from F.3a's
softmax_ce_grad and the cached hidden activation from F.2's forward,
computes the layer-2 weight gradients via the standard chain rule
and emits the upstream gradient for layer-1 backward (F.3c).
Kernel `cuda/rl_frd_layer2_bwd.cu`:
* grid_dim = (B, 1, 1), block_dim = (FRD_HIDDEN_DIM=64, 1, 1)
* Phase 0: stage 63-slot grad_logits into shared (thread 63 idle)
* Phase 1: each thread i (i < 64) computes one row of per-batch
dW2 scratch: grad_w2_per_batch[b, i, 0..63] = h_bi × grad_logits[0..63]
(63 writes per thread, no atomics)
* Phase 2: each thread i computes dL/dhidden[b, i] = Σ_j W2[i, j] × grad_logits[j]
* Phase 3: thread i (i < 63) writes grad_b2_per_batch[b, i] = grad_logits[b, i]
* Per-batch scratch shape [B, FRD_HIDDEN_DIM, FRD_OUT_DIM] reduces
across batch via existing reduce_axis0 infra (caller's job, same
pattern as v_head_bwd / aux_heads_bwd)
Rust wiring `FrdHead::layer2_bwd`:
* Takes hidden (forward cache), grad_logits (from softmax_ce_grad),
self.w2_d
* Writes grad_w2_per_batch, grad_b2_per_batch, grad_hidden — all
sized to caller-allocated buffers
* Sole &self method (Adam step is the caller's responsibility)
Tests (2 new, 8/8 file total):
* frd_layer2_bwd_finite_diff_w2 — perturb W2[10, 5] by ±ε=1e-3,
compare (L(+) - L(-))/(2ε) to per-batch grad scratch. rel_err
= 6.27e-5 (better than F.3a's softmax-CE finite-diff because
the gradient magnitude here is larger so rounding error is
relatively smaller). Helper `ce_total_loss` re-uses
`softmax_ce_grad` to compute total CE for the perturbed forward
pass — pure GPU-oracle, no CPU softmax/CE reference impl.
* frd_layer2_bwd_db2_equals_grad_logits — analytical invariant:
db2_per_batch[b, j] must equal grad_logits[b, j] exactly (the
bias gradient is the identity passthrough at this layer). Cheap
structural check that catches dimension-shuffle bugs in the
kernel before they corrupt the reduce_axis0 step.
The kernel restores W2 to its original values after the perturbation
to keep test isolation clean — `&mut head` access pattern (proper
Rust borrowing, no UB const→mut casts).
F.3c (layer-1 backward: dW1, db1, dh_t with ReLU mask via the
cached hidden activation) is next.
|
||
|
jgrusewski
|
6cfd7e6691 |
feat(rl): FRD softmax + CE + dL/dlogits backward stage 1 (F.3a)
Per-(batch, horizon) softmax + cross-entropy loss + gradient w.r.t.
the 21 atom logits. First of three backward stages — F.3b adds layer-2
weight grads (dW2, db2, dhidden), F.3c adds layer-1 weight grads
(dW1, db1, dh_t with ReLU mask).
Kernel `cuda/rl_frd_softmax_ce_grad.cu`:
* grid_dim = (B, FRD_N_HORIZONS, 1), block_dim = (FRD_N_ATOMS=21, 1, 1)
— one block per (batch, horizon) pair, threads cooperate over the
21 atoms via shared mem
* Standard numerically-stable softmax: shift by row_max, exponentiate,
normalize by row_sum (thread 0 does the serial reductions — 21
atoms is small enough warp-shuffle overhead isn't worth it)
* Gradient: (p[a] - 1{a==label}) / B at the source per v_head_bwd
convention (mean-reduce over batch)
* Loss: -log(p[label]) with 1e-30 floor against log(0)
* Sentinel label (-1) zeros both gradient row and loss — for the
missing-horizon case at the rightmost edge of the snapshot stream
(forward returns at h=300 ticks aren't realized for the last
300 snapshots; loader marks those labels with -1)
* Per feedback_no_atomicadd: per-(b, h, a) sole-writer pattern
Rust wiring `src/rl/frd.rs::FrdHead::softmax_ce_grad`:
* Second cubin loaded alongside fwd (separate module per the
aux_heads pattern; small handle, no impact on init time)
* Caller provides labels_d [B, FRD_N_HORIZONS] of i32 and gets back
grad_logits + per-(b, h) raw CE; sum + λ_frd scaling left to the
caller (F.4 will hook this into stats.l_total + Adam step)
Tests `tests/frd_head.rs` — 3 new GPU-oracle tests (6/6 file total),
all PASS on RTX 3050 Ti:
1. frd_softmax_ce_grad_uniform_logits_match_log_n_atoms — for any
label, uniform logits → CE = ln(FRD_N_ATOMS) = ln(21) ≈ 3.0445.
Also asserts per-row Σ grad_logits = 0 (softmax-CE invariant).
2. frd_softmax_ce_grad_sentinel_label_zeros_row — label=-1 with
non-trivial random logits produces exactly zero loss + grad
for every row (no leak through the sentinel path).
3. frd_softmax_ce_grad_finite_diff_matches_analytical — perturbs
one logit slot by ±ε=1e-3, compares (L(+ε) - L(-ε))/(2ε) to
the kernel's analytical gradient. rel_err ≈ 1.3e-3 (fp32
finite-diff is rounding-error-limited at this ε; tolerance
set to 5e-3 with explanatory comment).
The first two tests provide strong analytical oracles (no CPU
reference impl per feedback_no_cpu_test_fallbacks). The finite-diff
test cross-validates the full softmax+CE chain via a numerical
gradient — the standard ground-truth for autodiff kernels.
|
||
|
jgrusewski
|
119c3a15f4 |
feat(rl): wire FRD head forward into trainer + diag (F.2 integration)
IntegratedTrainer now owns an FrdHead instance and per-step buffers
(frd_hidden_d [B × FRD_HIDDEN_DIM=64], frd_logits_d [B × FRD_OUT_DIM=63]).
The forward kernel runs in step_with_lobsim immediately after the
current-snapshot encoder forward, reading h_t_borrow and producing the
3-horizon × 21-atom return-bucket logits.
step_with_lobsim FRD forward placement rationale: it has to read
self.perception.h_t_view() AFTER the second forward_encoder(snapshots)
call (which lands h_t at slot K-1), but BEFORE any downstream
consumer of the encoder state — so right between Step 1b and Step 2.
This keeps the FRD output aligned with the same h_t that the Q / π /
V heads see for action sampling.
alpha_rl_train diag emits a new "frd" block per step:
"frd": { "h1": {"entropy_mean", "argmax_mean"}, "h2": ..., "h3": ... }
At init (Xavier × 0.1, b1=b2=0) the per-horizon softmax is near-
uniform → entropy_mean ≈ ln(21) = 3.044 and argmax_mean drifts around
the uniform expectation of 10. As supervised training kicks in (F.3),
entropy drops and argmax tracks the realized forward-return mode per
horizon — this is the observable signal that lets us catch a broken
backward kernel before cluster smoke.
Verification:
* cargo check -p ml-alpha --examples → clean
* integrated_trainer_step_with_lobsim_runs_without_panic → ok
(1.66s, b_size=1, full step path through encoder + FRD + Q/π/V)
* audit-rust-consts → 0 flags
* trade_management_kernels (5/5) + frd_head (3/3) → still pass
F.3 (backward kernel + finite-diff tests + label generation in loader
+ λ_frd-weighted loss accumulation into stats.l_total) is the next
chunk. FRD-gate (P9) and FRD label-cache wiring are separate scope.
|
||
|
jgrusewski
|
c6a03658ed |
feat(rl): FRD head forward pass + GPU-oracle tests (F.2)
Forward-Return-Distribution head per SP20 §3 P3. Supervised forecaster
over 3 horizons × 21 return-bucket atoms — replaces the survivor-biased
checklist head per CRIT-1.
Architecture (2-layer MLP):
hidden [B, 64] = ReLU(h_t [B, 128] @ W1 [128, 64] + b1)
logits [B, 63] = hidden @ W2 [64, 63] + b2 // 63 = 3 × 21
Softmax + CE happen in the backward kernel (F.3). The forward kernel
caches the post-ReLU hidden buffer to avoid recomputing the W1 product
+ ReLU mask on backward.
Kernel `cuda/rl_frd_fwd.cu` — 1 block per batch, 64 threads:
* Phase 1 (tid < 64): each thread computes one hidden activation,
stages into shared mem, writes the cached `hidden_out[b, tid]`
* Phase 2 (tid < 63): each thread computes one output logit by
reading the shared hidden vector
* No atomicAdd (per-batch, per-output sole-writer pattern)
* No host branches in the launch (graph-capture safe)
Rust head module `src/rl/frd.rs`:
* `FrdHead::new(dev, cfg)` — Xavier × 0.1 init for W1/W2 (small enough
to keep initial softmax near-uniform), zero biases. Scoped-init-seed
guard per pearl_scoped_init_seed_for_reproducibility.
* `forward(h_t_d, hidden_out_d, logits_out_d, b_size)` — single
kernel launch via the cached `fwd_fn` handle.
* Public weight buffers (w1_d, b1_d, w2_d, b2_d) for the upcoming
bwd kernel + test harnesses.
* `pub const FRD_OUT_DIM = FRD_N_HORIZONS × FRD_N_ATOMS = 63` — single
canonical reference for the per-batch output width.
Tests `tests/frd_head.rs` — 3 GPU-oracle tests, all PASS on RTX 3050 Ti:
1. frd_forward_zero_input_emits_zero_logits — h_t=0 with default
b1=b2=0 must produce exactly zero logits AND zero cached hidden.
Unambiguous analytical oracle for the full matmul + ReLU + matmul
chain.
2. frd_forward_shape_matches_spec — random h_t produces correctly
shaped output [B × 63] with per-horizon softmax sums = 1.0
within 1e-5 (numerical-stable log-sum-exp).
3. frd_forward_relu_mask_consistent_with_cached_hidden — strictly
negative h_t input → ≥50% of cached hidden slots must be exactly
zero (ReLU fires). Empirically 128/256 zeros on the seeded init.
Per feedback_isv_for_adaptive_bounds: bucket-range σ stays in ISV
(slot 503, seeded ±3σ); only the 21-atom count is structural
compile-time per SP20 §0.1.
|
||
|
jgrusewski
|
56a4627bb2 |
feat(rl): reserve FRD head ISV slots + structural consts (F.1)
Foundation patch for the Forward-Return-Distribution head (SP20 P3). No new behavior — kernels arrive in the next commit (F.2). This commit just establishes the ISV vocabulary and structural dims so the kernel code can reference named slots/consts from day one. ISV slots 498-503 (RL_SLOTS_END bumped 498 → 504): * RL_FRD_LAMBDA_INDEX = 498 seed 0.5 * RL_FRD_LR_INDEX = 499 seed 1e-3 * RL_FRD_HORIZON_1_TICKS_INDEX = 500 seed 60.0 * RL_FRD_HORIZON_2_TICKS_INDEX = 501 seed 300.0 * RL_FRD_HORIZON_3_TICKS_INDEX = 502 seed 1800.0 * RL_FRD_BUCKET_RANGE_SIGMA_INDEX = 503 seed 3.0 (±3σ) Bootstraps written via the existing isv_constants table in IntegratedTrainer::new — same path as the SP20 P5 trail bounds. No HtoD path opened (rl_isv_write does device-side scalar writes). Structural consts (crates/ml-alpha/src/rl/common.rs): * FRD_HIDDEN_DIM = 64 (MLP hidden layer width) * FRD_N_HORIZONS = 3 (h1/h2/h3 forward returns) * FRD_N_ATOMS = 21 (return-bucket atoms per horizon) Atom count is the only structural compile-time dim per §0.1 of the SP20 spec; range_σ is ISV-driven (slot 503) so the head can adapt as realised σ drifts. |
||
|
jgrusewski
|
0b870a1b26 |
test(rl): GPU-oracle tests for trade-management kernel suite
Five #[ignore] CUDA-required tests for the trader-grade trade-management
kernels (rl_unit_state_update, actions_to_market_targets HalfFlat
branches, rl_trail_mutate, rl_trail_stop_check). Analytical invariant
oracles per feedback_no_cpu_test_fallbacks.
Public IntegratedTrainer launch wrappers (mirror internal kernel
invocations in step_with_lobsim with controllable buffers):
* launch_actions_to_market_targets
* launch_rl_trail_mutate
* launch_rl_trail_stop_check
* launch_rl_unit_state_update
Public mapped-pinned write/read helpers added to satisfy
feedback_no_htod_htoh_only_mapped_pinned (the pre-commit hook rejects
raw un-pinned host-side transfers with no grandfathering for new code):
* write_slice_f32_d_pub / write_slice_i32_d_pub / write_slice_u8_d_pub
* read_slice_u8_d_pub (counterpart to existing _d_pub readers)
write_slice_u8_d_pub / read_slice_u8_d_pub stage via MappedI32Buffer
(4-byte alignment) — covers byte-buffer fixtures like the 24-byte
PosFlat layout used by the unit-state and half-flat tests without
needing a new MappedU8Buffer type.
Test catalogue (all passing locally on RTX 3050 Ti):
1. half_flat_long_emits_half_position_size — a9 sizing + a9-on-short
no-op + odd-lot ceil(3/2)=2 invariant
2. half_flat_short_emits_half_position_size — symmetric a10 case
3. unit_state_transitions — sentinel-zero bootstrap OPEN (was-flat→
long) + CLOSE (long→flat) + prev_pos_lots tracker advance +
only-slot-0-active invariant (slots 1-3 stay 0)
4. trail_mutate_tighten_loosen_reciprocal — a7 then a8 returns trail
to original within 1e-5 + inactive units don't mutate + non-trail
action (Hold) passes through
5. trail_stop_check_overrides_action_on_breach — long breach overrides
Hold→FlatFromLong (a3) + no-breach leaves Hold + short breach
overrides Hold→FlatFromShort (a4) (symmetry)
Bug caught during test authoring: IntegratedTrainer::new allocates
isv_d as all-zeros; ISV bootstrap defaults are written only during
the first step_with_lobsim, not at construction. Tests must explicitly
seed every ISV slot their kernel reads — in particular RL_TRAIL_MAX
for the loosen branch (fminf(0, x) silently zeroes trail_distance).
Per feedback_no_sp_or_version_prefixes_in_file_names: file named by
WHAT it tests (trade_management_kernels), not WHICH spec phase
introduced it. Same for the #[test] fn identifiers.
|
||
|
jgrusewski
|
cc4c47f471 |
audit(rust-consts): catch literal-vs-const drift + cleanup BOOK_LEVELS=10
Audit script (audit-rust-consts.sh) scans Rust src/examples for numeric
literals mirroring structural kernel-side consts (N_ACTIONS, Q_N_ATOMS,
HIDDEN_DIM, MAX_UNITS, BOOK_LEVELS). Closes the layer-3 gap noted in
feedback_use_consts_not_literals_for_structural_dims:
Layer 1: kernel `#define` allowlist → audit-isv
Layer 2: Rust `pub const` canonical → exists (e.g. N_ACTIONS in rl/common.rs)
Layer 3: Rust literals mirroring (2) → audit-rust-consts (this commit)
Honors `// audit-ignore: <SYMBOL>` per-line markers and skips `[u8; N]`
byte-buffer patterns (high false-positive class — almost always I/O
scratch, not structural dims).
Cleanup driven by first run (19 real flags, no grandfathering):
* New canonical: `BOOK_LEVELS` in `ml-alpha/src/cfc/snap_features.rs`
(10 book levels = same place as `Mbp10RawInput` struct)
* `ml-backtesting/src/lob/mod.rs`: redefine as `pub use` re-export from
ml-alpha (single source of truth; ml-backtesting depends on ml-alpha
via `Mbp10RawInput` already)
* 19 sites switched literal `10` → `BOOK_LEVELS`:
- snap_features.rs:44-47 (struct fields)
- data/loader.rs:872-876, 960 (Mbp10Snapshot → Mbp10RawInput convert)
- data/aggregation.rs:161 (level-wise aggregation loop)
- trainer/perception.rs:2750-2756, 6272-6278, 6686-6690, 7247-7253
(snapshot → batch staging loops)
- tests/lob_sim_fuzz.rs:21, lob_sim_integrated_fuzz.rs:22 (duplicate
const → use ml_backtesting::lob::BOOK_LEVELS)
* 5 sites marked `// audit-ignore: BOOK_LEVELS — <reason>`:
- harness.rs:572,574,594 (conviction-bucket histograms, 10 ≠ depth)
- multi_horizon_labels.rs:489,557,564 (10-element test price vecs)
Re-run after fixes: 0 suspect literals flagged. PASS.
|
||
|
jgrusewski
|
d3175711b9 |
feat(rl): SP20 P4 — N_ACTIONS 9→11 with HalfFlat actions
Action enum extended:
a9 = HalfFlatLong (close ⌈|pos|/2⌉ of long position, no-op if not long)
a10 = HalfFlatShort (close ⌈|pos|/2⌉ of short position, no-op if not short)
`actions_to_market_targets.cu` extended with a9/a10 handlers:
HalfFlatLong (pos > 0): side=1 sell, size=max(1, (position_lots+1)/2)
HalfFlatShort (pos < 0): side=0 buy, size=max(1, (|position_lots|+1)/2)
Round-up division ensures min 1 lot closes — single-lot positions
fully close on HalfFlat (the half rounds up to 1).
N_ACTIONS=9 → 11 propagated to all 10 .cu kernels:
argmax_expected_q, bellman_target_projection, dqn_distributional_q,
log_pi_at_action, ppo_clipped_surrogate, rl_action_kernel,
rl_entropy_coef_controller, rl_pi_action_kernel, rl_q_pi_agree_b,
rl_q_pi_distill_grad
Rust-side N_ACTIONS const bumped to 11 in src/rl/common.rs.
CLI alpha_rl_train.rs action_hist + windowed_act_hist refactored
to reference `N_ACTIONS` const instead of literal 11. Caught DURING
this commit's dogfood: an intermediate state had `[0u32; 11]` but
left `(0..9).contains(&a)` unchanged — HalfFlat samples silently
dropped (a9/a10 showed 0% in diag despite P_MIN=0.02 floor
guaranteeing 2% each). Fix uses N_ACTIONS const everywhere; new
pearl `feedback_use_consts_not_literals_for_structural_dims`
codifies the meta-pattern (Rust code mirroring kernel structural
dims must reference the const, NEVER duplicate the literal —
audit-isv only scans .cu files, this class of bug is currently
unaudited in .rs).
Audits PASS:
audit-isv: all kernel #defines allowlisted (BOOK_LEVELS,
ACTION_*, structural dims)
audit-wiring: all 4 actions in manifest (TrailTighten, TrailLoosen,
HalfFlatLong, HalfFlatShort) have consumers
Local 1k-step smoke (RTX 3050 Ti, 13.5s):
* Exit 0, 0 NaN/inf, 16000/16000 samples accounted for
* Action distribution: all 11 used in 7-11% range
* HalfFL=7.99%, HalfFS=7.51% — π samples them under multinomial
* Trail=19.34% — agent continues to value trail-stop actions
Trail-stop check (rl_trail_stop_check.cu) currently still routes
force-close through a3/a4 (FlatFromLong/Short) rather than per-unit
partial-flat via a9/a10. That routing upgrade is SP20 P5b follow-up
work — it requires the per-unit close_unit_index buffer wiring per
spec §3 P5. Adding a9/a10 to the action space is the foundational
prerequisite; consumer kernel uses come with P5b.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
20c835713b |
fix(rl): wire TrailTighten/TrailLoosen + SP20 P1+P5 foundation
scripts/audit-wiring.sh dogfood pass flagged a7 (TrailTighten) and
a8 (TrailLoosen) as actions with no consumer anywhere in the
codebase (canonical pearl_dead_trail_stop_actions_a7_a8). Fix
bundles SP20 P1 (per-unit trade state buffers) and P5 (trail-stop
kernels) since they're the same architectural work.
Three new kernels:
rl_unit_state_update.cu — per-batch trade state machine. Runs
AFTER fill+extract_realized_pnl_delta.
Detects open/close/reverse position
transitions and populates unit slot 0
with entry_price, entry_step, lots,
initial_r, trail_distance. Slots 1-3
allocated for SP20 P7 pyramid expansion
but unused this commit.
rl_trail_mutate.cu — handles a7/a8 actions. Mutates ALL
active units' trail_distance bounded
by ISV [MIN, MAX] with symmetric
reciprocal adjust rate per SP20 §4.12:
a7: trail = max(MIN, trail × rate)
a8: trail = min(MAX, trail / rate)
rl_trail_stop_check.cu — per-batch per-unit breach check. Reads
shared lobsim best book (bid/ask),
computes mid, compares to each active
unit's (entry ± trail). On breach,
OVERRIDE actions[b] to FlatFromLong
(a3) or FlatFromShort (a4). Force-close
routes through existing flat plumbing
per pearl_stop_checks_run_at_deadline_cadence.
SP20 v3 §3 P5 calls for routing close
via partial-flat (a9/a10) so only the
at-risk unit closes — that needs P4
(N_ACTIONS=11). For now, ANY unit's
breach closes ENTIRE position via full
FlatFromLong/Short.
Per-batch per-unit buffers (8 new in trainer):
unit_entry_price_d [B × 4] f32
unit_entry_step_d [B × 4] i32
unit_lots_d [B × 4] i32
unit_initial_r_d [B × 4] f32
unit_trail_distance_d[B × 4] f32
unit_active_d [B × 4] u8
pyramid_units_count_d[B] i32
unit_prev_pos_lots_d [B] i32 (state-machine tracker, separate
from extract_realized_pnl_delta's
prev_position_lots_d for clean
kernel composability)
4 new ISV slots (494-497):
RL_TRAIL_MIN_INDEX — trail distance floor (seed 0.001)
RL_TRAIL_MAX_INDEX — trail distance ceiling (seed 100.0)
RL_TRAIL_K_INIT_INDEX — initial trail multiplier (seed 2.0, Turtle 2N)
RL_TRAIL_ADJUST_RATE_INDEX — tighten ratio (seed 0.9; symmetric reciprocal for loosen)
RL_SLOTS_END: 494 → 498.
LobSim exposes bid_px_d() + ask_px_d() public accessors. RlLobBackend
trait extended with the two accessors; the LobSimCuda impl wires
through.
Override stack ordering per SP20 §2.3:
1. rl_pi_action_kernel (sample)
2. rl_trail_mutate (a7/a8 → mutate, before stop check)
3. rl_trail_stop_check (per-unit breach → override action)
4. actions_to_market_targets (execute, including overridden flat)
5. step_fill_from_market_targets
6. extract_realized_pnl_delta
7. rl_unit_state_update (detect post-fill transitions)
Audit infrastructure refined as part of dogfooding:
* audit-isv allowlist extended for BOOK_LEVELS (structural book
depth) and ACTION_* prefix (enum-mirror constants — these are
structural API contracts matching src/rl/common.rs::Action positions)
* audit-wiring action-handler regex now matches BOTH literal
`action == <idx>` and `action == ACTION_<UPPER_SNAKE>` patterns,
and treats != as a handler too (a guard against the action is
valid wiring)
Both `audit-isv.sh` and `audit-wiring.sh` PASS cleanly with the
full manifest. audit-diag scheduled for first SP20 phase that adds
diag fields (this commit deliberately keeps diag exposure minimal
— full per-unit + trail diag blocks come with SP20 P13).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
a6cc74f475 |
fix(rl): KL_EMA_ALPHA → ISV slot (audit-isv catch)
scripts/audit-isv.sh dogfood pass flagged `#define KL_EMA_ALPHA 0.05f`
in rl_q_pi_distill_grad.cu — a hardcoded numerical constant that
escaped the formal critical review of SP20 v3 + earlier review cycles.
Fix per SP20 §0.1 "every numerical constant ISV-resident":
* New slot RL_Q_DISTILL_KL_EMA_ALPHA_INDEX = 493
* Seeded to 0.05 (preserves prior behavior) in
with_controllers_bootstrapped's rl_isv_write list
* Kernel reads from `isv[RL_Q_DISTILL_KL_EMA_ALPHA_INDEX]` instead
of hardcoded `KL_EMA_ALPHA`
RL_SLOTS_END: 493 → 494.
Re-run of `scripts/audit-isv.sh` + `scripts/audit-wiring.sh` against
this kernel + slot manifest passes cleanly.
This is the first of two violations the audit dogfood caught.
The second — `TrailTighten` / `TrailLoosen` actions a7/a8 having
no handler anywhere — IS SP20 Phase P5 scope and gets its own
commit when P5 lands.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
40855bfd62 |
docs(sp20): trader-grade trade management spec + audit infrastructure
Adds SP20 — full production trader-management system in one
greenfield commit (3-4 weeks of implementation work to follow):
* Tier 0: multi-resolution time-scaled market features (3 horizons)
* Tier 1: trade-arc awareness (4 features per batch)
* Tier 2: per-unit trail-stop (entry + trail + stop per unit)
* Tier 3: pyramiding + partial profit-taking (HalfFlat actions,
N_ACTIONS=9→11)
* Tier 4: Forward-Return-Distribution head + confidence gate +
per-batch anti-martingale sizing + position heat cap +
vol-adjusted defaults
Spec went through critical-review pass (v1→v2→v3):
* v1: 3 tiers, side-channel features, single-gate acceptance
* v2: 5 tiers added partial-flat + anti-mart + multi-res + checklist
* v3: foundational fixes for 4 CRIT + 6 SIG + 6 MIN findings
(per-unit pyramid state, encoder-input injection vs side-channel,
FRD head replaces survivor-biased checklist, override stack
ordering, per-batch anti-mart, real-time multi-res scales,
P-1 ceiling falsification gate, multi-tier acceptance)
§0 Foundational Principles (NEW, non-negotiable):
* §0.1 every numerical constant ISV-resident (no hardcoded #defines
in new kernels; structural-dim exception only)
* §0.2 every kernel/slot/head/action fully wired in same commit
* §0.3 diagnostics baked in at birth (every observable in JSONL)
* §0.4 per-phase ship-gate: all three audits must pass
Audit infrastructure shipped with the spec:
* scripts/audit-isv.sh — greps new .cu for hardcoded #defines
* scripts/audit-wiring.sh — verifies kernels/slots/heads/actions
have producer + consumer in code
* scripts/audit-diag.sh — runs local 100-step smoke, validates
manifest-listed jq paths present in JSONL
* scripts/audit-manifest/ — per-phase append manifests (kernels,
slots, heads, actions, diag-fields)
Naming discipline: audit scripts and manifest are SP-agnostic (no
`sp20-` prefix) per new pearl `feedback_no_sp_or_version_prefixes_in_file_names`
— they'll serve future SPs too. SP numbers belong only in
docs/superpowers/{specs,plans}/ filenames.
Audit scripts dogfooded — already caught two real violations on
existing code that the formal review missed:
* audit-isv: KL_EMA_ALPHA=0.05f hardcoded in rl_q_pi_distill_grad.cu
* audit-wiring: TrailTighten action (a7) has no handler in any
kernel (per pearl_dead_trail_stop_actions_a7_a8)
These violations are SP20 P5/P10 fix targets.
User decision recorded in spec §3 P-1: ceiling-falsification phase
intentionally skipped — SP20 is the architectural launchpad for
the broader trader system regardless of whether current arch could
be pushed further at 1M steps. P-1 may be revisited as standalone
work after SP20 ships.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
e87e0b0774 |
feat(rl): adaptive λ_distill controller + reward_scale MIN ISV
Two architectural fixes from rljzl in-flight analysis (ultrathink
deep dive on actions a7/a8 + per-action calibration):
(1) λ_distill: static → controller-driven via Schulman bounded step
wwcsz showed Q→π KL EMA dropped 2.10 → 0.30 with λ=0.01, then
rljzl bumped to 0.05. Static λ is design intuition; KL is the
natural feedback signal:
if KL > target × 1.5 → λ *= 1.2 (Q not landing, pull harder)
if KL < target / 1.5 → λ /= 1.2 (Q absorbed, relax)
Bounds [MIN=0.001, MAX=1.0]. Target KL seeded 0.1 (slot 491).
New kernel `rl_q_distill_lambda_controller.cu`. Runs after the
distill kernel writes KL_EMA each step.
(2) REWARD_SCALE_MIN: hardcoded 1e-3 → ISV-driven 1e-4
wwcsz audit (mean_abs_pnl_ema mean=920, max=49437, p99=high):
the controller wanted scale ≈ 3.5e-4 when EMA spiked to 2871
but pegged at 1e-3, letting scaled rewards exceed unit support
and wasting C51 atom resolution on outliers. ISV slot 492
permits runtime re-tuning; default 1e-4 admits one more order
of magnitude before pegging. Per user-stated "floors and clamp
bounds" exemption — ISV-resident for tunability, not because
required.
Diag exposes q_distill_kl_target + reward_scale_min so the new
adaptation chains are observable.
Investigation (ultrathink): actions 7/8 (TrailTighten/TrailLoosen)
have ZERO consumers across the codebase. Spec'd as "ISV mutation"
in actions_to_market_targets.cu header but no slot, no mutation
kernel, no stop-check kernel. ~10% of wwcsz policy mass goes to
dead no-ops. Documented in
`pearl_dead_trail_stop_actions_a7_a8.md` — implementation
deferred to its own SP (per-batch trail_distance buffer +
mutation kernel + stop-check integration with LobSim apply_fill_to_pos
per `pearl-stop-checks-run-at-deadline-cadence`). N_ACTIONS=9
preserved; alternative refactor to 7 actions captured as
"Path B" in the pearl.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
185add7dc8 |
feat(rl): adaptive RATIO + EWMA V_MIN/V_MAX + λ_distill bump
wwcsz analysis identified atom-resolution starvation + asymmetric
RATIO mismatch as the empirical ceiling on win rate (38.56% vs
break-even 45.3%). Three coupled fixes shipped in one pass per
"no deferrals":
(a) Adaptive RATIO from observed |loss|/|win| EMAs:
- apply_reward_scale tracks max(-scaled, 0) per step (slot 489)
- rl_reward_clamp_controller maintains neg_max_ema (slot 490,
sparse-aware like pos_max_ema)
- RATIO = clamp(MIN=1.0, neg_ema/pos_ema, MAX=3.0); writes to
slot 481
- Removes built-in 3:1 loss-aversion bias when reality is
symmetric (wwcsz: actual avg|loss|/avg(win) = 0.83). Floor
1.0 prevents inverted asymmetry; ceiling 3.0 preserves
original loss-aversion as the worst case.
(b) C51 V_MAX/V_MIN: ratchet → slow EWMA (α=0.001, half-life ~700
steps):
- Static ratchet wasted atom resolution on rare tails — wwcsz
had V_MIN=-60, V_MAX=20 but realized rewards mostly in [-5, +5]
(Δz=4 vs typical reward magnitude 1-5)
- Slow EWMA lets atom span shrink toward active reward range,
gaining resolution where data lives. Floors at [-1, +1]
preserve original C51 baseline as the worst case.
- Slow α gives Q's atom mapping time to be valid across
encoder/head co-adaptation (vs aggressive EWMA which would
invalidate Q's learned distribution every step)
(c) Q→π distillation λ bumped 0.01 → 0.05:
- wwcsz showed KL dropped 2.10 → 0.30 with λ=0.01 — Q signal
landing but conservatively. Bump tests whether stronger Q
pull translates to better policy → better R/done.
Diag exposes neg_scaled_max + neg_scaled_max_ema so the RATIO
adaptation chain is observable.
apply_reward_scale shared_mem doubled from 2× to 3× block × f32
to fit the three parallel reductions (abs, pos, neg).
Companion to investigation (e) — n_rollout_steps controller was
suspected of misalignment (256-8192 vs trade_duration ≈ 6 steps)
but turned out to be a K-loop param, not used in Bellman target.
1-step Bellman with γ-bootstrap is the actual mechanism; closed
without code change.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
79756a2153 |
fix(rl): sparse-aware EMA + Q→π distillation breaks defensive trap
Two coupled fixes addressing vj5f6 findings:
(1) WIN_clamp oscillation — sparse-aware EMA
vj5f6 showed WIN_clamp oscillating 1.0 ↔ 67.0 across 40k steps.
Root cause: the Wiener-α blend in rl_reward_clamp_controller
treated pos_max=0 as "no win this step ≡ win magnitude is zero,"
exponentially decaying the EMA toward 0 during dry-spell windows
(no closed winning trades). With α=0.4, ten dry steps decayed EMA
by 0.6^10 ≈ 0.006, collapsing WIN back to MIN_WIN=1.0 floor.
Fix: only update pos_max_ema AND clip_rate_ema AND MARGIN when
pos_max > 0. A dry step is "no signal," not "zero signal." The
EMA retains its last winning-period estimate; the controller
doesn't ratchet on stale data.
(2) Q→π distillation — couples Q's improved calibration to π
vj5f6 showed l_q dropping 100× (2.37 → 0.02) but reward economics
IDENTICAL to 8xwq8 (no C51 V_MAX lift). Per Option B, π drives
action selection but is trained by PPO surrogate using advantage
= returns - V. V regression doesn't benefit from C51 calibration,
so Q's improved knowledge stays trapped in the critic head.
Deep audit revealed a self-reinforcing defensive trap:
Q learned "big positions lose money" → π_target favors small
actions → π picks a3+a4 (tiny long / Hold) → position lots ≈ 0
→ rewards mostly 0 → V learns "everything is 0" → V_pred ≈ 0
→ advantage = returns - V_pred ≈ 0 → PPO gradient ≈ 0 → π
frozen at defensive attractor → loop. Trade count dropped 3×
(rdgzl 25k → 8xwq8/vj5f6 9k closes per 10k steps), win rate
inversely correlated with l_q (50% early → 22% late) because
only forced closes happen (stops = losses).
Fix: new rl_q_pi_distill_grad.cu computes
π_target = softmax(E_Q[s,*] / τ)
∂L/∂logits[a] = λ × (π_new(a) - π_target(a))
and ADDS this gradient to pi_grad_logits AFTER the PPO surrogate
backward. Couples Q's preferences directly into π's update without
going through advantage. λ=0.01 (small, PPO dominant), τ=1.0
(canonical Boltzmann). 3 new ISV slots (λ + τ + KL_ema diag).
Diag exposes c51_v_max/v_min, q_distill_lambda/temperature, and
q_distill_kl_ema so the adaptation + distillation loop is observable.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
2d498bec3a |
feat(rl): adaptive C51 atom span ratchet to lift Q learning ceiling
rdgzl follow-up — chain hypothesis layer 2:
reward clamp lift unlocked V regression + PPO advantage (R/done
-$1.39 → -$0.48), but Q's distributional learning was structurally
capped at hardcoded V_MAX=1.0 in bellman_target_projection.cu —
any Bellman target > 1.0 categorically projected to atom 20 (top)
regardless of clamp. Even with WIN=3.8 clamp, Q never saw a +3.8
reward signal as distinct from a +1.0 reward signal.
This commit makes V_MIN/V_MAX ISV-driven with monotone-grow ratchet
coupled to the reward clamp. The C51 distribution support adapts
WITHOUT destabilising Q's learned values — atom 20 always represents
at least the widest WIN we've ever admitted (only grows, never shrinks).
Implementation:
- 2 new ISV slots (484 V_MAX, 485 V_MIN) with [-1, +1] floors
seeded by rl_isv_write
- rl_reward_clamp_controller.cu also ratchets these slots:
V_MAX_new = max(V_MAX_prev, max(1.0, WIN_clamp))
V_MIN_new = min(V_MIN_prev, min(-1.0, -LOSS_clamp))
- bellman_target_projection.cu reads V_MIN/V_MAX from ISV, derives
DELTA_Z inline (was #define)
- New rl_atom_support_update.cu (21-thread block) refreshes
atom_supports_d = linspace(V_MIN, V_MAX, 21) per step so
downstream C51 kernels (argmax_expected_q, rl_action_kernel,
dqn_distributional_q) see the current span
- Trainer launches atom-support updater after each reward-clamp
controller launch (both helper + step_with_lobsim inline paths)
- Diag exposes c51_v_max + c51_v_min for adaptation visibility
Floors at [-1, +1] preserve original C51 design as hard minimum —
the atom support can only become wider, never narrower than the
baseline.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
13084f7746 |
feat(rl): MARGIN adaptive from clip-rate + remove MAX_WIN cap
rdgzl follow-up — chain hypothesis test:
clip rate stayed at 25-40% across windows (target ~5%)
win rate oscillated 27-47% with no clear trend
positive-tail distribution: p50=1.85 p90=10.1 p99=76.9 max=2230
MAX_WIN=20 hit ceiling in EVERY window (load-bearing cap)
static MARGIN=1.5 couldn't chase the tail
Two interventions in one commit:
(1) MARGIN is now adaptive in rl_reward_clamp_controller.cu via a
Schulman bounded-step on clip-rate EMA vs target:
clip_indicator = (pos_max > current_WIN && pos_max > 0) ? 1 : 0
clip_rate_ema = (1-α) * prev + α * indicator (α=0.05)
if clip_rate_ema > target × 1.5 → MARGIN *= 1.2 (up to MAX_MARGIN=5)
if clip_rate_ema < target / 1.5 → MARGIN /= 1.2 (down to MIN_MARGIN=1)
Target clip rate seeded at 0.05 — accept 5% tail outliers, capture
the rest. Two new ISV slots (482 clip-rate EMA, 483 target).
(2) MAX_WIN cap REMOVED — the hardcoded ceiling defeated the purpose
of adaptation. Safety reasoning: WIN = MARGIN × pos_max_ema with
MARGIN ∈ [1, 5] and pos_max_ema bounded by reward_scale × raw_PnL
(both finite). MIN_WIN=1.0 floor retained.
Diag exposes clip_rate_ema + reward_clamp_clip_rate_target so the
adaptation loop is observable in the JSONL.
KNOWN DOWNSTREAM CEILING: bellman_target_projection.cu hardcodes C51
atom span at V_MIN=-1.0, V_MAX=+1.0. Any Bellman target outside this
range is categorically clipped regardless of our reward clamp. So
lifting WIN > 1.0 helps V regression + PPO advantage (which see real
magnitude) but Q's distributional learning is structurally capped at
V_MAX=1.0. A separate intervention to lift C51 V_MAX would be needed
to unlock Q's atom-distribution learning beyond +1.0.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
51b9f46364 |
feat(rl): adaptive reward clamp from positive-tail EMA
alpha-rl-rmgm5 (commit
|
||
|
jgrusewski
|
a776fab31f |
fix(rl-cli): build B×K snapshot tensor per step at b_size>1
Crash in alpha-rl-ljn8k (commit
|
||
|
jgrusewski
|
9c6c280bd8 |
fix(rl): anti-collapse probability floor + argo b_size=16 default
Two fixes for alpha-rl-9k9x6 (commit |
||
|
jgrusewski
|
a01a376bd2 |
audit: split K-loop to DQN-only (avoid PPO/V overshoot at high K)
Per `pearl_q_thompson_actor_makes_pi_dead_weight` follow-up + #35
deferral: the K-loop in step_with_lobsim was running full
step_synthetic K times per env step (Q + π + V + encoder + LR
controller emit + ISV refresh + EMA inputs). At K=4 (default) or K=8
(prior K_MAX) this caused PPO overshoot — KL excursions to 12.44 in
f2ggr, policy drift faster than the env step rate, gradient
overtraining on the same env-step's h_t.
## Fix: extract dqn_replay_step helper
New public method `dqn_replay_step(b_size)`:
1. Forward Q on sampled_h_t + sampled_h_tp1 (Double-DQN argmax)
2. Bellman target via TARGET net at h_tp1 + select + project
3. Q backward (logits → grad_w/b/h_t)
4. Per-batch reduce → grad_w/grad_b
5. Q Adam (uses LR already set by step_synthetic — no re-fire of
the LR controller per K iter)
6. Writes td_per_sample_d for PER priority update by caller
Discards Q's grad_h_t per R7d stop-grad (same as step_synthetic).
What dqn_replay_step does NOT do:
* π forward / surrogate / Adam — runs once per env step in
step_synthetic
* V forward / backward / Adam — same
* Encoder backward / grad combine — same
* LR controller emit + ISV mirror refresh — same
* EMA inputs (entropy, KL, advantage_var, td_kurtosis) — same
## K-loop in step_with_lobsim
for k_iter in 0..k_updates {
let per_indices = sample_and_gather(b_size)?;
if k_iter == 0 {
stats = step_synthetic(snapshots)?; // full update
} else {
dqn_replay_step(b_size)?; // Q-only
}
// PER priority update
}
Result:
* Q gets K Adam updates per env step (K-fold variance reduction)
* π + V + encoder get 1 Adam update per env step (no overshoot)
* LR controllers fire once per env step (no double-counting of
plateau detection)
* At b_size=16 with low advantage_var_ratio (batch averaging
reduces noise), K-loop typically settles at K=1 — the split
becomes a no-op in the steady state. At b_size=1 fallback or
high-noise regimes, the split materially reduces PPO drift.
## Code duplication
dqn_replay_step duplicates ~120 lines of Q-section code from
step_synthetic. Acceptable temporary tech debt — full dedupe would
require restructuring step_synthetic to call dqn_replay_step
internally, which is a larger refactor with regression risk. Marked
TODO for a follow-up commit once the b_size=16 + π-actor + K-split
architecture is empirically validated.
## Verified gates (local sm_86)
G1 isv_bootstrap ✅
G3 controllers ✅
G4 target_update ✅
integrated_smoke ✅
## No smoke yet
alpha-rl-9k9x6 (commit
|
||
|
jgrusewski
|
3737feb664 |
audit: π drives actions (proper actor-critic) + bump b_size 1 → 16
Two coordinated architectural fixes addressing the deepest blockers
exposed by the audit:
## Option B: π-driven action selection
Per `pearl_q_thompson_actor_makes_pi_dead_weight`: the prior
architecture had Q acting as BOTH actor (via Thompson sample) AND
critic (via Bellman target). π trained by PPO surrogate against
Q's actions but never drove any decision — `q_pi_agree_ema`
decayed to 0 by step 5000 in every smoke because π converged to
Q's Thompson SAMPLING distribution, not Q's argmax. π was
dead-weight: 4 dedicated controllers (ε, ratio_clamp,
entropy_coef, KL EMA), shared encoder gradient interference, and
zero contribution to actor decisions.
### New kernel: rl_pi_action_kernel.cu
Single-thread-per-batch CUDA kernel that:
1. Computes numerically-stable softmax(pi_logits[b, :])
2. Draws u ∈ [0, 1) from per-batch xorshift32 PRNG
3. CDF-walks to pick the multinomial-sampled action
Per-batch xorshift32 PRNG state is the SAME `prng_state_d` buffer
already used by rl_action_kernel — no new state needed. Sampling
deterministic given (seed, b_size, pi_logits).
### Trainer wiring (1 site change in step_with_lobsim)
Replaced `rl_action_kernel(q_logits, atom_supports, ...)`
(Q-Thompson) with `rl_pi_action_kernel(pi_logits, ...)`
(π-multinomial). The argmax_expected_q call on h_{t+1} is
unchanged — Q remains the critic via canonical Double-DQN target.
PPO importance-ratio surrogate now has its canonical actor-critic
semantics: π_new(a|s) / π_old(a|s) where `a` was actually sampled
from π_old. Was nonsensical before (a was sampled from Q-Thompson,
not π, so the ratio measured something incoherent).
The rl_action_kernel (Q-Thompson) cubin + function field are kept
loaded for backward-compat tests and diagnostic comparison; no
longer in the hot path.
## b_size: 1 → 16
Per `pearl_b_size_1_signal_starvation_blocks_q_learning`: at
b_size=1 with 11% done-step rate and 70% loss rate per trade, Q
stayed at uniform baseline ln(21)=3.04 across all 16+ smokes
regardless of controller fixes. The architecture was structurally
signal-starved — 1 gradient sample per Adam step is fundamentally
too noisy.
LobSimCuda already supports b_size>1 (n_backtests parameter at
`crates/ml-backtesting/src/sim/mod.rs:355`). Trainer code is
already b_size-parametric throughout. The blocker was just the
CLI default at `--n-backtests=1`.
Default bumped to 16 (matches the doc note "production sweep at
32-64; L40S 48GB"). 16× more gradient samples per Adam step
gives Q proper batch variance reduction. The K-loop multiplier
(`isv[404]/2048`) will likely settle at K=1 since the
advantage_var_ratio drops with batch size.
## Expected behaviour
* `q_pi_agree_ema` becomes tautological/dropped (π IS the
policy now — comparing argmax(Q) to argmax(π) doesn't measure
a real consistency invariant any more)
* π gradient flows naturally drive π toward an actor that
optimises the PPO surrogate — Q's encoder gradient is no
longer competing with a different policy's gradient
* l_q should drop meaningfully below 3.04 for the first time
(was stuck at 2.7-2.9 across all prior smokes)
* reward/trade should approach 0 (was -$0.5 to -$0.8 across
every prior run)
* Wall-clock per env step ~16× slower (b_size=16) but training
cost per gradient step similar (denser sample = more
progress per step)
## Verified gates (local sm_86)
G1 isv_bootstrap ✅
G3 controllers ✅
G4 target_update ✅
integrated_smoke ✅
## Caveat: integrated_trainer_smoke runs at b_size=1
The default for the CLI is bumped to 16, but the local
`integrated_trainer_smoke` test passes its own b_size=1 to
verify the trainer mechanics. Real-world signal verification
happens via cluster smokes which now use b_size=16 by default.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
705d6c156b |
audit: ISV-ify 10 more design constants — Schulman + bootstraps + streaming α
Per `feedback_isv_for_adaptive_bounds` + user "do all except floors
and clamp bounds": 10 more constants moved from kernel-side `#define`s
into ISV slots (78 slots total now).
## Slot additions (468-477)
RL_SCHULMAN_TOLERANCE_INDEX (468, =1.5) — shared by 4 controllers
RL_SCHULMAN_ADJUST_RATE_INDEX (469, =1.5) — shared by 4 controllers
RL_STREAM_ALPHA_INDEX (470, =0.05) — shared by var + kurt streaming
RL_KURT_GAUSSIAN_INDEX (471, =3.0)
RL_KURT_NOISE_FLOOR_INDEX (472, =1.0)
RL_TAU_BOOTSTRAP_INDEX (473, =0.005)
RL_EPS_BOOTSTRAP_INDEX (474, =0.2)
RL_ROLLOUT_BOOTSTRAP_INDEX (475, =2048)
RL_REWARD_SCALE_BOOTSTRAP_INDEX (476, =1.0)
RL_PPO_RATIO_CLAMP_BOOTSTRAP_INDEX (477, =10.0)
## Skipped (per user "do all except floors and clamp bounds")
* `*_MIN`/`*_MAX` clamp bounds (algebraic domain — risk γ=1.5 nonsense)
* Numerical floors: ABS_MEAN_FLOOR=1e-6, M2_SQ_FLOOR=1e-12, EPS_PNL=1e-3
(risk div-by-zero if mis-tuned)
* C51 atom layout (V_MIN/V_MAX) — architecture, not config
## Wiring
* Shared Schulman pattern: 4 controllers (ppo_clip, target_tau,
rollout_steps, plus per_α independent KURT slots) now read TOLERANCE
+ ADJUST_RATE from the same 2 ISV slots. Single source of truth.
* Each controller's bootstrap (1st-emit on sentinel-zero) reads
isv[*_BOOTSTRAP_INDEX] instead of #define value. The `prev ==
BOOTSTRAP` first-observation replace-direct check also reads from
ISV.
* 2 streaming kernels (var + kurt) share RL_STREAM_ALPHA_INDEX.
## Diag bake-in
JSONL `isv_config` block grows by 10 new fields: schulman_tolerance,
schulman_adjust_rate, stream_alpha, kurt_gaussian, kurt_noise_floor,
tau_bootstrap, eps_bootstrap, rollout_bootstrap,
reward_scale_bootstrap, ppo_ratio_clamp_bootstrap. Total isv_config
fields: 26.
Also includes windowed action_entropy fix (was structurally 0 at
b_size=1) — accumulates EMA-smoothed action distribution over
~1k-step window, computes entropy on the windowed dist. Makes the
exploration metric meaningful at b_size=1.
## Slot total
RL_SLOTS_END: 468 → 478. **78 total ISV slots.**
## Verified gates (local sm_86)
G1 isv_bootstrap ✅ (with 10 new assertions)
G3 controllers ✅
G4 target_update ✅
integrated_smoke ✅
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
827a0e9416 |
fix(rl): ISV-ify ALL remaining tunable design constants (10 new slots)
Per `feedback_isv_for_adaptive_bounds`: every controller design knob
that's genuinely tunable now lives in ISV instead of as a kernel-side
`#define`. Tuning is a re-seed (kernel launch with new arg) rather
than a recompile.
## New ISV slots (10 design constants)
RL_REWARD_CLAMP_WIN_INDEX (452, =1.0) apply_reward_scale
RL_REWARD_CLAMP_LOSS_INDEX (453, =3.0) apply_reward_scale
RL_KL_TARGET_INDEX (454, =0.01) rl_ppo_clip_controller
RL_IMPROVEMENT_THRESHOLD_INDEX (455, =0.99) rl_lr_controller
RL_PLATEAU_PATIENCE_INDEX (456, =1000.0) rl_lr_controller
RL_DIV_TARGET_INDEX (457, =0.01) rl_target_tau_controller
RL_ENTROPY_TARGET_FRAC_INDEX (458, =0.7) rl_entropy_coef_controller
RL_KURT_LIFT_SCALE_INDEX (459, =7.0) rl_per_alpha_controller
RL_PPO_CLAMP_MARGIN_INDEX (460, =10.0) rl_ppo_ratio_clamp_controller
RL_LR_WARMUP_STEPS_INDEX (461, =2000.0) rl_lr_controller
RL_SLOTS_END: 452 → 462.
## Constants NOT converted (truly fundamental)
* All `*_INDEX` (ABI)
* All `*_MIN`/`*_MAX` clamp bounds (algebraic domain)
* All `*_BOOTSTRAP` (one-shot init)
* `WIENER_ALPHA_FLOOR` (per pearl_wiener_alpha_floor_for_nonstationary)
* Schulman pattern parameters (`*_TOLERANCE`/`*_ADJUST_RATE`)
* C51 (`Q_N_ATOMS`, `V_MIN/MAX`, `N_ACTIONS`)
* Kernel numerics (`STREAM_ALPHA`, `ABS_MEAN_FLOOR`, `EPS_PNL`)
* `KURT_GAUSSIAN` (statistical constant = 3.0 for Gaussian)
* `KURT_NOISE_FLOOR` (defensive)
* `LR_BOOTSTRAP`/`LR_MIN`/`LR_MAX`/`LR_LOSS_EMA_ALPHA`/`DECAY_FACTOR`
## New infrastructure
New CUDA kernel `rl_isv_write.cu` — generic single-thread device-side
seeder taking `(int slot, float value)`. Trainer loops calling it
once per design constant at init. Replaces the prior pattern of
extending `rl_streaming_clamp_init`'s arg list every time a new
constant was added.
## Ordering fix
Design constants must be seeded BEFORE controllers bootstrap — the
controllers' bootstrap paths read these slots (e.g.
`rl_entropy_coef_controller` reads `RL_ENTROPY_TARGET_FRAC_INDEX`
to derive its target). Without correct ordering, controllers see
sentinel 0.0 and bootstrap to wrong values (caught by failing G1
test before commit). Seed loop runs at TOP of
`with_controllers_bootstrapped`.
## Diag bake-in
JSONL gains `isv_config` block exposing all 10 design constants per
step:
isv_config.{reward_clamp_win, reward_clamp_loss, kl_target,
improvement_threshold, plateau_patience, div_target,
entropy_target_frac, kurt_lift_scale, ppo_clamp_margin,
lr_warmup_steps}
Post-hoc analysis can correlate any controller's behaviour with the
exact design constants it saw, without grepping the source for
`#define` defaults.
## Test updates
G1 (isv_bootstrap) + G3 (r5_controllers) — skip 10 new design-
constant slots in sentinel-zero loop, assert seeded values
separately.
## Verified gates (local sm_86)
G1 isv_bootstrap ✅ (with 10 new assertions)
G3 controllers ✅
G4 target_update ✅
integrated_smoke ✅
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|