The CLI default overrode the trainer config default. Without passing
--per-capacity explicitly, the Argo run used 4096 (256 unique steps
at b=16) instead of the intended 32768 (2048 steps).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Root cause of Q non-convergence: at reward_scale ≈ 0.003, a typical
±$5 trade maps to ±0.015 scaled reward. With 21 atoms spanning [-1,+1]
(step=0.1), the entire reward distribution falls within a SINGLE atom.
Q cannot distinguish wins from losses — confirmed by q_pi_agree ≈ 0.
Fix 1: Tighten atom span to [-0.5, +0.5]. Now atom_step = 0.05.
A ±$5 trade at scale=0.01 spans 2 atoms; at scale=0.05 spans 10.
The reward clamp controller ratchets the span at runtime if rewards
exceed the bounds.
Fix 2: PER capacity 4096 → 32768. At b=16, old capacity retained
only 256 unique steps — Q replayed each transition 1-2× before
eviction. New capacity retains 2048 steps, giving Q proper replay
depth for convergence.
Also aligns reward clamp bounds (WIN/LOSS) with the new span.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
With strict >, pos=8 passed the cap check, then a LongLarge(+2)
made pos=10 before the next step's cap fired. Using >= prevents
position from ever exceeding the cap by the fill size.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Controller oscillated: 3 dones at step 15k spiked EMA above 2×target,
triggering thousands of steps of tightening that killed trades.
Higher target (10% vs 2%) matches the natural trade frequency at
b=16. Wider dead zone (5× above / 0.2× below) prevents single-batch
spikes from triggering tighten/relax oscillation.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Confidence gate LCB fix worked (0 fires post-warmup) but FRD gate
blocked 5212 entries in 5k steps — the FRD head learned "forward
returns are negative" during warmup (losing trades), so favorable
tail mass dropped below 0.15 for all actions.
Lower default to 0.05 and floor to 0.0 so the adaptive controller
can fully disable when trades dry up.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The old formula conf = clamp((μ - λσ) / σ_norm, 0, 1) produced
conf=0 for ANY distribution with σ > μ — including uniform Q at
training start (μ=0, σ=0.577 → LCB=-0.577 → conf=0). This caused
permanent 100% block rate regardless of threshold setting.
New formula: conf = clamp((μ - V_MIN - λσ) / (V_MAX - V_MIN), 0, 1)
Shifts by V_MIN so conf measures "how far above worst case":
Uniform: conf = 0.21 (passes threshold 0.01)
Peaked V_MAX: conf ≈ 1.0 (high confidence)
Peaked V_MIN: conf ≈ 0.0 (blocked — correct)
Also sets conf gate floor to 0.0 so the adaptive controller can
fully disable the gate when trades dry up.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
When C51 LCB is ≈0 for all actions (typical early training), even
0.001 threshold blocks everything. Floor=0.0 lets the controller
drive to zero when trades dry up, then ratchet back as Q calibrates.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New rl_gate_threshold_controller.cu watches dones EMA and adjusts
confidence + FRD gate thresholds via Schulman bounded step:
- dones_ema < target×0.5 → relax thresholds (×0.95)
- dones_ema > target×2.0 → tighten thresholds (÷0.95)
ISV-driven: target (0.02), conf bounds (0.001-0.50), FRD bounds
(0.05-0.50), adjust rate (0.95). Runs per step after warmup.
Also lowers default thresholds: conf 0.10→0.01, FRD 0.35→0.15.
Post-warmup, the controller adapts these based on actual trade
flow instead of relying on static defaults.
ISV slots 525-531. RL_SLOTS_END → 532.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Both gates now read RL_GATE_WARMUP_STEPS_INDEX (slot 524, default
10000) and return early when current_step < warmup. During warmup,
the agent opens positions freely, collects reward signal, and
calibrates Q. After warmup, gates activate and filter low-quality
entries.
Without warmup, gates blocked 100% of opening actions from step 0
(uniform Q → zero confidence → permanent Hold attractor → zero
trades → zero reward → Q never learns). Confirmed by 50k-step
L40S run with zero dones across 800k action decisions.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replaces the 4-step DAG (check-cache → compile on CPU → warmup GPU →
train on GPU) with a single pod on the L40S that does everything:
git fetch → incremental cargo build (~3s warm) → train.
Eliminates: separate compile node, node autoscale wait, binary
transfer, fxcache step. The ci-builder image has CUDA 13.0 devel +
Rust — compilation and training use the same CUDA libs.
The cargo-target-cuda PVC provides incremental build cache across
runs. LD_LIBRARY_PATH strips the stubs dir so real CUDA libs load.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The MBP-10 predecoded sidecars live on feature-cache-pvc at
/feature-cache/predecoded/, not on training-data-pvc. Points
--predecoded-dir to the correct PVC mount.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
alpha_rl_train reads MBP-10 data with its own predecoded cache —
it never uses the fxcache pipeline. Skip the 15-minute feature
extraction step entirely for model=alpha-rl by depending only on
ensure-binary + gpu-warmup.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The ml-alpha branch uses alpha_rl_train (from ml-alpha crate) with
different CLI args than train_baseline_rl (from ml crate). Adds
model=alpha-rl case that:
- Builds ml-alpha --example alpha_rl_train
- Runs with --mbp10-data-dir, --predecoded-dir, --n-steps 50000,
--n-backtests 16 (matching the local smoke config)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
28 parallel alpha-feature chunks × ~3GB each = 84GB peak at merge.
Capping at 8 threads keeps peak at ~24GB (well within 96Gi limit)
while still saturating I/O. This is a one-time cost — after the
cache is populated on the PVC, subsequent runs skip entirely.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
With cargo-target-cuda PVC persisting across pods, CARGO_INCREMENTAL=1
gives true incremental builds: only recompile changed crates. A
single-crate ml-alpha change rebuilds in ~45s vs 9min full workspace.
sccache removed — it conflicts with incremental compilation and is
redundant when the target dir persists. The binary-by-SHA cache on
the training-data PVC (already implemented) provides the "skip
compile entirely on re-run" fast path.
Expected workflow time reduction:
Before: compile 9m + fxcache 18m + train ~30m = ~57m
After: compile <1m (incremental) + fxcache <1m (PVC cache) + train ~30m = ~32m
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The imbalance bar cache now writes to <mbp10_dir>/.cache/ on the PVC.
The ensure-fxcache step needs write access to persist the cache
across runs (was readOnly: true, causing silent cache-write failures).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Cache path moved from /tmp/.foxhunt_imbalance_cache/ (ephemeral,
lost between Argo pods) to <mbp10_dir>/.cache/ (on the training-data
PVC). Saves ~4 minutes per Argo submission by avoiding recomputation
of 209M trade ticks → 17.8M imbalance bars.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Completes P2 by injecting rl_encoder_context_broadcast launch inside
the perception trainer's forward_only path — runs after
snap_feature_assemble fills dims [0..40] and before VSN/Mamba2
consume the window tensor. Fills dims [40..56] with per-batch
trade_context (4) + multires (12) features.
Device pointers to the context buffers (owned by IntegratedTrainer)
are set on PerceptionTrainer before each forward_encoder call. The
broadcast kernel reads these and writes directly into window_tensor_d
at the correct column offsets for all B×K rows.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Introduces ENCODER_INPUT_DIM = 56 (FEATURE_DIM + 16). All encoder
first-layer weight matrices (VSN gate, Mamba2 L1 input projection)
now sized for 56 input dims. The extra 16 are per-batch state:
4 trade_context + 12 multires features.
- snap_feature_assemble_batched: output stride → ENCODER_INPUT_DIM,
zero-fills dims [40..56] for the broadcast kernel to overwrite.
- New rl_encoder_context_broadcast.cu: writes trade_context_d[B×4]
+ multires_output_d[B×12] into each of the K sequence rows per
batch at positions [40..56].
- CfcConfig.n_in, Mamba2 L1 in_dim, VSN gate, window_tensor_d,
all forward/backward scratch buffers updated to ENCODER_INPUT_DIM.
- CfcTrunk default config updated.
The broadcast kernel launch integration into the forward_only path
is the final wire-up step — until then dims 40-55 are zero-filled
(safe: Xavier init on new columns means encoder starts by learning
to ignore them, then gradually incorporates the signal).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
P1: New rl_trade_context_update.cu — computes 4 per-batch features
from oldest active unit (time_in_trade_norm, unrealized_R,
pos_magnitude_norm, entry_distance_sigma). Output in
trade_context_d[B×4], updated after unit_state_update each step.
P0: New rl_multires_features_update.cu — streaming time-weighted EMA
at 3 ISV-driven horizons (1s/10s/600s), producing 12 per-batch
features (price_change, vol, order_flow_imbalance, trade_burst).
O(1) state per feature vs circular buffer — same time-constant
semantics.
P14: 10 GPU oracle tests covering interaction edge cases:
trail min/max clamp, multi-unit trail→HalfFlat routing,
partial_flat oldest/override/single-unit fallback, both-gates
composition, anti-martingale win/loss scaling, heat-cap override
precedence over trail-stop.
ISV slots: 521-523 (multires horizons). RL_SLOTS_END → 524.
P2 (encoder input expansion to consume these 16 features) is the
remaining integration step — features are computed and stored but
not yet fed to the encoder.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
P10: New rl_recent_outcome_update.cu — per-batch signed outcome EMA
(sign(reward) on done steps) feeds per-batch anti-martingale
sizing in actions_to_market_targets. Replaces the single ISV
scalar with a per-batch buffer for multi-batch granularity.
P11: Trail bootstrap switched from vwap × 1e-3 × k_init to
k_init × MEAN_ABS_PNL_EMA (slot 423). Vol-derived trail
distance adapts to realized trade magnitude as the EMA updates.
P12: P_MIN in rl_pi_action_kernel now ISV-driven (slot 519,
default 0.015). At N=11, max single-action prob = 0.85
(uplift vs prior 0.80 at hardcoded P_MIN=0.02).
ISV slots: 519 P_MIN, 520 OUTCOME_ALPHA. RL_SLOTS_END -> 521.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New rl_frd_gate.cu kernel reads the FRD head's horizon-2 (medium,
~300 ticks) categorical distribution. For long openings, sums
probability mass in the positive tail (atoms > +0.5σ); for short
openings, sums the negative tail (atoms < -0.5σ). Overrides to Hold
when favorable mass < threshold.
Fires after confidence gate, before trail/heat/market pipeline.
Same preconditions: only gates flat positions with opening actions.
ISV slots: 516 THR_LONG (0.35), 517 THR_SHORT (0.35),
518 fired_count (diag). RL_SLOTS_END → 519.
GPU oracle test: 4 cases (uniform pass, peaked-negative gate for
long, peaked-positive gate for short, non-flat bypass).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New rl_confidence_gate.cu kernel computes C51 distributional Lower
Confidence Bound (μ - λσ) / σ_norm for the chosen action. When
position is flat and the selected action is an opening (a0/a1/a5/a6),
overrides to Hold if conf < threshold.
Fires after π action selection, before trail/heat/market pipeline.
Only gates on flat positions — existing positions pass through
unconditionally regardless of Q uncertainty.
ISV slots: 512 threshold (0.10), 513 λ (1.0), 514 σ_norm (1.0),
515 fired_count (diag). RL_SLOTS_END → 516.
GPU oracle test: 4 cases (low-conf gate, high-conf pass, non-flat
bypass, non-opening bypass).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Fixes the pre-existing fxcache_local_smoke test failure. Two changes:
1. Hash update: `13c0b086a975...` → `70e5bc3a401d...` — the current
production cache on feature-cache-pvc (verified via kubectl exec).
Both the local file (15.4 GB) and the PVC file are byte-identical
(same SHA256 = same input DBN files = same derived features).
Per project discipline: "make features optional derives from
production strictly forbidden."
2. Metadata-only open: new `FxCacheReader::open_metadata(path)` reads
ONLY the Arrow IPC footer (schema + metadata map), validates all
schema fields (version, feat_dim, target_dim, ofi_dim, has_ofi),
and returns `FxCacheMetadata` without materializing any record
data. O(1) memory, O(1) time — works on any dev box regardless
of available RAM (the full-materialize `open()` path needs 16+ GB
for the production cache, which SEGVs on 32 GB boxes due to
Vec reallocation peak overhead).
Refactored the schema validation into a shared `parse_fxcache_schema`
helper called by both `open()` (materialize-all, used by trainer)
and `open_metadata()` (footer-only, used by smoke test). Single
source of truth for field parsing + dim-mismatch assertions.
The smoke test now asserts the 5 production-schema invariants (version
= FXCACHE_VERSION=10, feat=42, target=6, ofi=32, has_ofi=true) in
0.00s with zero memory overhead. Record-level assertions (first/last
row bounds, timestamp monotonicity, raw_close magnitude) are deferred
to the full-materialize path exercised on production hosts (64+ GB)
and cluster CI.
Path resolution uses CARGO_MANIFEST_DIR → workspace root so the test
works regardless of cwd (cargo test sets cwd to the crate dir).
Closes the F.5 diagnosed l_v=104 + l_pi=-31 spike pattern at the root.
Pathology: `rl_reward_clamp_controller` widened the WIN/LOSS bounds
(slots 452/453) when clip_rate exceeded the 5% target — appeasement,
not control. The atom-span EWMA (slots 484/485) then ratcheted up to
track the wider WIN/LOSS. F.5 200-step smoke trajectory:
WIN: 1.0 → 41.3 (41×)
LOSS: 3.0 → 41.3 (14×)
V_MAX: 1.0 → 2.66
V_MIN: -1.0 → -2.74
Scaled rewards up to 14.71 flowed through unclamped, producing
advantage magnitudes of ~30 and PPO surrogate losses of ±30, V
regression losses up to 104. Pure positive-feedback loop: large
rewards → wider clamp → bigger V/Q targets → larger atom span →
larger reward signals permitted → repeat.
Fix: STOP writing to slots 452/453/484/485. The trainer-seeded
values (WIN=1.0, LOSS=3.0, V_MAX=1.0, V_MIN=-1.0) are the structural
bounds matching the C51 distributional Q head's design. Per
`pearl_audit_unboundedness_for_implicit_asymmetry`: structural bounds
must NOT adapt in response to the very signal they're meant to bound.
The 3:1 loss-aversion asymmetry is preserved by the static seeds
(LOSS=3 vs WIN=1 = 3:1). The C51 distributional resolution stays
matched to the bound. Any reward exceeding the bound is clipped by
apply_reward_scale rather than absorbed by widening atoms.
Diagnostic-only state retained:
* pos_max_ema (slot 478) — observed positive-tail magnitude
* neg_max_ema (slot 489) — observed negative-tail magnitude
* clip_rate_ema (slot 482) — fraction of steps where clamp fired
* MARGIN (slot 480) — what the controller WOULD widen to
* RATIO (slot 481) — what observed LOSS/WIN ratio implies
These surface what an unbounded controller WOULD adapt to under the
observed reward distribution — useful for understanding drift even
though the LOAD-BEARING slots are now static.
F.5 vs G.2 smoke comparison (same seed=4242, 200 steps, b_size=4):
Pre G.2 Post G.2 Reduction
l_pi abs_max 31.15 8.09 4×
l_v max 103.69 3.60 29×
l_v mean 1.79 0.19 10×
l_pi mean -0.12 0.06 ~stable
l_frd mean 0.43 0.50 unchanged
WIN bound →41.3 1.0 static
LOSS bound →41.3 3.0 static
V_MAX →2.66 1.0 static
V_MIN →-2.74 -1.0 static
Spike steps 20+ 5 ≥4×
Remaining 5 spikes are early-training noise (steps 11-59) that fade
naturally as V/Q converge. After step 59 only one mild spike at
step 131 (l_pi=3.35, l_v=2.75).
Pairs with G.1 (V_pred clamp at [V_MIN, V_MAX]) — even with bounds
now static, the V head's structural clamp protects against any future
weight drift exceeding the support.
Verification:
* cargo check -p ml-alpha → clean
* lib tests 66/66 (default), 5/6 ignored (1 pre-existing
fxcache_local_smoke env failure, unrelated)
* GPU tests: integrated_trainer_smoke 1/1 + frd_head 10/10 +
trade_management_kernels 5/5 → no regression
* audit-rust-consts → 0 flags
v_head_fwd now reads V_MIN/V_MAX from ISV slots 485/484 (same slots
the C51 atom support adapter writes) and clamps the linear output to
that range at the kernel boundary. Bounds advantage magnitude
(|returns − V_pred|) by 2 × V_MAX regardless of stale-V state.
Defensive fix per pearl_clamp_v_target_at_atom_span +
pearl_c51_atom_span_must_track_clamp_range — protects against the
canonical reward_scale↔V-head response-time pathology where V's stale
predictions amplify into PPO surrogate + V regression spikes when the
controller adapts reward_scale aggressively. In the F.5 200-step local
smoke this clamp didn't bite (V_pred stayed within bounds at the
short run length), but the structural protection matters for longer
production runs where V can drift before the controllers catch up.
Hard-saturated clamp (no straight-through estimator) — the gradient
at the boundary is zero in the "push further out" direction, normal
toward the interior. V can always learn back into bounds when its raw
output drifts out (target is inside bounds → grad pulls V back in),
but cannot push the prediction outside support.
API surface change: `ValueHead::forward(h_t, b_size, v_pred)` →
`ValueHead::forward(h_t, isv, b_size, v_pred)`. The 3 call sites in
IntegratedTrainer (step_synthetic + step_with_lobsim h_t/h_tp1) now
pass `&self.isv_d`.
Verification (RTX 3050 Ti):
* cargo check -p ml-alpha → clean
* integrated_trainer_smoke 1/1 → ok
* frd_head 10/10 + trade_management_kernels 5/5 → no regression
* audit-rust-consts → 0 flags
Independent finding from the smoke diag: the OBSERVED chronic spike
pattern (|l_pi|>30, l_v>100) traces to `rl_reward_clamp_controller`
widening WIN/LOSS bounds to 41.3 (vs seeds 1.0/3.0) when MARGIN hits
its MAX_MARGIN=5 ceiling. That's a separate failure mode addressed
in the next commit (structural cap on scaled reward magnitude).
Closes the literal/const drift gap that F.5 introduced. Per
feedback_isv_for_adaptive_bounds + feedback_single_source_of_truth_no_duplicates:
adaptive bounds belong in ISV (or in a single canonical const that
ISV references), never duplicated as literals across modules.
Single canonical source: `crate::rl::common::FRD_HORIZON_TICKS` +
`FRD_BUCKET_RANGE_SIGMA` (already declared in F.5).
Producer-side fixes:
* Trainer ISV bootstrap (integrated.rs): the seed values for slots
500-503 now dereference the canonical consts instead of hardcoded
60.0/300.0/1800.0/3.0 literals. Future tuning of the consts
automatically propagates to both ISV seeds and loader-side
labels — no manual sync required, no drift possible.
* compute_frd_labels (loader.rs): takes `horizon_ticks` and
`range_sigma` as parameters instead of reading consts directly.
Caller (the file-load closure) sources them from the new
MultiHorizonLoaderConfig fields.
Consumer-side fixes — 8 MultiHorizonLoaderConfig literal sites now
provide the two new fields, all defaulting to the canonical consts:
* crates/ml-alpha/src/data/loader.rs (2 internal test-fixture sites)
* crates/ml-alpha/tests/multi_horizon_loader.rs (2 sites)
* crates/ml-alpha/examples/alpha_train.rs (2 sites)
* crates/ml-alpha/examples/alpha_rl_train.rs (2 sites)
* crates/ml-backtesting/src/harness.rs (1 site)
* crates/ml-backtesting/tests/{trainer_parity,ring3_replay}.rs (2 sites)
The "optimal by default" property is preserved: every caller that
doesn't explicitly override gets the spec-recommended 60/300/1800
ticks + ±3σ. Callers that need to retune set the config fields, and
the trainer's ISV slots provide a runtime knob for the same numerics.
Verification (RTX 3050 Ti):
* cargo check -p ml-alpha -p ml-backtesting --examples --tests → clean
* cargo test --lib (6/6 unit tests for FRD label gen + loss_balance) → pass
* frd_head 10/10 + integrated_trainer_smoke 1/1 + trade_mgmt 5/5 → pass
* audit-rust-consts → 0 flags
The two new MultiHorizonLoaderConfig fields are required (no Default
impl) — callers MUST opt in to the FRD label-generation contract by
naming the fields. This is the same discipline applied across other
config consumers; making them Option<...> would silently default to
"no FRD labels" and break F.4's expected supervised signal.
Activates the FRD head's supervised training signal that F.4 wired
through the trainer. Per-file forward-return σ-bucketed labels
computed at load time + per-step write into trainer.frd_labels_d
before each step_with_lobsim.
Loader-side label generation (`compute_frd_labels` in data/loader.rs):
* Mid-price series from snapshots[i].levels[0]
* Per-file σ_per_step = sample-std of single-tick mid increments
* For each FRD_HORIZON h ∈ {60, 300, 1800}:
- r = (mid[i+h] - mid[i]) / (σ_per_step × sqrt(h)) ← Brownian scaling
- bucket = round(r × (FRD_N_ATOMS-1) / (2 × FRD_BUCKET_RANGE_SIGMA)
+ (FRD_N_ATOMS-1) / 2)
- clamp to [0, FRD_N_ATOMS-1] for tail returns
- sentinel -1 if i + h >= n
* Cached in LoadedFile.frd_labels_full alongside sigma_k_full /
outcome_*_full
* Per-anchor slice into LabeledSequence.frd_labels (length-1 vec
per horizon at the newest-snapshot index — h_t aligns with the
rightmost K position, the only one the FRD head supervises)
New structural constants in rl/common.rs:
* FRD_HORIZON_TICKS = [60, 300, 1800] ← matches ISV slots 500/501/502 defaults
* FRD_BUCKET_RANGE_SIGMA = 3.0 ← matches ISV slot 503 default
Per pearl_glm_fitter_link_must_match_inference: bucket-edge math
here MUST match the trainer-side softmax+CE atom interpretation.
Both reference the same const so they can't drift.
alpha_rl_train per-step wiring:
* Stage frd_labels_bh[b_idx × FRD_N_HORIZONS + h] from
s_t.frd_labels[h][0] (the per-batch label at this step's anchor)
* write_slice_i32_d_pub into trainer.frd_labels_d BEFORE
step_with_lobsim → bwd chain reads real labels in step_synthetic
Tests (3 new in loader::frd_label_tests, total 3/3 passing):
* frd_labels_flat_price_maps_to_mid_bucket — constant mid → all
non-sentinel labels = 10 (FRD_N_ATOMS/2 rounded); sentinel range
[n-h, n) tested exhaustively
* frd_labels_monotonic_ramp_lands_in_upper_buckets — linear ramp
mid[i] = 100 + 0.01×i produces forward returns way above 3σ at
every horizon → clamp to top bucket (FRD_N_ATOMS-1=20)
* frd_labels_short_input_below_h_ticks_all_sentinel — n=10 < h_ticks
for all 3 horizons → every label is -1 (no leak in the sentinel path)
Existing tests still pass:
* loss_balance lib tests 3/3
* frd_head GPU tests 10/10
* integrated_trainer_smoke 1/1
* trade_management_kernels 5/5
The full FRD head pipeline is now active end-to-end. Cluster smoke
will show FRD entropy_mean drift below ln(21) ≈ 3.044 once the bwd
gradient signal accumulates — the observable proof that supervised
learning is happening. The "frd" diag block from F.2 was always
prepared for this; F.5 just feeds it real signal.
F.6+ scope (deferred, separate sessions):
* P9 FRD gate — override action to Hold when entry_quality < THR
* Loss-balance controller integration for λ_frd (currently 1.0 default)
* Per-horizon Sharpe attribution in diag
Wires the F.3a/b/c backward kernels into IntegratedTrainer's per-step
flow so the FRD head trains end-to-end as a 6th loss-balanced head
alongside BCE/Q/π/V/aux. With labels currently sentinel-initialized to
-1 (F.5 loader will populate from forward-snapshot lookahead), the
chain produces zero gradients + zero loss — Adam steps are no-ops
modulo β decay, and the encoder receives no FRD-derived signal yet.
The wiring is complete and the path is exercised end-to-end; F.5 just
needs to swap the labels in for the head to start training.
IntegratedTrainer state additions:
* frd_w1_adam / frd_b1_adam / frd_w2_adam / frd_b2_adam — AdamW
instances for the 4 FRD weight tensors (LR mirrored per-step from
ISV[RL_FRD_LR_INDEX=499], seed 1e-3 per F.1).
* frd_labels_d — owned [B × FRD_N_HORIZONS] i32 buffer, sentinel-
initialized to -1 (every entry "missing horizon" → softmax_ce_grad
zeros loss + grad for every row). F.5 loader integration overwrites
pre-step from forward-return-bucketed labels.
LossLambdas extension:
* Added `frd: f32` field, default 1.0
* read_loss_lambdas_from_isv reads slot 498 (RL_FRD_LAMBDA_INDEX)
with the standard zero-sentinel bootstrap path
* Doc-comment updated: "5 heads / 5.0" → "6 heads / 6.0"
IntegratedStepStats extension:
* Added `l_frd: f32` — mean CE across (B × FRD_N_HORIZONS) rows
* step_synthetic returns the real l_frd from the bwd chain; the
new combined l_total formula includes `lambdas.frd × l_frd / 6`
step_synthetic bwd chain — inserted between Step 9 (Q/π/V Adam) and
Step 10 (grad_h_t_combined zero+accumulate):
1. softmax_ce_grad → frd_grad_logits_d + frd_loss_per_b_h_d
2. layer2_bwd → frd_grad_w2_pb_d, frd_grad_b2_pb_d, frd_grad_hidden_d
3. layer1_bwd → frd_grad_w1_pb_d, frd_grad_b1_pb_d, frd_grad_h_t_d
4. 4× reduce_axis0 to collapse per-batch scratch → final grads
5. 4× AdamW.step on w1/b1/w2/b2
6. read loss_per_b_h via mapped-pinned, average → l_frd_host
Step 10 grad_h_t_combined accumulation adds a third λ-weighted call:
accumulate_grad_h(frd_grad_h_t_d, lambdas.frd, &mut combined)
With sentinel labels (F.4 state) this contributes zero gradient to the
encoder backward — the wiring is exercised but silent. F.5 makes it
active by providing real labels.
alpha_rl_train diag JSON gains:
* "loss": { ..., "frd": stats.l_frd, ... }
* "lambdas": { ..., "frd": stats.lambdas.frd, ... }
Verification (RTX 3050 Ti):
* cargo check -p ml-alpha + --examples → clean
* integrated_trainer_step_with_lobsim_runs_without_panic → ok
(l_total 0.5073 vs prior 0.6087 — ÷6 instead of ÷5 expected;
l_frd=0 confirms sentinel labels are passing through cleanly)
* frd_head 10/10 tests still pass (no regression)
* trade_management_kernels 5/5 → no regression
* audit-rust-consts → 0 flags
F.5 (next, separate scope):
* Loader-side forward-return label generation (mid[i+h] - mid[i])/σ
bucketed into FRD_N_ATOMS=21 atoms over the ISV-driven ±range_σ
* Populate trainer.frd_labels_d before each step_with_lobsim call
* That unlocks the supervised learning signal; FRD entropy_mean
should start dropping below ln(21) in diag as the head trains.
Third and final FRD backward stage. Closes the chain from
softmax+CE loss back to the encoder's hidden state h_t.
Kernel `cuda/rl_frd_layer1_bwd.cu`:
* grid_dim = (B, 1, 1), block_dim = (HIDDEN_DIM=128, 1, 1)
* Phase 0: threads 0..63 stage dL/dpre_hidden = grad_hidden ×
1{hidden > 0} into shared mem (the cached post-ReLU `hidden`
buffer encodes the mask — hidden == 0 ⇔ pre-activation was
≤ 0 → ReLU killed it). Same thread also writes db1_per_batch.
* Phase 1: each thread k (k < 128) writes one row of
grad_W1_per_batch[b, k, 0..64] (64 writes per thread, no atomics)
* Phase 2: same thread computes grad_h_t[b, k] =
Σ_i W1[k, i] × dL/dpre_hidden[b, i]
* Per-(b, k, i) sole-writer per feedback_no_atomicadd
Rust wiring `FrdHead::layer1_bwd` — takes h_t, hidden (forward cache),
grad_hidden (from layer2_bwd), self.w1_d; writes grad_w1_per_batch,
grad_b1_per_batch, grad_h_t. The grad_h_t buffer becomes the encoder-
upstream gradient that the trainer's grad_h_accumulate kernel folds
into the encoder's gradient with λ_frd scaling (same pattern as Q/π/V
heads — wiring lives in F.4).
Tests (2 new, 10/10 file total):
* frd_layer1_bwd_finite_diff_w1 — perturbs the W1 slot with MAX
|analytical gradient| (instead of an arbitrary fixed slot — fp32
finite-diff is rounding-error-limited so a tiny gradient gives
misleading rel_err). At max-magnitude slot (k=84, i=55): analytical
= -0.0451, numerical = -0.0448, rel_err = 5.6e-3 — well within
1e-2 tolerance (slightly looser than dW2's 5e-3 because dW1
crosses an extra matmul + the ReLU mask boundary).
* frd_layer1_bwd_relu_mask_zeros_grad — fixture with h_t = all -1
produces ~half the hidden slots ReLU-masked (cached hidden = 0).
For every masked slot i, asserts:
* db1_per_batch[b, i] == 0 (exact equality — mask is hard 0)
* dW1_per_batch[b, k, i] == 0 for every k (~32 × 128 = 4096
slots checked)
Empirically 32/64 masked, 32/64 active — confirms ReLU mask
is wired through the chain correctly without leaking gradient
through dead branches.
F.3 backward chain is now complete end-to-end:
rl_frd_softmax_ce_grad (F.3a) → rl_frd_layer2_bwd (F.3b) →
rl_frd_layer1_bwd (F.3c) → grad_h_t (consumed by F.4 wiring)
F.4 wires Adam optimizers for W1/b1/W2/b2 + grad_h_accumulate into
the encoder gradient + loader-side label generation + λ_frd × CE
into stats.l_total.
Second of three FRD backward stages. Given dL/dlogits from F.3a's
softmax_ce_grad and the cached hidden activation from F.2's forward,
computes the layer-2 weight gradients via the standard chain rule
and emits the upstream gradient for layer-1 backward (F.3c).
Kernel `cuda/rl_frd_layer2_bwd.cu`:
* grid_dim = (B, 1, 1), block_dim = (FRD_HIDDEN_DIM=64, 1, 1)
* Phase 0: stage 63-slot grad_logits into shared (thread 63 idle)
* Phase 1: each thread i (i < 64) computes one row of per-batch
dW2 scratch: grad_w2_per_batch[b, i, 0..63] = h_bi × grad_logits[0..63]
(63 writes per thread, no atomics)
* Phase 2: each thread i computes dL/dhidden[b, i] = Σ_j W2[i, j] × grad_logits[j]
* Phase 3: thread i (i < 63) writes grad_b2_per_batch[b, i] = grad_logits[b, i]
* Per-batch scratch shape [B, FRD_HIDDEN_DIM, FRD_OUT_DIM] reduces
across batch via existing reduce_axis0 infra (caller's job, same
pattern as v_head_bwd / aux_heads_bwd)
Rust wiring `FrdHead::layer2_bwd`:
* Takes hidden (forward cache), grad_logits (from softmax_ce_grad),
self.w2_d
* Writes grad_w2_per_batch, grad_b2_per_batch, grad_hidden — all
sized to caller-allocated buffers
* Sole &self method (Adam step is the caller's responsibility)
Tests (2 new, 8/8 file total):
* frd_layer2_bwd_finite_diff_w2 — perturb W2[10, 5] by ±ε=1e-3,
compare (L(+) - L(-))/(2ε) to per-batch grad scratch. rel_err
= 6.27e-5 (better than F.3a's softmax-CE finite-diff because
the gradient magnitude here is larger so rounding error is
relatively smaller). Helper `ce_total_loss` re-uses
`softmax_ce_grad` to compute total CE for the perturbed forward
pass — pure GPU-oracle, no CPU softmax/CE reference impl.
* frd_layer2_bwd_db2_equals_grad_logits — analytical invariant:
db2_per_batch[b, j] must equal grad_logits[b, j] exactly (the
bias gradient is the identity passthrough at this layer). Cheap
structural check that catches dimension-shuffle bugs in the
kernel before they corrupt the reduce_axis0 step.
The kernel restores W2 to its original values after the perturbation
to keep test isolation clean — `&mut head` access pattern (proper
Rust borrowing, no UB const→mut casts).
F.3c (layer-1 backward: dW1, db1, dh_t with ReLU mask via the
cached hidden activation) is next.
Per-(batch, horizon) softmax + cross-entropy loss + gradient w.r.t.
the 21 atom logits. First of three backward stages — F.3b adds layer-2
weight grads (dW2, db2, dhidden), F.3c adds layer-1 weight grads
(dW1, db1, dh_t with ReLU mask).
Kernel `cuda/rl_frd_softmax_ce_grad.cu`:
* grid_dim = (B, FRD_N_HORIZONS, 1), block_dim = (FRD_N_ATOMS=21, 1, 1)
— one block per (batch, horizon) pair, threads cooperate over the
21 atoms via shared mem
* Standard numerically-stable softmax: shift by row_max, exponentiate,
normalize by row_sum (thread 0 does the serial reductions — 21
atoms is small enough warp-shuffle overhead isn't worth it)
* Gradient: (p[a] - 1{a==label}) / B at the source per v_head_bwd
convention (mean-reduce over batch)
* Loss: -log(p[label]) with 1e-30 floor against log(0)
* Sentinel label (-1) zeros both gradient row and loss — for the
missing-horizon case at the rightmost edge of the snapshot stream
(forward returns at h=300 ticks aren't realized for the last
300 snapshots; loader marks those labels with -1)
* Per feedback_no_atomicadd: per-(b, h, a) sole-writer pattern
Rust wiring `src/rl/frd.rs::FrdHead::softmax_ce_grad`:
* Second cubin loaded alongside fwd (separate module per the
aux_heads pattern; small handle, no impact on init time)
* Caller provides labels_d [B, FRD_N_HORIZONS] of i32 and gets back
grad_logits + per-(b, h) raw CE; sum + λ_frd scaling left to the
caller (F.4 will hook this into stats.l_total + Adam step)
Tests `tests/frd_head.rs` — 3 new GPU-oracle tests (6/6 file total),
all PASS on RTX 3050 Ti:
1. frd_softmax_ce_grad_uniform_logits_match_log_n_atoms — for any
label, uniform logits → CE = ln(FRD_N_ATOMS) = ln(21) ≈ 3.0445.
Also asserts per-row Σ grad_logits = 0 (softmax-CE invariant).
2. frd_softmax_ce_grad_sentinel_label_zeros_row — label=-1 with
non-trivial random logits produces exactly zero loss + grad
for every row (no leak through the sentinel path).
3. frd_softmax_ce_grad_finite_diff_matches_analytical — perturbs
one logit slot by ±ε=1e-3, compares (L(+ε) - L(-ε))/(2ε) to
the kernel's analytical gradient. rel_err ≈ 1.3e-3 (fp32
finite-diff is rounding-error-limited at this ε; tolerance
set to 5e-3 with explanatory comment).
The first two tests provide strong analytical oracles (no CPU
reference impl per feedback_no_cpu_test_fallbacks). The finite-diff
test cross-validates the full softmax+CE chain via a numerical
gradient — the standard ground-truth for autodiff kernels.
IntegratedTrainer now owns an FrdHead instance and per-step buffers
(frd_hidden_d [B × FRD_HIDDEN_DIM=64], frd_logits_d [B × FRD_OUT_DIM=63]).
The forward kernel runs in step_with_lobsim immediately after the
current-snapshot encoder forward, reading h_t_borrow and producing the
3-horizon × 21-atom return-bucket logits.
step_with_lobsim FRD forward placement rationale: it has to read
self.perception.h_t_view() AFTER the second forward_encoder(snapshots)
call (which lands h_t at slot K-1), but BEFORE any downstream
consumer of the encoder state — so right between Step 1b and Step 2.
This keeps the FRD output aligned with the same h_t that the Q / π /
V heads see for action sampling.
alpha_rl_train diag emits a new "frd" block per step:
"frd": { "h1": {"entropy_mean", "argmax_mean"}, "h2": ..., "h3": ... }
At init (Xavier × 0.1, b1=b2=0) the per-horizon softmax is near-
uniform → entropy_mean ≈ ln(21) = 3.044 and argmax_mean drifts around
the uniform expectation of 10. As supervised training kicks in (F.3),
entropy drops and argmax tracks the realized forward-return mode per
horizon — this is the observable signal that lets us catch a broken
backward kernel before cluster smoke.
Verification:
* cargo check -p ml-alpha --examples → clean
* integrated_trainer_step_with_lobsim_runs_without_panic → ok
(1.66s, b_size=1, full step path through encoder + FRD + Q/π/V)
* audit-rust-consts → 0 flags
* trade_management_kernels (5/5) + frd_head (3/3) → still pass
F.3 (backward kernel + finite-diff tests + label generation in loader
+ λ_frd-weighted loss accumulation into stats.l_total) is the next
chunk. FRD-gate (P9) and FRD label-cache wiring are separate scope.
Forward-Return-Distribution head per SP20 §3 P3. Supervised forecaster
over 3 horizons × 21 return-bucket atoms — replaces the survivor-biased
checklist head per CRIT-1.
Architecture (2-layer MLP):
hidden [B, 64] = ReLU(h_t [B, 128] @ W1 [128, 64] + b1)
logits [B, 63] = hidden @ W2 [64, 63] + b2 // 63 = 3 × 21
Softmax + CE happen in the backward kernel (F.3). The forward kernel
caches the post-ReLU hidden buffer to avoid recomputing the W1 product
+ ReLU mask on backward.
Kernel `cuda/rl_frd_fwd.cu` — 1 block per batch, 64 threads:
* Phase 1 (tid < 64): each thread computes one hidden activation,
stages into shared mem, writes the cached `hidden_out[b, tid]`
* Phase 2 (tid < 63): each thread computes one output logit by
reading the shared hidden vector
* No atomicAdd (per-batch, per-output sole-writer pattern)
* No host branches in the launch (graph-capture safe)
Rust head module `src/rl/frd.rs`:
* `FrdHead::new(dev, cfg)` — Xavier × 0.1 init for W1/W2 (small enough
to keep initial softmax near-uniform), zero biases. Scoped-init-seed
guard per pearl_scoped_init_seed_for_reproducibility.
* `forward(h_t_d, hidden_out_d, logits_out_d, b_size)` — single
kernel launch via the cached `fwd_fn` handle.
* Public weight buffers (w1_d, b1_d, w2_d, b2_d) for the upcoming
bwd kernel + test harnesses.
* `pub const FRD_OUT_DIM = FRD_N_HORIZONS × FRD_N_ATOMS = 63` — single
canonical reference for the per-batch output width.
Tests `tests/frd_head.rs` — 3 GPU-oracle tests, all PASS on RTX 3050 Ti:
1. frd_forward_zero_input_emits_zero_logits — h_t=0 with default
b1=b2=0 must produce exactly zero logits AND zero cached hidden.
Unambiguous analytical oracle for the full matmul + ReLU + matmul
chain.
2. frd_forward_shape_matches_spec — random h_t produces correctly
shaped output [B × 63] with per-horizon softmax sums = 1.0
within 1e-5 (numerical-stable log-sum-exp).
3. frd_forward_relu_mask_consistent_with_cached_hidden — strictly
negative h_t input → ≥50% of cached hidden slots must be exactly
zero (ReLU fires). Empirically 128/256 zeros on the seeded init.
Per feedback_isv_for_adaptive_bounds: bucket-range σ stays in ISV
(slot 503, seeded ±3σ); only the 21-atom count is structural
compile-time per SP20 §0.1.
Foundation patch for the Forward-Return-Distribution head (SP20 P3).
No new behavior — kernels arrive in the next commit (F.2). This commit
just establishes the ISV vocabulary and structural dims so the kernel
code can reference named slots/consts from day one.
ISV slots 498-503 (RL_SLOTS_END bumped 498 → 504):
* RL_FRD_LAMBDA_INDEX = 498 seed 0.5
* RL_FRD_LR_INDEX = 499 seed 1e-3
* RL_FRD_HORIZON_1_TICKS_INDEX = 500 seed 60.0
* RL_FRD_HORIZON_2_TICKS_INDEX = 501 seed 300.0
* RL_FRD_HORIZON_3_TICKS_INDEX = 502 seed 1800.0
* RL_FRD_BUCKET_RANGE_SIGMA_INDEX = 503 seed 3.0 (±3σ)
Bootstraps written via the existing isv_constants table in
IntegratedTrainer::new — same path as the SP20 P5 trail bounds. No
HtoD path opened (rl_isv_write does device-side scalar writes).
Structural consts (crates/ml-alpha/src/rl/common.rs):
* FRD_HIDDEN_DIM = 64 (MLP hidden layer width)
* FRD_N_HORIZONS = 3 (h1/h2/h3 forward returns)
* FRD_N_ATOMS = 21 (return-bucket atoms per horizon)
Atom count is the only structural compile-time dim per §0.1 of the
SP20 spec; range_σ is ISV-driven (slot 503) so the head can adapt
as realised σ drifts.
Five #[ignore] CUDA-required tests for the trader-grade trade-management
kernels (rl_unit_state_update, actions_to_market_targets HalfFlat
branches, rl_trail_mutate, rl_trail_stop_check). Analytical invariant
oracles per feedback_no_cpu_test_fallbacks.
Public IntegratedTrainer launch wrappers (mirror internal kernel
invocations in step_with_lobsim with controllable buffers):
* launch_actions_to_market_targets
* launch_rl_trail_mutate
* launch_rl_trail_stop_check
* launch_rl_unit_state_update
Public mapped-pinned write/read helpers added to satisfy
feedback_no_htod_htoh_only_mapped_pinned (the pre-commit hook rejects
raw un-pinned host-side transfers with no grandfathering for new code):
* write_slice_f32_d_pub / write_slice_i32_d_pub / write_slice_u8_d_pub
* read_slice_u8_d_pub (counterpart to existing _d_pub readers)
write_slice_u8_d_pub / read_slice_u8_d_pub stage via MappedI32Buffer
(4-byte alignment) — covers byte-buffer fixtures like the 24-byte
PosFlat layout used by the unit-state and half-flat tests without
needing a new MappedU8Buffer type.
Test catalogue (all passing locally on RTX 3050 Ti):
1. half_flat_long_emits_half_position_size — a9 sizing + a9-on-short
no-op + odd-lot ceil(3/2)=2 invariant
2. half_flat_short_emits_half_position_size — symmetric a10 case
3. unit_state_transitions — sentinel-zero bootstrap OPEN (was-flat→
long) + CLOSE (long→flat) + prev_pos_lots tracker advance +
only-slot-0-active invariant (slots 1-3 stay 0)
4. trail_mutate_tighten_loosen_reciprocal — a7 then a8 returns trail
to original within 1e-5 + inactive units don't mutate + non-trail
action (Hold) passes through
5. trail_stop_check_overrides_action_on_breach — long breach overrides
Hold→FlatFromLong (a3) + no-breach leaves Hold + short breach
overrides Hold→FlatFromShort (a4) (symmetry)
Bug caught during test authoring: IntegratedTrainer::new allocates
isv_d as all-zeros; ISV bootstrap defaults are written only during
the first step_with_lobsim, not at construction. Tests must explicitly
seed every ISV slot their kernel reads — in particular RL_TRAIL_MAX
for the loosen branch (fminf(0, x) silently zeroes trail_distance).
Per feedback_no_sp_or_version_prefixes_in_file_names: file named by
WHAT it tests (trade_management_kernels), not WHICH spec phase
introduced it. Same for the #[test] fn identifiers.
Action enum extended:
a9 = HalfFlatLong (close ⌈|pos|/2⌉ of long position, no-op if not long)
a10 = HalfFlatShort (close ⌈|pos|/2⌉ of short position, no-op if not short)
`actions_to_market_targets.cu` extended with a9/a10 handlers:
HalfFlatLong (pos > 0): side=1 sell, size=max(1, (position_lots+1)/2)
HalfFlatShort (pos < 0): side=0 buy, size=max(1, (|position_lots|+1)/2)
Round-up division ensures min 1 lot closes — single-lot positions
fully close on HalfFlat (the half rounds up to 1).
N_ACTIONS=9 → 11 propagated to all 10 .cu kernels:
argmax_expected_q, bellman_target_projection, dqn_distributional_q,
log_pi_at_action, ppo_clipped_surrogate, rl_action_kernel,
rl_entropy_coef_controller, rl_pi_action_kernel, rl_q_pi_agree_b,
rl_q_pi_distill_grad
Rust-side N_ACTIONS const bumped to 11 in src/rl/common.rs.
CLI alpha_rl_train.rs action_hist + windowed_act_hist refactored
to reference `N_ACTIONS` const instead of literal 11. Caught DURING
this commit's dogfood: an intermediate state had `[0u32; 11]` but
left `(0..9).contains(&a)` unchanged — HalfFlat samples silently
dropped (a9/a10 showed 0% in diag despite P_MIN=0.02 floor
guaranteeing 2% each). Fix uses N_ACTIONS const everywhere; new
pearl `feedback_use_consts_not_literals_for_structural_dims`
codifies the meta-pattern (Rust code mirroring kernel structural
dims must reference the const, NEVER duplicate the literal —
audit-isv only scans .cu files, this class of bug is currently
unaudited in .rs).
Audits PASS:
audit-isv: all kernel #defines allowlisted (BOOK_LEVELS,
ACTION_*, structural dims)
audit-wiring: all 4 actions in manifest (TrailTighten, TrailLoosen,
HalfFlatLong, HalfFlatShort) have consumers
Local 1k-step smoke (RTX 3050 Ti, 13.5s):
* Exit 0, 0 NaN/inf, 16000/16000 samples accounted for
* Action distribution: all 11 used in 7-11% range
* HalfFL=7.99%, HalfFS=7.51% — π samples them under multinomial
* Trail=19.34% — agent continues to value trail-stop actions
Trail-stop check (rl_trail_stop_check.cu) currently still routes
force-close through a3/a4 (FlatFromLong/Short) rather than per-unit
partial-flat via a9/a10. That routing upgrade is SP20 P5b follow-up
work — it requires the per-unit close_unit_index buffer wiring per
spec §3 P5. Adding a9/a10 to the action space is the foundational
prerequisite; consumer kernel uses come with P5b.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
scripts/audit-wiring.sh dogfood pass flagged a7 (TrailTighten) and
a8 (TrailLoosen) as actions with no consumer anywhere in the
codebase (canonical pearl_dead_trail_stop_actions_a7_a8). Fix
bundles SP20 P1 (per-unit trade state buffers) and P5 (trail-stop
kernels) since they're the same architectural work.
Three new kernels:
rl_unit_state_update.cu — per-batch trade state machine. Runs
AFTER fill+extract_realized_pnl_delta.
Detects open/close/reverse position
transitions and populates unit slot 0
with entry_price, entry_step, lots,
initial_r, trail_distance. Slots 1-3
allocated for SP20 P7 pyramid expansion
but unused this commit.
rl_trail_mutate.cu — handles a7/a8 actions. Mutates ALL
active units' trail_distance bounded
by ISV [MIN, MAX] with symmetric
reciprocal adjust rate per SP20 §4.12:
a7: trail = max(MIN, trail × rate)
a8: trail = min(MAX, trail / rate)
rl_trail_stop_check.cu — per-batch per-unit breach check. Reads
shared lobsim best book (bid/ask),
computes mid, compares to each active
unit's (entry ± trail). On breach,
OVERRIDE actions[b] to FlatFromLong
(a3) or FlatFromShort (a4). Force-close
routes through existing flat plumbing
per pearl_stop_checks_run_at_deadline_cadence.
SP20 v3 §3 P5 calls for routing close
via partial-flat (a9/a10) so only the
at-risk unit closes — that needs P4
(N_ACTIONS=11). For now, ANY unit's
breach closes ENTIRE position via full
FlatFromLong/Short.
Per-batch per-unit buffers (8 new in trainer):
unit_entry_price_d [B × 4] f32
unit_entry_step_d [B × 4] i32
unit_lots_d [B × 4] i32
unit_initial_r_d [B × 4] f32
unit_trail_distance_d[B × 4] f32
unit_active_d [B × 4] u8
pyramid_units_count_d[B] i32
unit_prev_pos_lots_d [B] i32 (state-machine tracker, separate
from extract_realized_pnl_delta's
prev_position_lots_d for clean
kernel composability)
4 new ISV slots (494-497):
RL_TRAIL_MIN_INDEX — trail distance floor (seed 0.001)
RL_TRAIL_MAX_INDEX — trail distance ceiling (seed 100.0)
RL_TRAIL_K_INIT_INDEX — initial trail multiplier (seed 2.0, Turtle 2N)
RL_TRAIL_ADJUST_RATE_INDEX — tighten ratio (seed 0.9; symmetric reciprocal for loosen)
RL_SLOTS_END: 494 → 498.
LobSim exposes bid_px_d() + ask_px_d() public accessors. RlLobBackend
trait extended with the two accessors; the LobSimCuda impl wires
through.
Override stack ordering per SP20 §2.3:
1. rl_pi_action_kernel (sample)
2. rl_trail_mutate (a7/a8 → mutate, before stop check)
3. rl_trail_stop_check (per-unit breach → override action)
4. actions_to_market_targets (execute, including overridden flat)
5. step_fill_from_market_targets
6. extract_realized_pnl_delta
7. rl_unit_state_update (detect post-fill transitions)
Audit infrastructure refined as part of dogfooding:
* audit-isv allowlist extended for BOOK_LEVELS (structural book
depth) and ACTION_* prefix (enum-mirror constants — these are
structural API contracts matching src/rl/common.rs::Action positions)
* audit-wiring action-handler regex now matches BOTH literal
`action == <idx>` and `action == ACTION_<UPPER_SNAKE>` patterns,
and treats != as a handler too (a guard against the action is
valid wiring)
Both `audit-isv.sh` and `audit-wiring.sh` PASS cleanly with the
full manifest. audit-diag scheduled for first SP20 phase that adds
diag fields (this commit deliberately keeps diag exposure minimal
— full per-unit + trail diag blocks come with SP20 P13).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
scripts/audit-isv.sh dogfood pass flagged `#define KL_EMA_ALPHA 0.05f`
in rl_q_pi_distill_grad.cu — a hardcoded numerical constant that
escaped the formal critical review of SP20 v3 + earlier review cycles.
Fix per SP20 §0.1 "every numerical constant ISV-resident":
* New slot RL_Q_DISTILL_KL_EMA_ALPHA_INDEX = 493
* Seeded to 0.05 (preserves prior behavior) in
with_controllers_bootstrapped's rl_isv_write list
* Kernel reads from `isv[RL_Q_DISTILL_KL_EMA_ALPHA_INDEX]` instead
of hardcoded `KL_EMA_ALPHA`
RL_SLOTS_END: 493 → 494.
Re-run of `scripts/audit-isv.sh` + `scripts/audit-wiring.sh` against
this kernel + slot manifest passes cleanly.
This is the first of two violations the audit dogfood caught.
The second — `TrailTighten` / `TrailLoosen` actions a7/a8 having
no handler anywhere — IS SP20 Phase P5 scope and gets its own
commit when P5 lands.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds SP20 — full production trader-management system in one
greenfield commit (3-4 weeks of implementation work to follow):
* Tier 0: multi-resolution time-scaled market features (3 horizons)
* Tier 1: trade-arc awareness (4 features per batch)
* Tier 2: per-unit trail-stop (entry + trail + stop per unit)
* Tier 3: pyramiding + partial profit-taking (HalfFlat actions,
N_ACTIONS=9→11)
* Tier 4: Forward-Return-Distribution head + confidence gate +
per-batch anti-martingale sizing + position heat cap +
vol-adjusted defaults
Spec went through critical-review pass (v1→v2→v3):
* v1: 3 tiers, side-channel features, single-gate acceptance
* v2: 5 tiers added partial-flat + anti-mart + multi-res + checklist
* v3: foundational fixes for 4 CRIT + 6 SIG + 6 MIN findings
(per-unit pyramid state, encoder-input injection vs side-channel,
FRD head replaces survivor-biased checklist, override stack
ordering, per-batch anti-mart, real-time multi-res scales,
P-1 ceiling falsification gate, multi-tier acceptance)
§0 Foundational Principles (NEW, non-negotiable):
* §0.1 every numerical constant ISV-resident (no hardcoded #defines
in new kernels; structural-dim exception only)
* §0.2 every kernel/slot/head/action fully wired in same commit
* §0.3 diagnostics baked in at birth (every observable in JSONL)
* §0.4 per-phase ship-gate: all three audits must pass
Audit infrastructure shipped with the spec:
* scripts/audit-isv.sh — greps new .cu for hardcoded #defines
* scripts/audit-wiring.sh — verifies kernels/slots/heads/actions
have producer + consumer in code
* scripts/audit-diag.sh — runs local 100-step smoke, validates
manifest-listed jq paths present in JSONL
* scripts/audit-manifest/ — per-phase append manifests (kernels,
slots, heads, actions, diag-fields)
Naming discipline: audit scripts and manifest are SP-agnostic (no
`sp20-` prefix) per new pearl `feedback_no_sp_or_version_prefixes_in_file_names`
— they'll serve future SPs too. SP numbers belong only in
docs/superpowers/{specs,plans}/ filenames.
Audit scripts dogfooded — already caught two real violations on
existing code that the formal review missed:
* audit-isv: KL_EMA_ALPHA=0.05f hardcoded in rl_q_pi_distill_grad.cu
* audit-wiring: TrailTighten action (a7) has no handler in any
kernel (per pearl_dead_trail_stop_actions_a7_a8)
These violations are SP20 P5/P10 fix targets.
User decision recorded in spec §3 P-1: ceiling-falsification phase
intentionally skipped — SP20 is the architectural launchpad for
the broader trader system regardless of whether current arch could
be pushed further at 1M steps. P-1 may be revisited as standalone
work after SP20 ships.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
