Prepared for next run after the γ=0.99 1M run completes:
- γ floor 0.99 → 0.995: horizon 100 → 200 steps (50 seconds).
Real ES directional moves (2-5 points) happen at this scale.
- PER 16384 → 32768: 2048 unique steps of replay depth. Supports
the 200-step γ horizon with margin.
- Min-hold 50 → 100 steps (25 seconds): commit to the full wave.
Short-hold penalty threshold matches.
Safe to push because raw-reward re-normalization eliminates scale
drift in the deeper buffer, and the ±2% scale clamp keeps targets
stable across the 2048-step replay window.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The agent was break-even at the wrong timescale — trading in
bid-ask noise at 1-4 second holds where there IS no edge. Real
directional moves live at 10-60 seconds.
Three changes push the agent to wave-scale:
1. γ floor 0.90 → 0.99: effective horizon 10 → 100 steps (25s).
Q now values what happens 25 seconds from now, not just the
next tick. The FRD's 10s and 600s horizons become relevant.
2. Min-hold 20 → 50 steps (12.5s): forces commitment to waves,
not ripples. Short-hold penalty threshold matches.
3. Long-ride bonus: at trade close, profitable rides get reward
multiplied by (1 + bonus × sqrt(hold_time)). A 100-step
winning ride gets 21× the raw PnL as reward. Combined with
per-step hold bonus boosted to $2.00. "The best wave of the
session deserves the biggest cheer."
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The reward_scale oscillated 10,873× (min 0.000123, max 1.33) because
the Wiener-α=0.4 blend tracked sparse trade PnL spikes instantly.
Old transitions in PER had stale-scale rewards even with raw-reward
re-normalization (the current scale itself was unstable).
Per-step clamp: scale can only move ±2% from previous value.
Doubling takes ~35 steps, halving takes ~35 steps — fast enough to
track regime changes, stable enough for Q targets across the replay
window. Max oscillation over 1000 steps: ~7× (was 10,873×).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Min-hold check was overriding heat cap's FlatFromLong/Short to Hold,
preventing the safety exit. Position grew to 9 (above cap 8) because
the flat was blocked by min-hold, then next step added lots.
Fix: min-hold reads pos_state and ISV heat cap slot. If |position|
>= cap, the close action passes through regardless of hold time.
Safety always overrides patience.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Overrides closing actions (a3/a4/a9/a10) to Hold when
steps_since_done < RL_MIN_HOLD_STEPS_INDEX (slot 536, default 20).
The agent CANNOT exit before the minimum — forced to ride the wave.
Trail stops still fire regardless (safety overrides patience) —
if the market moves against the position past the trail distance,
the stop-loss exits even within the min-hold window.
Pipeline order: action selection → confidence gate → FRD gate →
min_hold_check → trail_mutate → trail_stop → heat_cap →
actions_to_market_targets
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three ISV-driven reward shaping components discourage churning and
reward patience:
1. Entry cost ($15 default): subtracted when opening a new position.
Agent must expect profit > cost to justify entry. Slightly above
ES 1-tick spread ($12.50) so marginal trades are net-negative.
2. Short-hold penalty (0.5× for holds < 20 steps): multiplicative
penalty at trade close for quick flips. "Don't bail on the first
bump" — halves the reward for sub-5-second holds.
3. Hold bonus ($0.50/step × sqrt(hold_time)): per-step reward for
staying in a profitable position. "Ride the wave" — incentivizes
patience when the trade is working.
Runs BEFORE reward_scale so all costs/bonuses are in raw USD terms.
ISV slots 532-535. RL_SLOTS_END → 536.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Binary argmax agreement is pure noise for near-uniform distributions
(both Q and π give ~9% to each of 11 actions, so argmax flips on
0.001 differences → metric reads 0 even when KL=0.02).
Cosine similarity between E_Q[a] (expected Q-value per action) and
softmax(π)[a] measures full ranking direction agreement. Robust to
near-uniform: two identical distributions → cosine=1.0 regardless
of which action is marginally highest.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Root cause of q_pi_agree collapse: the symmetric Schulman controller
decayed λ from 0.5 to 0.001 in 34 steps (÷1.2/step), killing Q→π
coupling. Once decoupled, Q and π learned independent policies,
making q_pi_agree drop to 1e-22.
Three fixes:
1. MIN_LAMBDA 0.001 → 0.05: Q pull never drops below 5% strength
2. Asymmetric rates: ramp 1.2×/step (4 steps to double), decay
0.998×/step (347 steps to halve). Coupling establishes fast and
persists for thousands of steps.
3. Dead zone tolerance 1.5× → 3.0×: natural KL fluctuation stays
in-band instead of triggering constant ramp/decay cycles.
Seed λ raised to 0.1 so distillation is active from step 0.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Stored raw_reward alongside scaled reward in Transition. At sample
time, consumer re-applies current reward_scale to raw_reward instead
of using the stale-scale stored reward. This eliminates off-policy
scale drift that caused q_pi_agree to collapse from 0.125 to 1e-22
over 40k steps with PER=32768.
PER capacity set to 16384 (1024 unique steps at b=16) — balances
replay depth against h_t representation staleness.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The CLI default overrode the trainer config default. Without passing
--per-capacity explicitly, the Argo run used 4096 (256 unique steps
at b=16) instead of the intended 32768 (2048 steps).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Root cause of Q non-convergence: at reward_scale ≈ 0.003, a typical
±$5 trade maps to ±0.015 scaled reward. With 21 atoms spanning [-1,+1]
(step=0.1), the entire reward distribution falls within a SINGLE atom.
Q cannot distinguish wins from losses — confirmed by q_pi_agree ≈ 0.
Fix 1: Tighten atom span to [-0.5, +0.5]. Now atom_step = 0.05.
A ±$5 trade at scale=0.01 spans 2 atoms; at scale=0.05 spans 10.
The reward clamp controller ratchets the span at runtime if rewards
exceed the bounds.
Fix 2: PER capacity 4096 → 32768. At b=16, old capacity retained
only 256 unique steps — Q replayed each transition 1-2× before
eviction. New capacity retains 2048 steps, giving Q proper replay
depth for convergence.
Also aligns reward clamp bounds (WIN/LOSS) with the new span.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
With strict >, pos=8 passed the cap check, then a LongLarge(+2)
made pos=10 before the next step's cap fired. Using >= prevents
position from ever exceeding the cap by the fill size.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Controller oscillated: 3 dones at step 15k spiked EMA above 2×target,
triggering thousands of steps of tightening that killed trades.
Higher target (10% vs 2%) matches the natural trade frequency at
b=16. Wider dead zone (5× above / 0.2× below) prevents single-batch
spikes from triggering tighten/relax oscillation.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Confidence gate LCB fix worked (0 fires post-warmup) but FRD gate
blocked 5212 entries in 5k steps — the FRD head learned "forward
returns are negative" during warmup (losing trades), so favorable
tail mass dropped below 0.15 for all actions.
Lower default to 0.05 and floor to 0.0 so the adaptive controller
can fully disable when trades dry up.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The old formula conf = clamp((μ - λσ) / σ_norm, 0, 1) produced
conf=0 for ANY distribution with σ > μ — including uniform Q at
training start (μ=0, σ=0.577 → LCB=-0.577 → conf=0). This caused
permanent 100% block rate regardless of threshold setting.
New formula: conf = clamp((μ - V_MIN - λσ) / (V_MAX - V_MIN), 0, 1)
Shifts by V_MIN so conf measures "how far above worst case":
Uniform: conf = 0.21 (passes threshold 0.01)
Peaked V_MAX: conf ≈ 1.0 (high confidence)
Peaked V_MIN: conf ≈ 0.0 (blocked — correct)
Also sets conf gate floor to 0.0 so the adaptive controller can
fully disable the gate when trades dry up.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
When C51 LCB is ≈0 for all actions (typical early training), even
0.001 threshold blocks everything. Floor=0.0 lets the controller
drive to zero when trades dry up, then ratchet back as Q calibrates.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New rl_gate_threshold_controller.cu watches dones EMA and adjusts
confidence + FRD gate thresholds via Schulman bounded step:
- dones_ema < target×0.5 → relax thresholds (×0.95)
- dones_ema > target×2.0 → tighten thresholds (÷0.95)
ISV-driven: target (0.02), conf bounds (0.001-0.50), FRD bounds
(0.05-0.50), adjust rate (0.95). Runs per step after warmup.
Also lowers default thresholds: conf 0.10→0.01, FRD 0.35→0.15.
Post-warmup, the controller adapts these based on actual trade
flow instead of relying on static defaults.
ISV slots 525-531. RL_SLOTS_END → 532.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Both gates now read RL_GATE_WARMUP_STEPS_INDEX (slot 524, default
10000) and return early when current_step < warmup. During warmup,
the agent opens positions freely, collects reward signal, and
calibrates Q. After warmup, gates activate and filter low-quality
entries.
Without warmup, gates blocked 100% of opening actions from step 0
(uniform Q → zero confidence → permanent Hold attractor → zero
trades → zero reward → Q never learns). Confirmed by 50k-step
L40S run with zero dones across 800k action decisions.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replaces the 4-step DAG (check-cache → compile on CPU → warmup GPU →
train on GPU) with a single pod on the L40S that does everything:
git fetch → incremental cargo build (~3s warm) → train.
Eliminates: separate compile node, node autoscale wait, binary
transfer, fxcache step. The ci-builder image has CUDA 13.0 devel +
Rust — compilation and training use the same CUDA libs.
The cargo-target-cuda PVC provides incremental build cache across
runs. LD_LIBRARY_PATH strips the stubs dir so real CUDA libs load.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The MBP-10 predecoded sidecars live on feature-cache-pvc at
/feature-cache/predecoded/, not on training-data-pvc. Points
--predecoded-dir to the correct PVC mount.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
alpha_rl_train reads MBP-10 data with its own predecoded cache —
it never uses the fxcache pipeline. Skip the 15-minute feature
extraction step entirely for model=alpha-rl by depending only on
ensure-binary + gpu-warmup.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The ml-alpha branch uses alpha_rl_train (from ml-alpha crate) with
different CLI args than train_baseline_rl (from ml crate). Adds
model=alpha-rl case that:
- Builds ml-alpha --example alpha_rl_train
- Runs with --mbp10-data-dir, --predecoded-dir, --n-steps 50000,
--n-backtests 16 (matching the local smoke config)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
28 parallel alpha-feature chunks × ~3GB each = 84GB peak at merge.
Capping at 8 threads keeps peak at ~24GB (well within 96Gi limit)
while still saturating I/O. This is a one-time cost — after the
cache is populated on the PVC, subsequent runs skip entirely.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
With cargo-target-cuda PVC persisting across pods, CARGO_INCREMENTAL=1
gives true incremental builds: only recompile changed crates. A
single-crate ml-alpha change rebuilds in ~45s vs 9min full workspace.
sccache removed — it conflicts with incremental compilation and is
redundant when the target dir persists. The binary-by-SHA cache on
the training-data PVC (already implemented) provides the "skip
compile entirely on re-run" fast path.
Expected workflow time reduction:
Before: compile 9m + fxcache 18m + train ~30m = ~57m
After: compile <1m (incremental) + fxcache <1m (PVC cache) + train ~30m = ~32m
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The imbalance bar cache now writes to <mbp10_dir>/.cache/ on the PVC.
The ensure-fxcache step needs write access to persist the cache
across runs (was readOnly: true, causing silent cache-write failures).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Cache path moved from /tmp/.foxhunt_imbalance_cache/ (ephemeral,
lost between Argo pods) to <mbp10_dir>/.cache/ (on the training-data
PVC). Saves ~4 minutes per Argo submission by avoiding recomputation
of 209M trade ticks → 17.8M imbalance bars.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Completes P2 by injecting rl_encoder_context_broadcast launch inside
the perception trainer's forward_only path — runs after
snap_feature_assemble fills dims [0..40] and before VSN/Mamba2
consume the window tensor. Fills dims [40..56] with per-batch
trade_context (4) + multires (12) features.
Device pointers to the context buffers (owned by IntegratedTrainer)
are set on PerceptionTrainer before each forward_encoder call. The
broadcast kernel reads these and writes directly into window_tensor_d
at the correct column offsets for all B×K rows.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Introduces ENCODER_INPUT_DIM = 56 (FEATURE_DIM + 16). All encoder
first-layer weight matrices (VSN gate, Mamba2 L1 input projection)
now sized for 56 input dims. The extra 16 are per-batch state:
4 trade_context + 12 multires features.
- snap_feature_assemble_batched: output stride → ENCODER_INPUT_DIM,
zero-fills dims [40..56] for the broadcast kernel to overwrite.
- New rl_encoder_context_broadcast.cu: writes trade_context_d[B×4]
+ multires_output_d[B×12] into each of the K sequence rows per
batch at positions [40..56].
- CfcConfig.n_in, Mamba2 L1 in_dim, VSN gate, window_tensor_d,
all forward/backward scratch buffers updated to ENCODER_INPUT_DIM.
- CfcTrunk default config updated.
The broadcast kernel launch integration into the forward_only path
is the final wire-up step — until then dims 40-55 are zero-filled
(safe: Xavier init on new columns means encoder starts by learning
to ignore them, then gradually incorporates the signal).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
P1: New rl_trade_context_update.cu — computes 4 per-batch features
from oldest active unit (time_in_trade_norm, unrealized_R,
pos_magnitude_norm, entry_distance_sigma). Output in
trade_context_d[B×4], updated after unit_state_update each step.
P0: New rl_multires_features_update.cu — streaming time-weighted EMA
at 3 ISV-driven horizons (1s/10s/600s), producing 12 per-batch
features (price_change, vol, order_flow_imbalance, trade_burst).
O(1) state per feature vs circular buffer — same time-constant
semantics.
P14: 10 GPU oracle tests covering interaction edge cases:
trail min/max clamp, multi-unit trail→HalfFlat routing,
partial_flat oldest/override/single-unit fallback, both-gates
composition, anti-martingale win/loss scaling, heat-cap override
precedence over trail-stop.
ISV slots: 521-523 (multires horizons). RL_SLOTS_END → 524.
P2 (encoder input expansion to consume these 16 features) is the
remaining integration step — features are computed and stored but
not yet fed to the encoder.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
P10: New rl_recent_outcome_update.cu — per-batch signed outcome EMA
(sign(reward) on done steps) feeds per-batch anti-martingale
sizing in actions_to_market_targets. Replaces the single ISV
scalar with a per-batch buffer for multi-batch granularity.
P11: Trail bootstrap switched from vwap × 1e-3 × k_init to
k_init × MEAN_ABS_PNL_EMA (slot 423). Vol-derived trail
distance adapts to realized trade magnitude as the EMA updates.
P12: P_MIN in rl_pi_action_kernel now ISV-driven (slot 519,
default 0.015). At N=11, max single-action prob = 0.85
(uplift vs prior 0.80 at hardcoded P_MIN=0.02).
ISV slots: 519 P_MIN, 520 OUTCOME_ALPHA. RL_SLOTS_END -> 521.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New rl_frd_gate.cu kernel reads the FRD head's horizon-2 (medium,
~300 ticks) categorical distribution. For long openings, sums
probability mass in the positive tail (atoms > +0.5σ); for short
openings, sums the negative tail (atoms < -0.5σ). Overrides to Hold
when favorable mass < threshold.
Fires after confidence gate, before trail/heat/market pipeline.
Same preconditions: only gates flat positions with opening actions.
ISV slots: 516 THR_LONG (0.35), 517 THR_SHORT (0.35),
518 fired_count (diag). RL_SLOTS_END → 519.
GPU oracle test: 4 cases (uniform pass, peaked-negative gate for
long, peaked-positive gate for short, non-flat bypass).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New rl_confidence_gate.cu kernel computes C51 distributional Lower
Confidence Bound (μ - λσ) / σ_norm for the chosen action. When
position is flat and the selected action is an opening (a0/a1/a5/a6),
overrides to Hold if conf < threshold.
Fires after π action selection, before trail/heat/market pipeline.
Only gates on flat positions — existing positions pass through
unconditionally regardless of Q uncertainty.
ISV slots: 512 threshold (0.10), 513 λ (1.0), 514 σ_norm (1.0),
515 fired_count (diag). RL_SLOTS_END → 516.
GPU oracle test: 4 cases (low-conf gate, high-conf pass, non-flat
bypass, non-opening bypass).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Fixes the pre-existing fxcache_local_smoke test failure. Two changes:
1. Hash update: `13c0b086a975...` → `70e5bc3a401d...` — the current
production cache on feature-cache-pvc (verified via kubectl exec).
Both the local file (15.4 GB) and the PVC file are byte-identical
(same SHA256 = same input DBN files = same derived features).
Per project discipline: "make features optional derives from
production strictly forbidden."
2. Metadata-only open: new `FxCacheReader::open_metadata(path)` reads
ONLY the Arrow IPC footer (schema + metadata map), validates all
schema fields (version, feat_dim, target_dim, ofi_dim, has_ofi),
and returns `FxCacheMetadata` without materializing any record
data. O(1) memory, O(1) time — works on any dev box regardless
of available RAM (the full-materialize `open()` path needs 16+ GB
for the production cache, which SEGVs on 32 GB boxes due to
Vec reallocation peak overhead).
Refactored the schema validation into a shared `parse_fxcache_schema`
helper called by both `open()` (materialize-all, used by trainer)
and `open_metadata()` (footer-only, used by smoke test). Single
source of truth for field parsing + dim-mismatch assertions.
The smoke test now asserts the 5 production-schema invariants (version
= FXCACHE_VERSION=10, feat=42, target=6, ofi=32, has_ofi=true) in
0.00s with zero memory overhead. Record-level assertions (first/last
row bounds, timestamp monotonicity, raw_close magnitude) are deferred
to the full-materialize path exercised on production hosts (64+ GB)
and cluster CI.
Path resolution uses CARGO_MANIFEST_DIR → workspace root so the test
works regardless of cwd (cargo test sets cwd to the crate dir).
Closes the F.5 diagnosed l_v=104 + l_pi=-31 spike pattern at the root.
Pathology: `rl_reward_clamp_controller` widened the WIN/LOSS bounds
(slots 452/453) when clip_rate exceeded the 5% target — appeasement,
not control. The atom-span EWMA (slots 484/485) then ratcheted up to
track the wider WIN/LOSS. F.5 200-step smoke trajectory:
WIN: 1.0 → 41.3 (41×)
LOSS: 3.0 → 41.3 (14×)
V_MAX: 1.0 → 2.66
V_MIN: -1.0 → -2.74
Scaled rewards up to 14.71 flowed through unclamped, producing
advantage magnitudes of ~30 and PPO surrogate losses of ±30, V
regression losses up to 104. Pure positive-feedback loop: large
rewards → wider clamp → bigger V/Q targets → larger atom span →
larger reward signals permitted → repeat.
Fix: STOP writing to slots 452/453/484/485. The trainer-seeded
values (WIN=1.0, LOSS=3.0, V_MAX=1.0, V_MIN=-1.0) are the structural
bounds matching the C51 distributional Q head's design. Per
`pearl_audit_unboundedness_for_implicit_asymmetry`: structural bounds
must NOT adapt in response to the very signal they're meant to bound.
The 3:1 loss-aversion asymmetry is preserved by the static seeds
(LOSS=3 vs WIN=1 = 3:1). The C51 distributional resolution stays
matched to the bound. Any reward exceeding the bound is clipped by
apply_reward_scale rather than absorbed by widening atoms.
Diagnostic-only state retained:
* pos_max_ema (slot 478) — observed positive-tail magnitude
* neg_max_ema (slot 489) — observed negative-tail magnitude
* clip_rate_ema (slot 482) — fraction of steps where clamp fired
* MARGIN (slot 480) — what the controller WOULD widen to
* RATIO (slot 481) — what observed LOSS/WIN ratio implies
These surface what an unbounded controller WOULD adapt to under the
observed reward distribution — useful for understanding drift even
though the LOAD-BEARING slots are now static.
F.5 vs G.2 smoke comparison (same seed=4242, 200 steps, b_size=4):
Pre G.2 Post G.2 Reduction
l_pi abs_max 31.15 8.09 4×
l_v max 103.69 3.60 29×
l_v mean 1.79 0.19 10×
l_pi mean -0.12 0.06 ~stable
l_frd mean 0.43 0.50 unchanged
WIN bound →41.3 1.0 static
LOSS bound →41.3 3.0 static
V_MAX →2.66 1.0 static
V_MIN →-2.74 -1.0 static
Spike steps 20+ 5 ≥4×
Remaining 5 spikes are early-training noise (steps 11-59) that fade
naturally as V/Q converge. After step 59 only one mild spike at
step 131 (l_pi=3.35, l_v=2.75).
Pairs with G.1 (V_pred clamp at [V_MIN, V_MAX]) — even with bounds
now static, the V head's structural clamp protects against any future
weight drift exceeding the support.
Verification:
* cargo check -p ml-alpha → clean
* lib tests 66/66 (default), 5/6 ignored (1 pre-existing
fxcache_local_smoke env failure, unrelated)
* GPU tests: integrated_trainer_smoke 1/1 + frd_head 10/10 +
trade_management_kernels 5/5 → no regression
* audit-rust-consts → 0 flags
v_head_fwd now reads V_MIN/V_MAX from ISV slots 485/484 (same slots
the C51 atom support adapter writes) and clamps the linear output to
that range at the kernel boundary. Bounds advantage magnitude
(|returns − V_pred|) by 2 × V_MAX regardless of stale-V state.
Defensive fix per pearl_clamp_v_target_at_atom_span +
pearl_c51_atom_span_must_track_clamp_range — protects against the
canonical reward_scale↔V-head response-time pathology where V's stale
predictions amplify into PPO surrogate + V regression spikes when the
controller adapts reward_scale aggressively. In the F.5 200-step local
smoke this clamp didn't bite (V_pred stayed within bounds at the
short run length), but the structural protection matters for longer
production runs where V can drift before the controllers catch up.
Hard-saturated clamp (no straight-through estimator) — the gradient
at the boundary is zero in the "push further out" direction, normal
toward the interior. V can always learn back into bounds when its raw
output drifts out (target is inside bounds → grad pulls V back in),
but cannot push the prediction outside support.
API surface change: `ValueHead::forward(h_t, b_size, v_pred)` →
`ValueHead::forward(h_t, isv, b_size, v_pred)`. The 3 call sites in
IntegratedTrainer (step_synthetic + step_with_lobsim h_t/h_tp1) now
pass `&self.isv_d`.
Verification (RTX 3050 Ti):
* cargo check -p ml-alpha → clean
* integrated_trainer_smoke 1/1 → ok
* frd_head 10/10 + trade_management_kernels 5/5 → no regression
* audit-rust-consts → 0 flags
Independent finding from the smoke diag: the OBSERVED chronic spike
pattern (|l_pi|>30, l_v>100) traces to `rl_reward_clamp_controller`
widening WIN/LOSS bounds to 41.3 (vs seeds 1.0/3.0) when MARGIN hits
its MAX_MARGIN=5 ceiling. That's a separate failure mode addressed
in the next commit (structural cap on scaled reward magnitude).
Closes the literal/const drift gap that F.5 introduced. Per
feedback_isv_for_adaptive_bounds + feedback_single_source_of_truth_no_duplicates:
adaptive bounds belong in ISV (or in a single canonical const that
ISV references), never duplicated as literals across modules.
Single canonical source: `crate::rl::common::FRD_HORIZON_TICKS` +
`FRD_BUCKET_RANGE_SIGMA` (already declared in F.5).
Producer-side fixes:
* Trainer ISV bootstrap (integrated.rs): the seed values for slots
500-503 now dereference the canonical consts instead of hardcoded
60.0/300.0/1800.0/3.0 literals. Future tuning of the consts
automatically propagates to both ISV seeds and loader-side
labels — no manual sync required, no drift possible.
* compute_frd_labels (loader.rs): takes `horizon_ticks` and
`range_sigma` as parameters instead of reading consts directly.
Caller (the file-load closure) sources them from the new
MultiHorizonLoaderConfig fields.
Consumer-side fixes — 8 MultiHorizonLoaderConfig literal sites now
provide the two new fields, all defaulting to the canonical consts:
* crates/ml-alpha/src/data/loader.rs (2 internal test-fixture sites)
* crates/ml-alpha/tests/multi_horizon_loader.rs (2 sites)
* crates/ml-alpha/examples/alpha_train.rs (2 sites)
* crates/ml-alpha/examples/alpha_rl_train.rs (2 sites)
* crates/ml-backtesting/src/harness.rs (1 site)
* crates/ml-backtesting/tests/{trainer_parity,ring3_replay}.rs (2 sites)
The "optimal by default" property is preserved: every caller that
doesn't explicitly override gets the spec-recommended 60/300/1800
ticks + ±3σ. Callers that need to retune set the config fields, and
the trainer's ISV slots provide a runtime knob for the same numerics.
Verification (RTX 3050 Ti):
* cargo check -p ml-alpha -p ml-backtesting --examples --tests → clean
* cargo test --lib (6/6 unit tests for FRD label gen + loss_balance) → pass
* frd_head 10/10 + integrated_trainer_smoke 1/1 + trade_mgmt 5/5 → pass
* audit-rust-consts → 0 flags
The two new MultiHorizonLoaderConfig fields are required (no Default
impl) — callers MUST opt in to the FRD label-generation contract by
naming the fields. This is the same discipline applied across other
config consumers; making them Option<...> would silently default to
"no FRD labels" and break F.4's expected supervised signal.
Activates the FRD head's supervised training signal that F.4 wired
through the trainer. Per-file forward-return σ-bucketed labels
computed at load time + per-step write into trainer.frd_labels_d
before each step_with_lobsim.
Loader-side label generation (`compute_frd_labels` in data/loader.rs):
* Mid-price series from snapshots[i].levels[0]
* Per-file σ_per_step = sample-std of single-tick mid increments
* For each FRD_HORIZON h ∈ {60, 300, 1800}:
- r = (mid[i+h] - mid[i]) / (σ_per_step × sqrt(h)) ← Brownian scaling
- bucket = round(r × (FRD_N_ATOMS-1) / (2 × FRD_BUCKET_RANGE_SIGMA)
+ (FRD_N_ATOMS-1) / 2)
- clamp to [0, FRD_N_ATOMS-1] for tail returns
- sentinel -1 if i + h >= n
* Cached in LoadedFile.frd_labels_full alongside sigma_k_full /
outcome_*_full
* Per-anchor slice into LabeledSequence.frd_labels (length-1 vec
per horizon at the newest-snapshot index — h_t aligns with the
rightmost K position, the only one the FRD head supervises)
New structural constants in rl/common.rs:
* FRD_HORIZON_TICKS = [60, 300, 1800] ← matches ISV slots 500/501/502 defaults
* FRD_BUCKET_RANGE_SIGMA = 3.0 ← matches ISV slot 503 default
Per pearl_glm_fitter_link_must_match_inference: bucket-edge math
here MUST match the trainer-side softmax+CE atom interpretation.
Both reference the same const so they can't drift.
alpha_rl_train per-step wiring:
* Stage frd_labels_bh[b_idx × FRD_N_HORIZONS + h] from
s_t.frd_labels[h][0] (the per-batch label at this step's anchor)
* write_slice_i32_d_pub into trainer.frd_labels_d BEFORE
step_with_lobsim → bwd chain reads real labels in step_synthetic
Tests (3 new in loader::frd_label_tests, total 3/3 passing):
* frd_labels_flat_price_maps_to_mid_bucket — constant mid → all
non-sentinel labels = 10 (FRD_N_ATOMS/2 rounded); sentinel range
[n-h, n) tested exhaustively
* frd_labels_monotonic_ramp_lands_in_upper_buckets — linear ramp
mid[i] = 100 + 0.01×i produces forward returns way above 3σ at
every horizon → clamp to top bucket (FRD_N_ATOMS-1=20)
* frd_labels_short_input_below_h_ticks_all_sentinel — n=10 < h_ticks
for all 3 horizons → every label is -1 (no leak in the sentinel path)
Existing tests still pass:
* loss_balance lib tests 3/3
* frd_head GPU tests 10/10
* integrated_trainer_smoke 1/1
* trade_management_kernels 5/5
The full FRD head pipeline is now active end-to-end. Cluster smoke
will show FRD entropy_mean drift below ln(21) ≈ 3.044 once the bwd
gradient signal accumulates — the observable proof that supervised
learning is happening. The "frd" diag block from F.2 was always
prepared for this; F.5 just feeds it real signal.
F.6+ scope (deferred, separate sessions):
* P9 FRD gate — override action to Hold when entry_quality < THR
* Loss-balance controller integration for λ_frd (currently 1.0 default)
* Per-horizon Sharpe attribution in diag
Wires the F.3a/b/c backward kernels into IntegratedTrainer's per-step
flow so the FRD head trains end-to-end as a 6th loss-balanced head
alongside BCE/Q/π/V/aux. With labels currently sentinel-initialized to
-1 (F.5 loader will populate from forward-snapshot lookahead), the
chain produces zero gradients + zero loss — Adam steps are no-ops
modulo β decay, and the encoder receives no FRD-derived signal yet.
The wiring is complete and the path is exercised end-to-end; F.5 just
needs to swap the labels in for the head to start training.
IntegratedTrainer state additions:
* frd_w1_adam / frd_b1_adam / frd_w2_adam / frd_b2_adam — AdamW
instances for the 4 FRD weight tensors (LR mirrored per-step from
ISV[RL_FRD_LR_INDEX=499], seed 1e-3 per F.1).
* frd_labels_d — owned [B × FRD_N_HORIZONS] i32 buffer, sentinel-
initialized to -1 (every entry "missing horizon" → softmax_ce_grad
zeros loss + grad for every row). F.5 loader integration overwrites
pre-step from forward-return-bucketed labels.
LossLambdas extension:
* Added `frd: f32` field, default 1.0
* read_loss_lambdas_from_isv reads slot 498 (RL_FRD_LAMBDA_INDEX)
with the standard zero-sentinel bootstrap path
* Doc-comment updated: "5 heads / 5.0" → "6 heads / 6.0"
IntegratedStepStats extension:
* Added `l_frd: f32` — mean CE across (B × FRD_N_HORIZONS) rows
* step_synthetic returns the real l_frd from the bwd chain; the
new combined l_total formula includes `lambdas.frd × l_frd / 6`
step_synthetic bwd chain — inserted between Step 9 (Q/π/V Adam) and
Step 10 (grad_h_t_combined zero+accumulate):
1. softmax_ce_grad → frd_grad_logits_d + frd_loss_per_b_h_d
2. layer2_bwd → frd_grad_w2_pb_d, frd_grad_b2_pb_d, frd_grad_hidden_d
3. layer1_bwd → frd_grad_w1_pb_d, frd_grad_b1_pb_d, frd_grad_h_t_d
4. 4× reduce_axis0 to collapse per-batch scratch → final grads
5. 4× AdamW.step on w1/b1/w2/b2
6. read loss_per_b_h via mapped-pinned, average → l_frd_host
Step 10 grad_h_t_combined accumulation adds a third λ-weighted call:
accumulate_grad_h(frd_grad_h_t_d, lambdas.frd, &mut combined)
With sentinel labels (F.4 state) this contributes zero gradient to the
encoder backward — the wiring is exercised but silent. F.5 makes it
active by providing real labels.
alpha_rl_train diag JSON gains:
* "loss": { ..., "frd": stats.l_frd, ... }
* "lambdas": { ..., "frd": stats.lambdas.frd, ... }
Verification (RTX 3050 Ti):
* cargo check -p ml-alpha + --examples → clean
* integrated_trainer_step_with_lobsim_runs_without_panic → ok
(l_total 0.5073 vs prior 0.6087 — ÷6 instead of ÷5 expected;
l_frd=0 confirms sentinel labels are passing through cleanly)
* frd_head 10/10 tests still pass (no regression)
* trade_management_kernels 5/5 → no regression
* audit-rust-consts → 0 flags
F.5 (next, separate scope):
* Loader-side forward-return label generation (mid[i+h] - mid[i])/σ
bucketed into FRD_N_ATOMS=21 atoms over the ISV-driven ±range_σ
* Populate trainer.frd_labels_d before each step_with_lobsim call
* That unlocks the supervised learning signal; FRD entropy_mean
should start dropping below ln(21) in diag as the head trains.
Third and final FRD backward stage. Closes the chain from
softmax+CE loss back to the encoder's hidden state h_t.
Kernel `cuda/rl_frd_layer1_bwd.cu`:
* grid_dim = (B, 1, 1), block_dim = (HIDDEN_DIM=128, 1, 1)
* Phase 0: threads 0..63 stage dL/dpre_hidden = grad_hidden ×
1{hidden > 0} into shared mem (the cached post-ReLU `hidden`
buffer encodes the mask — hidden == 0 ⇔ pre-activation was
≤ 0 → ReLU killed it). Same thread also writes db1_per_batch.
* Phase 1: each thread k (k < 128) writes one row of
grad_W1_per_batch[b, k, 0..64] (64 writes per thread, no atomics)
* Phase 2: same thread computes grad_h_t[b, k] =
Σ_i W1[k, i] × dL/dpre_hidden[b, i]
* Per-(b, k, i) sole-writer per feedback_no_atomicadd
Rust wiring `FrdHead::layer1_bwd` — takes h_t, hidden (forward cache),
grad_hidden (from layer2_bwd), self.w1_d; writes grad_w1_per_batch,
grad_b1_per_batch, grad_h_t. The grad_h_t buffer becomes the encoder-
upstream gradient that the trainer's grad_h_accumulate kernel folds
into the encoder's gradient with λ_frd scaling (same pattern as Q/π/V
heads — wiring lives in F.4).
Tests (2 new, 10/10 file total):
* frd_layer1_bwd_finite_diff_w1 — perturbs the W1 slot with MAX
|analytical gradient| (instead of an arbitrary fixed slot — fp32
finite-diff is rounding-error-limited so a tiny gradient gives
misleading rel_err). At max-magnitude slot (k=84, i=55): analytical
= -0.0451, numerical = -0.0448, rel_err = 5.6e-3 — well within
1e-2 tolerance (slightly looser than dW2's 5e-3 because dW1
crosses an extra matmul + the ReLU mask boundary).
* frd_layer1_bwd_relu_mask_zeros_grad — fixture with h_t = all -1
produces ~half the hidden slots ReLU-masked (cached hidden = 0).
For every masked slot i, asserts:
* db1_per_batch[b, i] == 0 (exact equality — mask is hard 0)
* dW1_per_batch[b, k, i] == 0 for every k (~32 × 128 = 4096
slots checked)
Empirically 32/64 masked, 32/64 active — confirms ReLU mask
is wired through the chain correctly without leaking gradient
through dead branches.
F.3 backward chain is now complete end-to-end:
rl_frd_softmax_ce_grad (F.3a) → rl_frd_layer2_bwd (F.3b) →
rl_frd_layer1_bwd (F.3c) → grad_h_t (consumed by F.4 wiring)
F.4 wires Adam optimizers for W1/b1/W2/b2 + grad_h_accumulate into
the encoder gradient + loader-side label generation + λ_frd × CE
into stats.l_total.
Second of three FRD backward stages. Given dL/dlogits from F.3a's
softmax_ce_grad and the cached hidden activation from F.2's forward,
computes the layer-2 weight gradients via the standard chain rule
and emits the upstream gradient for layer-1 backward (F.3c).
Kernel `cuda/rl_frd_layer2_bwd.cu`:
* grid_dim = (B, 1, 1), block_dim = (FRD_HIDDEN_DIM=64, 1, 1)
* Phase 0: stage 63-slot grad_logits into shared (thread 63 idle)
* Phase 1: each thread i (i < 64) computes one row of per-batch
dW2 scratch: grad_w2_per_batch[b, i, 0..63] = h_bi × grad_logits[0..63]
(63 writes per thread, no atomics)
* Phase 2: each thread i computes dL/dhidden[b, i] = Σ_j W2[i, j] × grad_logits[j]
* Phase 3: thread i (i < 63) writes grad_b2_per_batch[b, i] = grad_logits[b, i]
* Per-batch scratch shape [B, FRD_HIDDEN_DIM, FRD_OUT_DIM] reduces
across batch via existing reduce_axis0 infra (caller's job, same
pattern as v_head_bwd / aux_heads_bwd)
Rust wiring `FrdHead::layer2_bwd`:
* Takes hidden (forward cache), grad_logits (from softmax_ce_grad),
self.w2_d
* Writes grad_w2_per_batch, grad_b2_per_batch, grad_hidden — all
sized to caller-allocated buffers
* Sole &self method (Adam step is the caller's responsibility)
Tests (2 new, 8/8 file total):
* frd_layer2_bwd_finite_diff_w2 — perturb W2[10, 5] by ±ε=1e-3,
compare (L(+) - L(-))/(2ε) to per-batch grad scratch. rel_err
= 6.27e-5 (better than F.3a's softmax-CE finite-diff because
the gradient magnitude here is larger so rounding error is
relatively smaller). Helper `ce_total_loss` re-uses
`softmax_ce_grad` to compute total CE for the perturbed forward
pass — pure GPU-oracle, no CPU softmax/CE reference impl.
* frd_layer2_bwd_db2_equals_grad_logits — analytical invariant:
db2_per_batch[b, j] must equal grad_logits[b, j] exactly (the
bias gradient is the identity passthrough at this layer). Cheap
structural check that catches dimension-shuffle bugs in the
kernel before they corrupt the reduce_axis0 step.
The kernel restores W2 to its original values after the perturbation
to keep test isolation clean — `&mut head` access pattern (proper
Rust borrowing, no UB const→mut casts).
F.3c (layer-1 backward: dW1, db1, dh_t with ReLU mask via the
cached hidden activation) is next.
Per-(batch, horizon) softmax + cross-entropy loss + gradient w.r.t.
the 21 atom logits. First of three backward stages — F.3b adds layer-2
weight grads (dW2, db2, dhidden), F.3c adds layer-1 weight grads
(dW1, db1, dh_t with ReLU mask).
Kernel `cuda/rl_frd_softmax_ce_grad.cu`:
* grid_dim = (B, FRD_N_HORIZONS, 1), block_dim = (FRD_N_ATOMS=21, 1, 1)
— one block per (batch, horizon) pair, threads cooperate over the
21 atoms via shared mem
* Standard numerically-stable softmax: shift by row_max, exponentiate,
normalize by row_sum (thread 0 does the serial reductions — 21
atoms is small enough warp-shuffle overhead isn't worth it)
* Gradient: (p[a] - 1{a==label}) / B at the source per v_head_bwd
convention (mean-reduce over batch)
* Loss: -log(p[label]) with 1e-30 floor against log(0)
* Sentinel label (-1) zeros both gradient row and loss — for the
missing-horizon case at the rightmost edge of the snapshot stream
(forward returns at h=300 ticks aren't realized for the last
300 snapshots; loader marks those labels with -1)
* Per feedback_no_atomicadd: per-(b, h, a) sole-writer pattern
Rust wiring `src/rl/frd.rs::FrdHead::softmax_ce_grad`:
* Second cubin loaded alongside fwd (separate module per the
aux_heads pattern; small handle, no impact on init time)
* Caller provides labels_d [B, FRD_N_HORIZONS] of i32 and gets back
grad_logits + per-(b, h) raw CE; sum + λ_frd scaling left to the
caller (F.4 will hook this into stats.l_total + Adam step)
Tests `tests/frd_head.rs` — 3 new GPU-oracle tests (6/6 file total),
all PASS on RTX 3050 Ti:
1. frd_softmax_ce_grad_uniform_logits_match_log_n_atoms — for any
label, uniform logits → CE = ln(FRD_N_ATOMS) = ln(21) ≈ 3.0445.
Also asserts per-row Σ grad_logits = 0 (softmax-CE invariant).
2. frd_softmax_ce_grad_sentinel_label_zeros_row — label=-1 with
non-trivial random logits produces exactly zero loss + grad
for every row (no leak through the sentinel path).
3. frd_softmax_ce_grad_finite_diff_matches_analytical — perturbs
one logit slot by ±ε=1e-3, compares (L(+ε) - L(-ε))/(2ε) to
the kernel's analytical gradient. rel_err ≈ 1.3e-3 (fp32
finite-diff is rounding-error-limited at this ε; tolerance
set to 5e-3 with explanatory comment).
The first two tests provide strong analytical oracles (no CPU
reference impl per feedback_no_cpu_test_fallbacks). The finite-diff
test cross-validates the full softmax+CE chain via a numerical
gradient — the standard ground-truth for autodiff kernels.
