Phase 2 of the Q-learning improvements spec. Adds an Implicit
Quantile Network head alongside the existing C51 head:
- rl_iqn_forward.cu: quantile embedding φ(τ) = ReLU(W × cos(iπτ)),
element-wise h_t ⊙ φ(τ), action-value projection. Plus
rl_iqn_expected_q for tau-mean reduction.
- rl_iqn_loss.cu: quantile Huber loss ρ_τ(δ) = |τ-1(δ<0)| × Huber(δ).
Block tree-reduce per batch (no atomicAdd).
- rl/iqn.rs: IqnHead struct with online + target weights, Xavier init,
forward/forward_target/expected_q/compute_loss methods.
ISV slots: 543 N_TAU (32), 544 ensemble_alpha (0.5), 545 LR (1e-3).
Not yet wired into the trainer — head is constructible and kernels
compile. Ensemble integration is the next step.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replaces 1-step Bellman r + γQ(s') with n-step R_n + γⁿQ(s_{t+n}).
At γ=0.995, n=10 gives the agent 10 real rewards (~2.5 seconds)
before bootstrapping from Q, dramatically reducing bootstrap error.
Implementation:
- NStepEntry ring buffer per batch in IntegratedTrainer
- push_to_replay accumulates entries, flushes when len==n or done
- R_n = Σ γᵏ rₖ computed at flush time (both scaled + raw)
- n_step_gamma = γⁿ (or 0 if any done in window) stored per transition
- bellman_target_projection.cu uses per-transition n_step_gamma
instead of the global ISV γ for the bootstrap discount
- project_bellman_target wrapper takes n_step_gammas_d buffer
ISV slot 542 (RL_N_STEP_INDEX, default 10). Local smoke: 1k steps,
no crash, replay=150 (correct for n=10 with dones).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds RL_N_STEP_INDEX (slot 542, default 10), n_step_gamma field to
Transition, and NStepEntry ring buffer struct to IntegratedTrainer.
Remaining: implement n-step accumulation in push_to_replay and
modify bellman_target_projection.cu to use n_step_gamma.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Enumerates all project rules that apply to the Q-learning
improvements: CPU read-only, no atomicAdd, mapped-pinned only,
ISV-driven params, first-observation bootstrap, bilateral clamp,
raw reward in replay, no stubs/TODO/feature flags, GPU oracle
tests, local smoke before cluster, surfer philosophy constraints.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three-phase plan to push Q convergence past profitability:
1. N-step returns (n=10): 10× more direct reward signal
2. IQN complementary head alongside C51: ensemble action selection
3. Noisy linear layers: state-dependent exploration
C51 stays for the confidence gate's distributional LCB. IQN adds
flexible quantile estimation. Ensemble combines both for action
selection. All ISV-driven.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Prepared for next run after the γ=0.99 1M run completes:
- γ floor 0.99 → 0.995: horizon 100 → 200 steps (50 seconds).
Real ES directional moves (2-5 points) happen at this scale.
- PER 16384 → 32768: 2048 unique steps of replay depth. Supports
the 200-step γ horizon with margin.
- Min-hold 50 → 100 steps (25 seconds): commit to the full wave.
Short-hold penalty threshold matches.
Safe to push because raw-reward re-normalization eliminates scale
drift in the deeper buffer, and the ±2% scale clamp keeps targets
stable across the 2048-step replay window.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The agent was break-even at the wrong timescale — trading in
bid-ask noise at 1-4 second holds where there IS no edge. Real
directional moves live at 10-60 seconds.
Three changes push the agent to wave-scale:
1. γ floor 0.90 → 0.99: effective horizon 10 → 100 steps (25s).
Q now values what happens 25 seconds from now, not just the
next tick. The FRD's 10s and 600s horizons become relevant.
2. Min-hold 20 → 50 steps (12.5s): forces commitment to waves,
not ripples. Short-hold penalty threshold matches.
3. Long-ride bonus: at trade close, profitable rides get reward
multiplied by (1 + bonus × sqrt(hold_time)). A 100-step
winning ride gets 21× the raw PnL as reward. Combined with
per-step hold bonus boosted to $2.00. "The best wave of the
session deserves the biggest cheer."
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The reward_scale oscillated 10,873× (min 0.000123, max 1.33) because
the Wiener-α=0.4 blend tracked sparse trade PnL spikes instantly.
Old transitions in PER had stale-scale rewards even with raw-reward
re-normalization (the current scale itself was unstable).
Per-step clamp: scale can only move ±2% from previous value.
Doubling takes ~35 steps, halving takes ~35 steps — fast enough to
track regime changes, stable enough for Q targets across the replay
window. Max oscillation over 1000 steps: ~7× (was 10,873×).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Min-hold check was overriding heat cap's FlatFromLong/Short to Hold,
preventing the safety exit. Position grew to 9 (above cap 8) because
the flat was blocked by min-hold, then next step added lots.
Fix: min-hold reads pos_state and ISV heat cap slot. If |position|
>= cap, the close action passes through regardless of hold time.
Safety always overrides patience.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Overrides closing actions (a3/a4/a9/a10) to Hold when
steps_since_done < RL_MIN_HOLD_STEPS_INDEX (slot 536, default 20).
The agent CANNOT exit before the minimum — forced to ride the wave.
Trail stops still fire regardless (safety overrides patience) —
if the market moves against the position past the trail distance,
the stop-loss exits even within the min-hold window.
Pipeline order: action selection → confidence gate → FRD gate →
min_hold_check → trail_mutate → trail_stop → heat_cap →
actions_to_market_targets
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three ISV-driven reward shaping components discourage churning and
reward patience:
1. Entry cost ($15 default): subtracted when opening a new position.
Agent must expect profit > cost to justify entry. Slightly above
ES 1-tick spread ($12.50) so marginal trades are net-negative.
2. Short-hold penalty (0.5× for holds < 20 steps): multiplicative
penalty at trade close for quick flips. "Don't bail on the first
bump" — halves the reward for sub-5-second holds.
3. Hold bonus ($0.50/step × sqrt(hold_time)): per-step reward for
staying in a profitable position. "Ride the wave" — incentivizes
patience when the trade is working.
Runs BEFORE reward_scale so all costs/bonuses are in raw USD terms.
ISV slots 532-535. RL_SLOTS_END → 536.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Binary argmax agreement is pure noise for near-uniform distributions
(both Q and π give ~9% to each of 11 actions, so argmax flips on
0.001 differences → metric reads 0 even when KL=0.02).
Cosine similarity between E_Q[a] (expected Q-value per action) and
softmax(π)[a] measures full ranking direction agreement. Robust to
near-uniform: two identical distributions → cosine=1.0 regardless
of which action is marginally highest.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Root cause of q_pi_agree collapse: the symmetric Schulman controller
decayed λ from 0.5 to 0.001 in 34 steps (÷1.2/step), killing Q→π
coupling. Once decoupled, Q and π learned independent policies,
making q_pi_agree drop to 1e-22.
Three fixes:
1. MIN_LAMBDA 0.001 → 0.05: Q pull never drops below 5% strength
2. Asymmetric rates: ramp 1.2×/step (4 steps to double), decay
0.998×/step (347 steps to halve). Coupling establishes fast and
persists for thousands of steps.
3. Dead zone tolerance 1.5× → 3.0×: natural KL fluctuation stays
in-band instead of triggering constant ramp/decay cycles.
Seed λ raised to 0.1 so distillation is active from step 0.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Stored raw_reward alongside scaled reward in Transition. At sample
time, consumer re-applies current reward_scale to raw_reward instead
of using the stale-scale stored reward. This eliminates off-policy
scale drift that caused q_pi_agree to collapse from 0.125 to 1e-22
over 40k steps with PER=32768.
PER capacity set to 16384 (1024 unique steps at b=16) — balances
replay depth against h_t representation staleness.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The CLI default overrode the trainer config default. Without passing
--per-capacity explicitly, the Argo run used 4096 (256 unique steps
at b=16) instead of the intended 32768 (2048 steps).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Root cause of Q non-convergence: at reward_scale ≈ 0.003, a typical
±$5 trade maps to ±0.015 scaled reward. With 21 atoms spanning [-1,+1]
(step=0.1), the entire reward distribution falls within a SINGLE atom.
Q cannot distinguish wins from losses — confirmed by q_pi_agree ≈ 0.
Fix 1: Tighten atom span to [-0.5, +0.5]. Now atom_step = 0.05.
A ±$5 trade at scale=0.01 spans 2 atoms; at scale=0.05 spans 10.
The reward clamp controller ratchets the span at runtime if rewards
exceed the bounds.
Fix 2: PER capacity 4096 → 32768. At b=16, old capacity retained
only 256 unique steps — Q replayed each transition 1-2× before
eviction. New capacity retains 2048 steps, giving Q proper replay
depth for convergence.
Also aligns reward clamp bounds (WIN/LOSS) with the new span.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
With strict >, pos=8 passed the cap check, then a LongLarge(+2)
made pos=10 before the next step's cap fired. Using >= prevents
position from ever exceeding the cap by the fill size.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Controller oscillated: 3 dones at step 15k spiked EMA above 2×target,
triggering thousands of steps of tightening that killed trades.
Higher target (10% vs 2%) matches the natural trade frequency at
b=16. Wider dead zone (5× above / 0.2× below) prevents single-batch
spikes from triggering tighten/relax oscillation.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Confidence gate LCB fix worked (0 fires post-warmup) but FRD gate
blocked 5212 entries in 5k steps — the FRD head learned "forward
returns are negative" during warmup (losing trades), so favorable
tail mass dropped below 0.15 for all actions.
Lower default to 0.05 and floor to 0.0 so the adaptive controller
can fully disable when trades dry up.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The old formula conf = clamp((μ - λσ) / σ_norm, 0, 1) produced
conf=0 for ANY distribution with σ > μ — including uniform Q at
training start (μ=0, σ=0.577 → LCB=-0.577 → conf=0). This caused
permanent 100% block rate regardless of threshold setting.
New formula: conf = clamp((μ - V_MIN - λσ) / (V_MAX - V_MIN), 0, 1)
Shifts by V_MIN so conf measures "how far above worst case":
Uniform: conf = 0.21 (passes threshold 0.01)
Peaked V_MAX: conf ≈ 1.0 (high confidence)
Peaked V_MIN: conf ≈ 0.0 (blocked — correct)
Also sets conf gate floor to 0.0 so the adaptive controller can
fully disable the gate when trades dry up.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
When C51 LCB is ≈0 for all actions (typical early training), even
0.001 threshold blocks everything. Floor=0.0 lets the controller
drive to zero when trades dry up, then ratchet back as Q calibrates.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New rl_gate_threshold_controller.cu watches dones EMA and adjusts
confidence + FRD gate thresholds via Schulman bounded step:
- dones_ema < target×0.5 → relax thresholds (×0.95)
- dones_ema > target×2.0 → tighten thresholds (÷0.95)
ISV-driven: target (0.02), conf bounds (0.001-0.50), FRD bounds
(0.05-0.50), adjust rate (0.95). Runs per step after warmup.
Also lowers default thresholds: conf 0.10→0.01, FRD 0.35→0.15.
Post-warmup, the controller adapts these based on actual trade
flow instead of relying on static defaults.
ISV slots 525-531. RL_SLOTS_END → 532.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Both gates now read RL_GATE_WARMUP_STEPS_INDEX (slot 524, default
10000) and return early when current_step < warmup. During warmup,
the agent opens positions freely, collects reward signal, and
calibrates Q. After warmup, gates activate and filter low-quality
entries.
Without warmup, gates blocked 100% of opening actions from step 0
(uniform Q → zero confidence → permanent Hold attractor → zero
trades → zero reward → Q never learns). Confirmed by 50k-step
L40S run with zero dones across 800k action decisions.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replaces the 4-step DAG (check-cache → compile on CPU → warmup GPU →
train on GPU) with a single pod on the L40S that does everything:
git fetch → incremental cargo build (~3s warm) → train.
Eliminates: separate compile node, node autoscale wait, binary
transfer, fxcache step. The ci-builder image has CUDA 13.0 devel +
Rust — compilation and training use the same CUDA libs.
The cargo-target-cuda PVC provides incremental build cache across
runs. LD_LIBRARY_PATH strips the stubs dir so real CUDA libs load.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The MBP-10 predecoded sidecars live on feature-cache-pvc at
/feature-cache/predecoded/, not on training-data-pvc. Points
--predecoded-dir to the correct PVC mount.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
alpha_rl_train reads MBP-10 data with its own predecoded cache —
it never uses the fxcache pipeline. Skip the 15-minute feature
extraction step entirely for model=alpha-rl by depending only on
ensure-binary + gpu-warmup.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The ml-alpha branch uses alpha_rl_train (from ml-alpha crate) with
different CLI args than train_baseline_rl (from ml crate). Adds
model=alpha-rl case that:
- Builds ml-alpha --example alpha_rl_train
- Runs with --mbp10-data-dir, --predecoded-dir, --n-steps 50000,
--n-backtests 16 (matching the local smoke config)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
28 parallel alpha-feature chunks × ~3GB each = 84GB peak at merge.
Capping at 8 threads keeps peak at ~24GB (well within 96Gi limit)
while still saturating I/O. This is a one-time cost — after the
cache is populated on the PVC, subsequent runs skip entirely.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
With cargo-target-cuda PVC persisting across pods, CARGO_INCREMENTAL=1
gives true incremental builds: only recompile changed crates. A
single-crate ml-alpha change rebuilds in ~45s vs 9min full workspace.
sccache removed — it conflicts with incremental compilation and is
redundant when the target dir persists. The binary-by-SHA cache on
the training-data PVC (already implemented) provides the "skip
compile entirely on re-run" fast path.
Expected workflow time reduction:
Before: compile 9m + fxcache 18m + train ~30m = ~57m
After: compile <1m (incremental) + fxcache <1m (PVC cache) + train ~30m = ~32m
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The imbalance bar cache now writes to <mbp10_dir>/.cache/ on the PVC.
The ensure-fxcache step needs write access to persist the cache
across runs (was readOnly: true, causing silent cache-write failures).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Cache path moved from /tmp/.foxhunt_imbalance_cache/ (ephemeral,
lost between Argo pods) to <mbp10_dir>/.cache/ (on the training-data
PVC). Saves ~4 minutes per Argo submission by avoiding recomputation
of 209M trade ticks → 17.8M imbalance bars.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Completes P2 by injecting rl_encoder_context_broadcast launch inside
the perception trainer's forward_only path — runs after
snap_feature_assemble fills dims [0..40] and before VSN/Mamba2
consume the window tensor. Fills dims [40..56] with per-batch
trade_context (4) + multires (12) features.
Device pointers to the context buffers (owned by IntegratedTrainer)
are set on PerceptionTrainer before each forward_encoder call. The
broadcast kernel reads these and writes directly into window_tensor_d
at the correct column offsets for all B×K rows.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Introduces ENCODER_INPUT_DIM = 56 (FEATURE_DIM + 16). All encoder
first-layer weight matrices (VSN gate, Mamba2 L1 input projection)
now sized for 56 input dims. The extra 16 are per-batch state:
4 trade_context + 12 multires features.
- snap_feature_assemble_batched: output stride → ENCODER_INPUT_DIM,
zero-fills dims [40..56] for the broadcast kernel to overwrite.
- New rl_encoder_context_broadcast.cu: writes trade_context_d[B×4]
+ multires_output_d[B×12] into each of the K sequence rows per
batch at positions [40..56].
- CfcConfig.n_in, Mamba2 L1 in_dim, VSN gate, window_tensor_d,
all forward/backward scratch buffers updated to ENCODER_INPUT_DIM.
- CfcTrunk default config updated.
The broadcast kernel launch integration into the forward_only path
is the final wire-up step — until then dims 40-55 are zero-filled
(safe: Xavier init on new columns means encoder starts by learning
to ignore them, then gradually incorporates the signal).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
P1: New rl_trade_context_update.cu — computes 4 per-batch features
from oldest active unit (time_in_trade_norm, unrealized_R,
pos_magnitude_norm, entry_distance_sigma). Output in
trade_context_d[B×4], updated after unit_state_update each step.
P0: New rl_multires_features_update.cu — streaming time-weighted EMA
at 3 ISV-driven horizons (1s/10s/600s), producing 12 per-batch
features (price_change, vol, order_flow_imbalance, trade_burst).
O(1) state per feature vs circular buffer — same time-constant
semantics.
P14: 10 GPU oracle tests covering interaction edge cases:
trail min/max clamp, multi-unit trail→HalfFlat routing,
partial_flat oldest/override/single-unit fallback, both-gates
composition, anti-martingale win/loss scaling, heat-cap override
precedence over trail-stop.
ISV slots: 521-523 (multires horizons). RL_SLOTS_END → 524.
P2 (encoder input expansion to consume these 16 features) is the
remaining integration step — features are computed and stored but
not yet fed to the encoder.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
P10: New rl_recent_outcome_update.cu — per-batch signed outcome EMA
(sign(reward) on done steps) feeds per-batch anti-martingale
sizing in actions_to_market_targets. Replaces the single ISV
scalar with a per-batch buffer for multi-batch granularity.
P11: Trail bootstrap switched from vwap × 1e-3 × k_init to
k_init × MEAN_ABS_PNL_EMA (slot 423). Vol-derived trail
distance adapts to realized trade magnitude as the EMA updates.
P12: P_MIN in rl_pi_action_kernel now ISV-driven (slot 519,
default 0.015). At N=11, max single-action prob = 0.85
(uplift vs prior 0.80 at hardcoded P_MIN=0.02).
ISV slots: 519 P_MIN, 520 OUTCOME_ALPHA. RL_SLOTS_END -> 521.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New rl_frd_gate.cu kernel reads the FRD head's horizon-2 (medium,
~300 ticks) categorical distribution. For long openings, sums
probability mass in the positive tail (atoms > +0.5σ); for short
openings, sums the negative tail (atoms < -0.5σ). Overrides to Hold
when favorable mass < threshold.
Fires after confidence gate, before trail/heat/market pipeline.
Same preconditions: only gates flat positions with opening actions.
ISV slots: 516 THR_LONG (0.35), 517 THR_SHORT (0.35),
518 fired_count (diag). RL_SLOTS_END → 519.
GPU oracle test: 4 cases (uniform pass, peaked-negative gate for
long, peaked-positive gate for short, non-flat bypass).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New rl_confidence_gate.cu kernel computes C51 distributional Lower
Confidence Bound (μ - λσ) / σ_norm for the chosen action. When
position is flat and the selected action is an opening (a0/a1/a5/a6),
overrides to Hold if conf < threshold.
Fires after π action selection, before trail/heat/market pipeline.
Only gates on flat positions — existing positions pass through
unconditionally regardless of Q uncertainty.
ISV slots: 512 threshold (0.10), 513 λ (1.0), 514 σ_norm (1.0),
515 fired_count (diag). RL_SLOTS_END → 516.
GPU oracle test: 4 cases (low-conf gate, high-conf pass, non-flat
bypass, non-opening bypass).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Fixes the pre-existing fxcache_local_smoke test failure. Two changes:
1. Hash update: `13c0b086a975...` → `70e5bc3a401d...` — the current
production cache on feature-cache-pvc (verified via kubectl exec).
Both the local file (15.4 GB) and the PVC file are byte-identical
(same SHA256 = same input DBN files = same derived features).
Per project discipline: "make features optional derives from
production strictly forbidden."
2. Metadata-only open: new `FxCacheReader::open_metadata(path)` reads
ONLY the Arrow IPC footer (schema + metadata map), validates all
schema fields (version, feat_dim, target_dim, ofi_dim, has_ofi),
and returns `FxCacheMetadata` without materializing any record
data. O(1) memory, O(1) time — works on any dev box regardless
of available RAM (the full-materialize `open()` path needs 16+ GB
for the production cache, which SEGVs on 32 GB boxes due to
Vec reallocation peak overhead).
Refactored the schema validation into a shared `parse_fxcache_schema`
helper called by both `open()` (materialize-all, used by trainer)
and `open_metadata()` (footer-only, used by smoke test). Single
source of truth for field parsing + dim-mismatch assertions.
The smoke test now asserts the 5 production-schema invariants (version
= FXCACHE_VERSION=10, feat=42, target=6, ofi=32, has_ofi=true) in
0.00s with zero memory overhead. Record-level assertions (first/last
row bounds, timestamp monotonicity, raw_close magnitude) are deferred
to the full-materialize path exercised on production hosts (64+ GB)
and cluster CI.
Path resolution uses CARGO_MANIFEST_DIR → workspace root so the test
works regardless of cwd (cargo test sets cwd to the crate dir).
Closes the F.5 diagnosed l_v=104 + l_pi=-31 spike pattern at the root.
Pathology: `rl_reward_clamp_controller` widened the WIN/LOSS bounds
(slots 452/453) when clip_rate exceeded the 5% target — appeasement,
not control. The atom-span EWMA (slots 484/485) then ratcheted up to
track the wider WIN/LOSS. F.5 200-step smoke trajectory:
WIN: 1.0 → 41.3 (41×)
LOSS: 3.0 → 41.3 (14×)
V_MAX: 1.0 → 2.66
V_MIN: -1.0 → -2.74
Scaled rewards up to 14.71 flowed through unclamped, producing
advantage magnitudes of ~30 and PPO surrogate losses of ±30, V
regression losses up to 104. Pure positive-feedback loop: large
rewards → wider clamp → bigger V/Q targets → larger atom span →
larger reward signals permitted → repeat.
Fix: STOP writing to slots 452/453/484/485. The trainer-seeded
values (WIN=1.0, LOSS=3.0, V_MAX=1.0, V_MIN=-1.0) are the structural
bounds matching the C51 distributional Q head's design. Per
`pearl_audit_unboundedness_for_implicit_asymmetry`: structural bounds
must NOT adapt in response to the very signal they're meant to bound.
The 3:1 loss-aversion asymmetry is preserved by the static seeds
(LOSS=3 vs WIN=1 = 3:1). The C51 distributional resolution stays
matched to the bound. Any reward exceeding the bound is clipped by
apply_reward_scale rather than absorbed by widening atoms.
Diagnostic-only state retained:
* pos_max_ema (slot 478) — observed positive-tail magnitude
* neg_max_ema (slot 489) — observed negative-tail magnitude
* clip_rate_ema (slot 482) — fraction of steps where clamp fired
* MARGIN (slot 480) — what the controller WOULD widen to
* RATIO (slot 481) — what observed LOSS/WIN ratio implies
These surface what an unbounded controller WOULD adapt to under the
observed reward distribution — useful for understanding drift even
though the LOAD-BEARING slots are now static.
F.5 vs G.2 smoke comparison (same seed=4242, 200 steps, b_size=4):
Pre G.2 Post G.2 Reduction
l_pi abs_max 31.15 8.09 4×
l_v max 103.69 3.60 29×
l_v mean 1.79 0.19 10×
l_pi mean -0.12 0.06 ~stable
l_frd mean 0.43 0.50 unchanged
WIN bound →41.3 1.0 static
LOSS bound →41.3 3.0 static
V_MAX →2.66 1.0 static
V_MIN →-2.74 -1.0 static
Spike steps 20+ 5 ≥4×
Remaining 5 spikes are early-training noise (steps 11-59) that fade
naturally as V/Q converge. After step 59 only one mild spike at
step 131 (l_pi=3.35, l_v=2.75).
Pairs with G.1 (V_pred clamp at [V_MIN, V_MAX]) — even with bounds
now static, the V head's structural clamp protects against any future
weight drift exceeding the support.
Verification:
* cargo check -p ml-alpha → clean
* lib tests 66/66 (default), 5/6 ignored (1 pre-existing
fxcache_local_smoke env failure, unrelated)
* GPU tests: integrated_trainer_smoke 1/1 + frd_head 10/10 +
trade_management_kernels 5/5 → no regression
* audit-rust-consts → 0 flags
v_head_fwd now reads V_MIN/V_MAX from ISV slots 485/484 (same slots
the C51 atom support adapter writes) and clamps the linear output to
that range at the kernel boundary. Bounds advantage magnitude
(|returns − V_pred|) by 2 × V_MAX regardless of stale-V state.
Defensive fix per pearl_clamp_v_target_at_atom_span +
pearl_c51_atom_span_must_track_clamp_range — protects against the
canonical reward_scale↔V-head response-time pathology where V's stale
predictions amplify into PPO surrogate + V regression spikes when the
controller adapts reward_scale aggressively. In the F.5 200-step local
smoke this clamp didn't bite (V_pred stayed within bounds at the
short run length), but the structural protection matters for longer
production runs where V can drift before the controllers catch up.
Hard-saturated clamp (no straight-through estimator) — the gradient
at the boundary is zero in the "push further out" direction, normal
toward the interior. V can always learn back into bounds when its raw
output drifts out (target is inside bounds → grad pulls V back in),
but cannot push the prediction outside support.
API surface change: `ValueHead::forward(h_t, b_size, v_pred)` →
`ValueHead::forward(h_t, isv, b_size, v_pred)`. The 3 call sites in
IntegratedTrainer (step_synthetic + step_with_lobsim h_t/h_tp1) now
pass `&self.isv_d`.
Verification (RTX 3050 Ti):
* cargo check -p ml-alpha → clean
* integrated_trainer_smoke 1/1 → ok
* frd_head 10/10 + trade_management_kernels 5/5 → no regression
* audit-rust-consts → 0 flags
Independent finding from the smoke diag: the OBSERVED chronic spike
pattern (|l_pi|>30, l_v>100) traces to `rl_reward_clamp_controller`
widening WIN/LOSS bounds to 41.3 (vs seeds 1.0/3.0) when MARGIN hits
its MAX_MARGIN=5 ceiling. That's a separate failure mode addressed
in the next commit (structural cap on scaled reward magnitude).
