A) TD-propagation diagnostic smoke test (sparse rewards)
- New smoke_tests/td_propagation.rs captures training_sharpe_ema across
20 epochs with micro_reward_scale=0 to answer: "can sparse-reward
Q-learning extract policy from trade-completion P&L alone?"
- Asserts: finite sharpe_ema, no monotonic degradation (tolerance 0.1
over first-3 vs last-3 epoch avg), q_gap > 0.05
- First run answered YES: val Sharpe peaks at +12.49 at epoch 8 with
pure sparse rewards, confirming the objective is learnable. The
remaining issue is stability/overfitting, not TD propagation.
B) Entropy → plasticity gate in training_loop
- D3/N3 shrink_perturb trigger now fires on `last_action_entropy < 0.3`
OR `health_value < 0.3` (OR semantics, 3 consecutive epochs).
- Catches action-collapse cases the generic health metric misses: Q-values
separate cleanly but argmax stays pinned to a single branch.
- tracing::info now logs both signals.
C) commitment_lambda coverage verified
- Almgren-Chriss sqrt market-impact already in compute_tx_cost
(trade_physics.cuh:168-171). commitment_lambda was pure duplication.
- Inventory doc updated: P2 marked satisfied, table row status updated.
Files touched:
- crates/ml/src/trainers/dqn/smoke_tests/mod.rs (+2)
- crates/ml/src/trainers/dqn/smoke_tests/td_propagation.rs (new)
- crates/ml/src/trainers/dqn/trainer/training_loop.rs (+15/-3)
- docs/superpowers/specs/2026-04-21-phase1-reward-inventory.md (+6/-3)
Smoke test PASSED locally (RTX 3050 Ti, 30.99s end-to-end).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 2 P1 relocations — the intents behind two deleted reward shaping
terms now live at the correct layer: diagnostic monitoring, not reward.
1. training_sharpe_ema (was w_dsr reward input):
Field already existed. Added to HEALTH_DIAG log line as `sharpe_ema`.
Available for early-stopping, model selection, and exploration gating
without perturbing the objective.
2. action_entropy (was position_entropy_weight reward):
Computed from summary.action_counts at GPU summary download —
Shannon entropy normalized to [0, 1] by ln(n_bins). One-epoch lag
is fine for a diagnostic. Added to HEALTH_DIAG as `action_entropy`.
Stored in new trainer field `last_action_entropy: Option<f32>`.
New HEALTH_DIAG suffix: `diag [sharpe_ema={:.3} action_entropy={:.2}]`.
Verified visible on E1 smoke test epoch 18/19 output.
Also: honest recalibration of Phase 2 results added to the inventory
doc. The earlier commit messages framed "-150 → -17 Sharpe" as a 5×
improvement; in absolute terms Sharpe -17 is still catastrophic (you'd
blow the account). Phase 2 stopped active capital destruction but did
not find alpha. Sharpe_raw per bar went from -0.39 (lot of loss per bar)
to -0.09 (little loss per bar) — still net losing. q_gap=0.17 is a
capacity metric (network CAN differentiate), not profitability.
Path to actual profitability (Sharpe > 0) remains:
- TD propagation verification (Task #8)
- Unified env kernel (Phase 3)
- Possibly a different objective entirely
- Richer features (42 market + 20 OFI may be information-starved)
The rule going forward: before celebrating future improvements, ask
whether the result *crosses zero* (profitable) or is just *less
negative* (still losing).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mass deletion of the "v7 gem" reward terms identified in the Phase 1
inventory as category errors — each one rewarded an outcome-adjacent
behavior instead of encoding the underlying physics, and each
empirically hurt validation metrics more than it helped training
stability.
Deleted from experience_kernels.cu and all plumbing (Rust configs,
launch args, hyperopt logs, TOML entries):
* order_credit_weight - reward redundant with compute_tx_cost
(order_type_idx already differentiates
fills by order type)
* risk_efficiency_weight - reward double-counted drawdown penalty
asymmetrically (only on winners)
* urgency_credit_weight - reward was vol-normalized unrealized P&L,
pure rename of core return
* commitment_lambda - triple-counted churn + tx_cost
* w_dsr - kernel wrote DSR EMA but no longer added
to reward (dead); removed the EMA
bookkeeping too
* dsr_eta - kernel arg for the deleted DSR EMA
* position_entropy_weight - rewarded action-bucket diversity
regardless of outcome; histogram buffer
+ zero-init removed too
* exit_timing_weight - already inactive (used raw_next future
price, comment-deleted earlier)
* ofi_reward_weight - dead plumbing; OFI already passed as
feature through state[OFI_START..]
* opportunity_cost_scale - penalized flat when Q-gap wide;
redundant with Q-values themselves
Kernel arg count: experience_env_step_batch shrank from ~55 to ~45 args.
Rust-side config surface reduced correspondingly.
Results on E1 smoke test (20-epoch):
BEFORE any Phase 2 work:
Val Sharpe -120 to -150, MaxDD 10-15%, Sharpe_raw -0.39
AFTER reward_noise + Kelly (both envs) + urgency + this sweep:
Val Sharpe -17 to -22 (7× better)
Val MaxDD 0.27% (40× better)
Val Sharpe_raw ~-0.09 (4× better)
Training Sharpe_raw ~0 (stabilized from ±20 swings)
Final q_gap 0.1712 (highest yet, collapse mechanism fine)
The extreme train-Sharpe swings (+17 one epoch, -13 next) were not
learning dynamics — they were shaping-term noise. Core reward (P&L +
drawdown + churn + holding + tx_cost + Kelly physics cap) gives training
metrics that actually reflect what the model does.
Inventory doc (docs/superpowers/specs/2026-04-21-phase1-reward-inventory.md)
extended with a "better-form taxonomy" section: every deleted gem has
a correct layer it belongs to (physics, feature, diagnostic, gradient-
level regularization — not reward). Kelly cap and Q-target smoothing
are already relocated; others are scheduled per the taxonomy's P1/P2/P3
priority list.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extends the reward inventory with a review pass that preserves the
*ideas* behind behavioral terms while deleting the wrong-level mechanics.
Gems (relocate, don't delete):
- Probabilistic fill model (real physics, not shaping) → apply in both
modes via shared Philox RNG + exploration_scale
- Kelly-as-physics-constraint → hard position-size cap in trade_physics,
not a soft reward
- Path quality via per-step drawdown integration → no separate term,
just run drawdown penalty every bar
- Saboteur relocation → both modes, scaled by exploration_scale (training
1.0, validation 0.5, diagnostic 0.0) — avoids creating a new clean-vs-
noisy mismatch in the opposite direction
Pearls:
- Sparse reward is fine if TD propagation works — diagnose before
deleting micro_reward_scale; measure cov(Q(s_entry,a), trade_return)
- Every behavioral term maps to one of four failure modes: double-count,
misplaced physics, wrong-level regularization, or compensating for a
downstream bug. None are semantically "neutral" shaping.
Novel architectural moves:
- Diagnostic-only entropy tracking (no reward, just HEALTH_DIAG
logging)
- Single exploration_scale scalar replacing all mode toggles
(feedback_no_feature_flags compliant)
- Layered reward with per-term budget caps — makes the train/val Sharpe
gap computable instead of uncomputable magic
- Q-target smoothing replacing reward_noise_scale (regularization at
gradient level, not reward level)
Updated disposition table: 4 DELETE, 2 MOVE, 2 RELOCATE-to-physics,
1 DIAGNOSE-then-DELETE, 4 delete-dead-plumbing.
Three relocations (Kelly cap, saboteur scaling, Q-target smoothing) can
land as independent PRs before the unified_env_kernel rewrite, shrinking
Phase 2's change surface.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Enumerates every term that contributes to training or validation reward,
classifies each as P&L-aligned (keeper) or behavioral (remover), with
source line references into experience_kernels.cu and backtest_env_kernel.cu.
Findings:
- 7 P&L-aligned terms to preserve in the unified env (core return,
tx_cost, drawdown, inventory, churn, capital floor, + vol normalization)
- 8 active behavioral training-only terms to delete:
* order_credit_weight (rewards theoretical limit-order savings,
double-counts actual tx_cost)
* risk_efficiency_weight (double-counts drawdown asymmetrically)
* urgency_credit_weight (redundant with core return)
* kelly_sizing_weight (rewards matching a formula, not outcomes)
* micro_reward_scale (OFI-momentum signal follower)
* commitment_lambda (triple-counts churn)
* reward_noise_scale (belongs at gradient level, not reward level)
* position_entropy_weight (changes what "optimal" means)
- 4 dormant/dead terms to clean up (w_dsr, exit_timing, ofi_reward,
opp_cost_scale — all plumbing with no kernel effect)
Biggest offenders by magnitude: urgency_credit and risk_efficiency can
dominate the core signal on a single positioned bar.
Phase 2 (unified_env_kernel implementation) can now proceed with a
concrete deletion list, not a judgment call.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Design document for the follow-up to the adaptive-learning-rootcause
session. Lays out the evidence, root cause, options considered, and
recommended path for closing the 70% structural gap between training
and validation Sharpe that remained after the distillation collapse
fix landed.
Key findings documented:
- Reward-shaping ablation (2026-04-20) closed ~30% of the gap; ~70%
remains architectural
- Two env kernels (experience_env_step vs backtest_env_step) have
drifted: spread scaling, fill model, saboteur noise, reward terms,
action selection, position dynamics all differ
- Hint from history: experience_kernels.cu:1418 comment "Regime-
adaptive scaling removed to eliminate train/eval mismatch" shows
someone aligned *some* things previously
Recommended path (Option C, "unified env with layered reward"):
- Single unified_env_step kernel replaces both
- Core reward = pure P&L; shaping is additive and P&L-units-aligned
- Validation = training with exploration_scale=0 AND shaping_scale=0
- Scale factors are pinned device-mapped scalars (same pattern used by
the distillation alpha fix)
Phased implementation plan with ~8-day budget and concrete success
criteria: validation Sharpe_raw within 0.05 of training Sharpe_raw by
epoch 30 on L40S production run.
Rejected alternatives: backtest-matches-training (hides real issue),
training-matches-backtest (regresses stability), two-environment with
divergence as metric (fallback only).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Layer 2 labeled as G2-G5, G5 explicit for gradient budget
- P2 spectral_gap_norm added as 7th component in composition (rebalanced weights)
- N6 ensemble oracle has explicit threshold (>0.8 triggers N3 immediately)
- N8 meta-Q wiring clarified: logged but not in composition until validated
- Files Changed deduplicated, paths qualified
- Success criteria: WinRate >55% (above random baseline ~25%)
- HEALTH_DIAG log line includes all 7 components
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fixes Q-value collapse via unified LearningHealth signal that senses
training health (6 components) and continuously adapts:
- CQL regularization (regime + health gated)
- Gradient budget (IQN/CQL/Ens/C51 dynamic allocation)
- Tau target EMA (health-coupled)
- Expected SARSA temperature (continuous, no hardcoded threshold)
Plus 4 pearls (PER priorities, spectral detection, gradient consistency,
adaptive gamma) and 8 novels (self-distillation, barrier loss, plasticity
injection, CF curriculum, information bottleneck, ensemble oracle,
contrarian override, meta-Q network).
Core principle: training hyperparameters are OUTPUTS of the temporal
pipeline, not static schedules. The system meta-learns its own settings.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Training and validation use different kernels with different state layouts,
making generalization impossible. Single state_layout.cuh + one assembly
function + STATE_DIM as compile-time constant.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Removed the legacy non-branching Sequential q_network and target_network
from DQNAgent. These were never used (branching+dueling always active)
but allocated VRAM and ran noise resets every step. -190 lines.
Config tuning for dense micro-reward system:
- n_steps: 5→1 (TD(0), micro-rewards cancel over n>1)
- tau: 0.007→0.01 (faster target tracking for TD(0))
- c51_alpha_max: 0.5→1.0 (full C51, PopArt handles normalization)
- curiosity_weight: 0.1→0.0 (dense micro-reward replaces curiosity)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
target_dim expansion: adds raw_open (OHLCV) and mid_price_open
(MBP-10 midpoint at bar formation) to fxcache targets. FXCACHE_VERSION
2→3 for auto-rebuild. Legacy v2 files handled with close-price fallback.
Spec v5 adds 3 pearls:
- Bar duration encoding in Mamba2 (continuous-time SSM awareness)
- Order book center of mass from all 10 MBP-10 levels (aggression signal)
- Retrospective hold quality bonus (teaches exit timing)
Plus: Hold action (4th direction), DSR Sharpe EMA fix, counterfactual
magnitude/order sign fix, MFT mid-price mark-to-market.
OFI embed MLP now 18→10 (was 16→8). Mamba2 width SH2+10 (was SH2+8).
Attention width D+10 (was D+8).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
target_dim 4→6: adds raw_open (OHLCV) and mid_price_open (MBP-10
best bid/ask midpoint at bar formation) to the targets buffer.
Micro-reward now uses real intra-bar move (close - open) / atr
instead of close-to-close approximation. Also adds spread cost
awareness via |open - mid| penalty. Requires fxcache recomputation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Major changes from v1:
- Fix OFI data pipeline (ofi_gpu never wired, indices 42→66)
- Expand PORTFOLIO_STRIDE 30→38 for prev-OFI storage
- Add Mamba2 d_h_history backward (2 new cuBLAS GEMMs)
- Expose attention d_input_scratch for gradient flow
- Pre-compute OFI deltas in experience collection (state[74..82])
- Lower rank normalization threshold to 1e-5 for micro-rewards
- Use close-to-close return instead of unavailable open_price
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Dense per-bar reward using order flow momentum × price confirmation.
OFI embedding MLP (16→8 via cuBLAS) feeds into Mamba2 history AND
attention input. Replaces sparse exit-only reward with continuous
temporal signal for the SSM and attention heads to learn from.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Single cuGraphExecLaunch per step. PER sampling, gather, training,
priority update ALL as child graph nodes in one parent. Direct-to-trainer
gather eliminates DtoD copies. GPU-side counters eliminate host writes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: CUfunction sharing between graphed children and ungraphed
outside-graph ops corrupts kernel state on Hopper → 3100ms adam replay.
Fix: capture EVERYTHING unconditionally in child graphs. Selectivity,
denoise, causal intervention, vaccine, Q-stats all become graphed children.
Zero ungraphed launches = zero CUfunction conflicts.
7 children, ~190 kernel nodes, one capture, one replay per step.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 1 results: 4.0s/step GPU compute (graph replay = ungraphed, cuBLAS
replaced manual matmul but feature count increased). CPU pipeline fully
async (0.1ms/step). Total ~718s/epoch — 100% GPU bound.
Updated Phase 2 targets: aux parallelism (2b) is highest priority —
80/177 kernels with 4 independent trainers. nsys profiling should run
first to validate SM vs memory-bound assumptions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Inventory penalty makes flat optimal — model learns to do nothing.
Instead: penalize inaction proportional to model's own predicted edge
(Q-value spread). Creates virtuous cycle: better temporal attention →
higher self-imposed penalty for missed trades → more trading on signal.
No hindsight bias (uses predicted edge, not actual price change).
Micro-reward (already exists) rewards correct positioning.
Churn penalty (new) prevents rapid flips.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
forward_child: 4 branch streams fork after h_s2, join before loss
aux_child: IQL/IQN/attention on parallel streams
Uses CU_STREAM_CAPTURE_MODE_RELAXED with fork-join events as graph edges
Existing branch_streams[4] + events in batched_forward.rs ready to activate
Target: <40s epochs (from <80s Phase 1)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Portfolio slots ps[15]-ps[19] are Kelly accumulators (active).
Plan params moved to ps[23]-ps[29]. PORTFOLIO_STRIDE grows 23→30.
Fixed asymmetry: scales profit_target only (not stop_loss).
Clarified epsilon mid-plan: overridden by direction lock (intended).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Components were fighting a broken signal, not broken themselves.
With correct signal: adapt DSR to trade-level, rank only non-zero
rewards, keep commitment as soft signal, let E1 enrichment handle
Q-drift instead of hardcoded kernel penalty.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: training environment differs from validation backtest in 3 ways:
1. 100-bar episodes force exits (val runs continuous) → position-gated done
2. Rank normalization destroys sparse trade-level reward → remove it
3. Different Sharpe computation (per-trade vs per-bar, different annualization)
Single execution path:
Phase 1: Episode alignment (100→5000 bars, position-gated done, soft reset)
Phase 2: Reward alignment (remove rank norm, raw trade P&L to replay)
Phase 3: Metrics alignment (un-annualized per-trade Sharpe, both paths)
Expected: training Sharpe within 2× of validation with same weights.
Removes 5 unnecessary shaping components that fought the broken signal.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Every epoch's validation backtest feeds back into the next epoch:
E1: Q-value reality check (bias correction, replaces drift penalty)
E2: Adaptive epsilon (performance-driven, replaces schedule)
E3: Dynamic gamma (from trade duration, replaces manual annealing)
E4: Trade autopsy (per-branch LR scaling from error rates)
E5: Ensemble agreement tuning (auto-tune epistemic gate)
E6: Winner distillation (boost top 10% trades in replay)
E7: Hindsight optimal labels (correct actions for losers)
E8: Curriculum weights (oversample failure regimes)
E9: State confidence (tradability scores from eval)
Zero hardcoded schedules. The model adapts from its own performance.
~215 lines Rust, no new CUDA kernels. ~15s overhead per epoch.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two fundamental fixes for training Sharpe breakthrough:
1. Trade-level rewards: replace per-bar noise (SNR=0.01) with trade
P&L attribution (SNR=0.1-0.5). C51 atoms model trade outcome
distributions, not random walk noise.
2. Exploration risk budget: protection stack (CVaR, epistemic gate,
commitment, DSR) scaled by iqn_readiness². Loose during exploration,
tight when converged. Model can discover edges before being punished.
Also: homeostatic regularization spec (unified adaptive penalties).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Primary goal: val/OOS Sharpe gap < 15%. If val_Sharpe drops to 25
post-generalization, OOS should be > 21. If val drops to 15, OOS > 13.
OOS Sharpe > 10 sustained as secondary target.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Walk-forward: strictly chronological, no future leakage.
Weight decay mask: indices 0-7 (trunk + value head), not just trunk.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Observed train-vpb4w freeze at epoch 23 (val_Sharpe=51.11):
- avg_grad=10.0 (nonzero!) but Q-stats FROZEN for 4+ epochs
- Root cause: Adam momentum converged to fixed point, not zero gradient
Component 10: Targeted Adam momentum reset for branch weights only
(indices 8-41). Lighter than full rewind, preserves trunk convergence.
Component 11: Q-gap target attractor for spread gradient. Amplifies
spread when Q-gap is below target, reduces when above. Prevents
spread from being too weak at high Q-gap levels.
Component 12: Cosine LR schedule with warm restarts (T=20 epochs).
LR decay prevents overshoot, restart jumps break fixed points.
Half-momentum at restart preserves gradient direction.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Observed in train-vpb4w (live H100 run):
- Q-mean drifts 0→+0.47 over 18 epochs (bootstrapping bias)
- Atom utilization drops 100%→20% (distributional collapse)
- val_Sharpe peaks 54.59 then decays to 31.52
Component 8: Mean-center Q-values before action selection pipeline.
Zero params, preserves ranking, prevents drift accumulation.
Component 9: Adaptive atom entropy regularization. Ramps entropy_coeff
when utilization drops below 50%. Uses existing c51_grad_kernel param.
Both are zero-param fixes moved to top of implementation order.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
