Compares the 51-feature TLOBFeatureExtractor (crates/ml-supervised/src/tlob/
features.rs + mbp10_feature_extractor.rs) against Foxhunt's 20-slot OFI
vector persisted in .fxcache.
Findings: of TLOB's 51 slots, 23 are placeholders (sine waves / hardcoded
constants), 12 duplicate existing OFI or 42-dim market features, and only
~10 carry genuinely novel information. Meanwhile the existing
MicrostructureState struct in ml-features already computes 10 features
(realized variance, Hawkes intensity, weighted book pressure, spread
dynamics, aggression ratio, queue-depletion asymmetry, order-count flux,
intra-bar momentum, regime score, OFI trajectory) that are dropped before
reaching fxcache — these dominate the TLOB novel candidates and require
zero new math.
Phase B plan (drafted inline): bump OFI_DIM 20→28, bump FXCACHE_VERSION
4→5, prioritize persisting the 10 Foxhunt-internal MicrostructureState
features before any TLOB-derived additions, sweep all hard-coded 18/20
constants across CUDA kernels + gpu_dqn_trainer.rs, and rely on the
ensure-fxcache Argo step for cache regeneration.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extends grad_decomp_kernel to snapshot the trunk tensor slice (tensors
0..4 = w_s1, b_s1, w_s2, b_s2) in addition to the existing direction +
magnitude branch slices (8..12 / 12..16). Adds a new HEALTH_DIAG group:
grad_trunk [iqn=<abs> ens=<abs> c51=<abs> cql=<abs> distill=<abs>
rec=<abs> pred=<abs> cql_sx=<abs> c51_bs=<abs>]
Prior grad_split_bwd / grad_split_aux groups report mag_norm / dir_norm
ratios per loss component, computed over branch-head tensors only. That
measurement range structurally reports 0.0000 for any loss component
that writes exclusively to the trunk — IQN and Ens in particular. This
caused the persistent misdiagnosis that IQN-to-trunk was not wired; the
prior scoping in /tmp/foxhunt_research/iqn-to-trunk-wiring-scoping.md
confirmed the wiring is live (apply_iqn_trunk_gradient at
gpu_dqn_trainer.rs:4882) and that the zero reading was a blind spot in
the measurement pipeline.
Smoke confirms the diagnostic: after iqn_readiness ramps up (late
epochs), grad_trunk reports iqn=100..381 (real trunk SAXPY amplitude),
ens=0.07..3.57, c51=2.46..8.91 (value-head dueling path contributes
through trunk), while cql/cql_sx/distill/rec/pred stay near-zero — a
clean diagnostic baseline.
Also fixes stale documentation at dual-distributional-c51-iqn-design.md
that claimed "IQN trains in isolation — its gradients don't flow back
to the shared trunk": reworded to reflect current wired state with
file:function citation and explicit iqn_readiness gating note. Updated
the "What Changes" table ("IQN training") and "Implementation Order"
(Phase 1 marked DONE) with the same citation.
Changes:
- grad_decomp_kernel.cu: per-component result slot 2 → 3 floats
(mag_norm, dir_norm, trunk_norm); extra __shared__ sum_trunk +
tree reduction; new grad_trunk_start/trunk_len kernel args.
- gpu_dqn_trainer.rs: pinned result buffer 18 → 27 floats; snapshot
now does two copy_f32 passes (trunk → dst[0..trunk_len), branch →
dst[trunk_len..]); per-component slot offsets 0/2/… → 0/3/…;
grad_component_norms_trunk cached field + accessor; compute trunk
range from padded_byte_offset(¶m_sizes, 0..4).
- fused_training.rs: grad_trunk_norms_by_component() + per-component
grad_trunk_*_abs() accessors.
- training_loop.rs: HEALTH_DIAG emits new grad_trunk group ordered
[iqn ens c51 cql distill rec pred cql_sx c51_bs]; extended doc
comment explaining the three groups' roles.
- design spec (Problem #1 + What Changes row + Implementation Order):
stale "IQN trains in isolation" replaced by current wired-state
description, cites gpu_dqn_trainer.rs:4882 and readiness ramp at
gpu_dqn_trainer.rs:4228-4243.
Pure diagnostic — no training dynamics change, no atomicAdd, no tuning
knobs, no TF32 changes. Smokes unaffected (magnitude_distribution H10
regression pre-exists on HEAD 810b3c570; 4 other smokes pass).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Task 2.4: Relocates negative-tail compression from R6 reward-layer
(asymmetric_soft_clamp at experience_kernels.cu:78-81) to C51 Bellman
target smoothing (c51_loss_kernel.cu::block_bellman_project_f).
Functionality preserved — same invariant, better location. Upper +10
cap kept inline as fminf(reward, 10.0f) for numerical safety.
Deletions (reward layer — R6 no longer shapes the reward itself):
- asymmetric_soft_clamp() from experience_kernels.cu:78-81 (no callers)
- Reward-layer clamp replaced with fminf(reward, 10.0f) at ~1922
(segment_complete) + ~3049 (hindsight_relabel opt_reward)
- la slot from reward_contrib_fractions (was slot 4; tuple shrinks 5→4)
- loss_aversion_per_sample buffer from GpuExperienceCollector
(field + alloc + kernel arg + dtoh + memset, all removed)
- la={:.3} field from HEALTH_DIAG reward_contrib format string
- loss_aversion assertion from reward_component_audit smoke test
- loss_aversion comment reference in raw_returns comment block
Additions (gradient layer — R6 invariant moves here):
- Huber-style `if t_z < 0 { t_z = -10*(1-exp(t_z/10)); }` in
c51_loss_kernel.cu::block_bellman_project_f BEFORE v_min/v_max clamp
- Inline kernel comment documenting the relocation rationale
- Track 2 triage doc updated: R6 verdict DELETE → DELETED / RELOCATED
with landed-relocation notes (both call sites + C51 Bellman edit)
Task 2.5 Bug #6: Stale `patience_mult` docstring at
experience_kernels.cu:1144 referenced the defunct R7 V8 reward (deleted
in Task 0.8). Rewrote the reward-shape docstring to reflect current
post-V7 / Task 0.8 reality (sparse = 2.0 * vol_normalized_return, capped
inline) and notes the R6 relocation. Per feedback_trust_code_not_docs.md.
Per feedback_no_functionality_removal.md: R6's invariant is RELOCATED,
not deleted. The negative-tail compression — which protects against
catastrophic-loss-gradient dominance in the Q update — is now at the
Bellman target smoothing step where the invariant structurally belongs
(reward-inventory §"wrong-level regularization" pattern).
Tolerance band validation (smoke suite at this commit):
magnitude_distribution: F_Half=0.150 F_Full=0.237 (≥0.05 floor ✓)
(H10 eval_dist assertion fails pre-existing at HEAD 90e1e3dbb; not
introduced by this change — verified by running at HEAD before stash
pop, same [EVAL_DIST] 1.000/0.000/0.000 collapse.)
reward_component_audit: cf_flip=0.584 trail=0.304 (cf_flip≥0.1 ✓, PASS)
controller_activity: [CTRL_FIRE] anti_lr=0.000 tau=0.000 gamma=0.000
clip=0.400 cql=0.000 cost=0.000 (PASS)
exploration_coverage: entropy @ep5=0.988 @ep20=0.985 (PASS)
multi_fold_convergence: Best Sharpe 81.54/38.82/84.18 (≥20 floor ✓)
best_val_metric 0.043/0.024/0.049 (baseline was 0.028/0.018/0.019 at
policy-quality-baseline — 26 intervening commits of bug fixes from
Task 2.5 bugs #1–#7 would account for persistent drift; within
run-to-run variance of HEAD-pre-change)
Scoping output from the Full=0% investigation (post-Task-2.2 smoke showed
eval_dist [eq=0.580 eh=0.420 ef=0.000] — Quarter+Half recovered,
Full still 0%).
Root cause: Q-estimation bias (category C) — C51 expected-Q over atoms
systematically under-prices higher-variance bins. Full has 4x Quarter's
PnL variance per experience_kernels.cu risk scaling. Bias is already
documented at experience_kernels.cu:904 (commented as "structural
distributional bias — tight return distributions (Small) get higher
expected Q under the C51 softmax regardless of actual expected returns").
Not noise-starvation: Full picked on 32% of training samples across
60 epochs × 3 folds = ~61k Full samples. Ample data.
Proposed Task 2.X = Rank 1 + Rank 2 combined (~90 LOC):
Rank 1: per-bin variance weighting in C51 branch loss — amplify
Full's gradient by sqrt(var_f / mean_var)
Rank 2: Q-spread regularization — lambda * ReLU(target_spread -
Q_spread)^2 penalty preventing degenerate fixpoint
Alternative fixes ranked but not recommended:
Rank 3 per-magnitude reward shaping (~80 LOC) — only if data-disfavor
is confirmed at L40S
Rank 4 magnitude curriculum (~20 LOC) — rejected (noise-starvation ruled out)
Rank 5 state-vector enrichment (~150 LOC) — premature
Prerequisite: ~40 LOC of per-magnitude win-rate + realized-variance
instrumentation to confirm category C vs A at L40S scale.
L40S gating recommendation was:
ef >= 0.15 at L40S → Task 2.X not needed (smoke artefact)
0.05 <= ef < 0.15 → Task 2.X scoped but optional
ef < 0.05 → Task 2.X required
Per feedback_fix_aggressively.md (landed this session): we ship Task 2.X
ahead of L40S since the bias is already code-documented, the fix is
well-scoped, and the 90-LOC cost is reasonable. L40S validation (Task
2.8) will verify the fixed state end-to-end instead of the un-fixed one.
No deletions anywhere per feedback_no_functionality_removal.md.
Per feedback_no_functionality_removal.md: R5 was originally scoped as
DELETE in the Phase 2 plan and the Track 2 triage because
reward_contrib[3] = 0.000 across 60 / 60 smoke epochs and
dqn-smoketest.toml sets micro_reward_scale = 0.0. Re-examination during
Phase 2 Task 2.3 rejected the DELETE path:
- dqn-production.toml already sets micro_reward_scale = 0.1, so R5 is
load-bearing in production, not dead code. The 0.0 value in smoke is
deliberate test-isolation (td_propagation / magnitude_distribution /
reward_component_audit all want the sparse-reward TD path isolated).
- The state-vector OFI block at state[SL_OFI_START..SL_OFI_START+SL_OFI_DIM)
= [42..62) provides representation features for the encoder (policy
side). R5 is a per-bar reward gradient on the critic (critic side).
Different mechanisms — production deploys both together.
- R5 also reads PREV_MID (retrospective hold quality), which is NOT in
the state vector. That signal exists only in the kernel branch.
Changes — pure documentation, no behavior change:
- experience_kernels.cu: ~30-line comment block at the R5 wiring site
(~L1915) documenting the parameter-not-flag status, production vs
smoke values, why state-vector OFI is complementary not redundant,
and the feedback_no_functionality_removal.md seal.
- experience_kernels.cu: fix stale kernel-signature comment that claimed
OFI was at state[66..74). Correct range is [42..62) per state_layout.cuh.
- config.rs: extend DQNHyperparameters::micro_reward_scale docstring and
add a comment at the Default impl pointing back at the kernel site.
- gpu_experience_collector.rs: extend reward_contrib_fractions docstring
to mark the micro=0.000 slot as a SEMANTIC value when the loaded profile
has micro_reward_scale=0.0, not a wiring regression.
- track2-triage.md: R5 verdict changed from DELETE to FIX-documented-disable
with the rationale above; "Proposed Phase 2 changes" section 1 and
"Next Track 2 steps" updated accordingly.
Smoke tests: 3 / 4 pass (reward_component_audit, controller_activity,
exploration_coverage). magnitude_distribution is failing on baseline
HEAD c0fee5a9b as well — pre-existing H10 magnitude-head collapse
regression (eval dist_mag ef=0.000, eh=0.000), unrelated to R5.
Verified via git-stash round-trip: baseline fails, changes do not
introduce new failures.
Closes Track 2 R5 verdict (originally DELETE, now FIX-documented-disable).
V7 audit of the 7 adaptive controllers named in spec §5.3 against the
baseline controller_activity smoke run (3 folds × 20 epochs = 60 epochs,
intervention-based fire detection per commit ed4b30b49):
* C1 anti_lr: DIAGNOSTIC (0/60; wiring surprise — fire detector reads
base scheduler, not post-anti_lr LR, flagged for Phase 2 fix)
* C2 adaptive tau: DIAGNOSTIC (2/60; fires at fold boundaries only)
* C3 adaptive gamma: DIAGNOSTIC (1/60; pinned to floor by health-coupled
correction; re-measure at L40S when health is real)
* C4 adaptive grad_clip: CANDIDATE FOR DELETE pending ablation (12/60,
20% — above diagnostic, below load-bearing; needs Track 3 Step 2)
* C5 cql_alpha: DIAGNOSTIC (2/60; regime_stability gate uninformative
at smoke scale)
* C6 cost_anneal: DIAGNOSTIC by design (deterministic sigmoid; fire_cost
hard-coded false at training_loop.rs:2475)
* C7 base LR scheduler: DIAGNOSTIC by construction (pure open-loop, not
in the closed-loop controller battery)
Cross-controller finding: no controller fires in > 50% of epochs, so
the policy is not currently held on the rails by adaptive intervention
at smoke scale. However, C2 / C3 / C5 are structurally inert here —
their trigger signals (health, regime_stability) are degenerate at
smoke scale, so their 0-ish fire rates are lower bounds, not
representative. Promote to final after L40S validation.
Wiring surprise (anti_lr): fire_lr detection reads lr_scheduler.get_lr()
*before* the anti_lr multiplier applies. Under smoke Constant LR this
reports 0 correctly-for-the-wrong-reason; under L40S Cosine/Linear it
will false-positive on pure decay drift. Recommend detecting via the
anti_mult != 1.0 branch directly (training_loop.rs:2936–2942) for an
unambiguous intervention semantic matching C4/C6.
Follow-ups flagged for Phase 2:
- Fix C1 fire-detection wiring before L40S re-run
- Run C4 grad-clip ablation (25 min, gates the final C4 verdict)
- Optional: reset fire_counts per fold to remove C2/C5 boundary artefact
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Preliminary triage of spec §5.1 hypotheses H1–H10 using the Phase 0
baseline capture on RTX 3050 Ti. L40S validation pending per plan.
Verdicts:
H1 PENDING (needs forced-exploration instrumentation not yet wired)
H2 REJECTED var_scale=0.96 across 19/20 epochs; Var[Q] inactive at smoke scale
H3 INCONCLUSIVE kelly degenerate (insufficient win/loss counts at smoke scale)
H4 CONFIRMED grad_ratio_mag_dir=0.0000 across 20/20 epochs (threshold <0.1)
H5 REJECTED ent_mag stays ≥0.98 throughout; no bootstrap collapse
H6 REJECTED Full fire rate (0) is lower than Quarter fire rate, not higher
H7 REJECTED vsn and sigma symmetric between mag and dir branches
H8 REJECTED target-net drift equal (mag=dir=0.001)
H9 PENDING (same instrumentation gap as H1)
H10 CONFIRMED training ent_mag=0.98, eval F_Quarter=100%
Synthesis: H4 is the root cause. Magnitude branch receives ~0 gradient →
weights stay near init → three magnitude Q-values near-identical → argmax
picks bin 0 (Quarter) on ties → H10 manifests at eval time. H2, H5, H7,
H8 all ruled out as contributors.
Proposed Phase 2 priority: fix H4 (gradient-flow path into magnitude head
— likely per-component advantage weighting or direction-conditioning of
w_b1fc) + H10 (Q-margin argmax + stochastic eval rollouts as safety net).
Phase 1 next: validate preliminary verdicts on L40S, instrument
per-component gradient decomposition for magnitude, proceed with
Tracks 2/3/4 in parallel.
All <PENDING> markers replaced with real values from a clean 20-epoch
magnitude_distribution smoke run + 3-fold multi_fold_convergence run
on RTX 3050 Ti at HEAD 0472b9730.
All 5 Phase 0 smoke tests PASS:
magnitude_distribution — Quarter=0.596 Half=0.150 Full=0.255
reward_component_audit — cf_flip=0.619 trail=0.290 la=0.013
controller_activity — all 6 controllers < 20% firing
exploration_coverage — ent_mag @ep5=0.987 @ep20=0.985
multi_fold_convergence — 3/3 folds, Best Sharpe 51 / 39 / 87
This commit closes Phase 0. The next commit tags it as
policy-quality-baseline — the reference state against which Phase 2
interventions are measured.
Phase 0 instrumentation is complete (Tasks 0.4 / 0.5 / 0.8 / 0.10 / 0.16
shipped this session; 0.6 partial; 0.3 partial). This scaffold encodes
the full metric set with explicit <PENDING …> markers per row so the
GPU-run capture can fill in values without forgetting any HEALTH_DIAG
field.
The policy-quality-baseline tag is intentionally NOT applied yet —
tagging requires real captured values, not the scaffold. Runner workflow:
for t in magnitude_distribution reward_component_audit \\
controller_activity exploration_coverage multi_fold_convergence; do
SQLX_OFFLINE=true CUBLAS_WORKSPACE_CONFIG=:4096:8 \\
FOXHUNT_TEST_DATA=test_data/futures-baseline \\
cargo test -p ml --release --lib -- \$t --ignored --nocapture 2>&1 | tee out_\$t.log
done
Then edit this doc, replace markers with values, amend, and tag.
User preference: everything lands on main, ~500 LOC is small enough
that feature-branch isolation isn't needed.
Spec changes:
* §2.3 outcome paths: drop "unmerged branch" vocabulary
* §3 architecture: main-branch timeline, no feat/, no wip/
* §4 Phase 0: commits on main, tag policy-quality-baseline at end
* §5 Phase 1: experiments are worktree edits reverted after, only
triage docs commit (to main)
* §6 Phase 2: commits land on main, author chooses granularity,
each leaves smokes green
* §7.3 outcome handling: iterate on main, baseline tag for rollback
* §7.4 closing: tag policy-quality-v1, update status
Plan changes:
* Task 0.1: verify-starting-state (not branch creation)
* Task 1.x: temp-edit-then-revert (not wip branches)
* Task 1.5: triage docs commit directly to main
* Phase 4: tag + close (not merge + cleanup)
* Rollback: one git reset --hard command
Addressed functional concerns from third review:
1. Surrogate-noise gate was statistically weak — "random Sharpe ≤ 0.5×
trained Sharpe on same val data" could be satisfied by sample
noise. Replaced with percentile-based test:
* Pool all 6 val folds (~600 trades) for statistical power.
* Run N=30 surrogate random-action rollouts (matched marginal
action distribution — so difference is state-action mapping,
not action frequency).
* Assert trained pooled Sharpe > 95th-percentile of surrogate
distribution. Explicit false-positive control.
2. Escalation path ("scope exceeds sub-project A v2") was hand-wavy.
Now concrete: failure after 3 iterations → paused, not closed →
triage spec opened with evidence summary + diagnosis (policy/
reward/state/architecture/data bottleneck) + explicit decision
(continue on feat/policy-quality or fork feat/policy-quality-v2).
Triage itself is a full brainstorming cycle.
3. HEALTH_DIAG §4.2 Track 1 fields only covered H1–H5 — H6–H10
needed their own detection signals. Added:
- trail_fire_rate_{quarter,half,full} (H6)
- hold_time_at_exit_{quarter,half,full} (H6)
- vsn_mask_{magnitude,direction} (H7)
- noisy_sigma_{mag_head,dir_head} (H7)
- target_drift_{mag_head,dir_head} (H8)
- action_dist_eval_{quarter,half,full} (H10)
Fixed inconsistencies between sections after the rev-2 feature-branch
refactor (sections had gotten out of sync):
1. §2 (success criteria) rewritten — was stale from rev 1. Now has
clean mandatory/soft split matching §7.2, corrects the trade-count
gate (per-fold not averaged), references the surrogate-noise gate.
New §2.3 enumerates the outcome paths.
2. §4 and §6 no longer say "commit on main" — both land on
feat/policy-quality per the feature-branch architecture. Phase 4
merges to main at §7.4.
3. §7.3 outcome handling rewritten — was written for single-branch
model ("Phase 2 commit is NOT reverted"). New wording matches
feature-branch semantics: failing mandatory = don't merge; failing
soft = merge + follow-up.
4. §6 smoke-tests rule now lists all six Phase 0 smokes, not just
Track 1/2.
5. §7.4 Phase-4 merge-strategy added — --no-ff merge commit, tag
policy-quality-v1, cleanup sequence documented.
6. §5.5 conflict-resolution added — tracks can reach contradictory
conclusions (e.g. T1 says fix, T2 says delete). Resolution rules:
mandatory-gate dominance, simplification wins, escalation.
7. §4.3 baseline metrics now committed to the feature branch (not
"investigation-only"), matching the crash-safety discipline.
8. Budget / risk register aligned — 5.5–6.5 hrs plan + 1 buffer,
hard-capped at 3 Phase-3 validation runs.
If H9 confirms and Phase 2 removes the magnitude branch, F_Half/F_Full
gates become vacuous. Substitute: direction bin distribution healthy
(F_Short + F_Long each >= 20% of non-Flat actions). Preserves the
'model actually trades' intent of Criterion A.
Addresses critique from second self-review:
1. SCOPE GAP fixed: spec now covers all 4 V7 categories the user asked
for. Added Track 3 (controllers) and Track 4 (exploration) with their
own smoke tests (controller_activity, exploration_coverage) and
HEALTH_DIAG fields. Previous version only had action-space + reward.
2. H1-H5 expanded to H1-H10 with 5 new codebase-specific hypotheses:
- H6: Regime-adaptive trailing stop closing Full-positions early
- H7: VSN / NoisyNets masking magnitude features
- H8: Target tau too slow for magnitude propagation
- H9: Data genuinely favors Quarter (no bug, delete magnitude branch)
- H10: Entropy regularization + argmax tie ordering bias
3. H3 REFRAMED from "Kelly bug" to design critique — Kelly as hard
multiplier vs soft gate. Fix sketch updated to offer replacement.
4. Feature branch + tag strategy replaces "single branch" containment.
feat/policy-quality isolates work from main until Phase 4 merge via
PR. git tag policy-quality-baseline gives named rollback anchor.
5. SURROGATE-NOISE CHECK added as MANDATORY validation gate — catches
look-ahead/leakage bugs that would otherwise produce false-positive
edges. Permanent smoke artifact.
6. Validation gate restructured: B + surrogate-noise MANDATORY;
A criteria soft; controller-activity soft. Partial success path
explicit per-gate.
7. Risk register expanded — added scenarios for B-mandatory failure
(feat branch stays unmerged), all-10-hypotheses-REJECTED (fall back
to H9 delete-branch), surrogate-noise failure (halt spec, open
diagnostic spec), multi-load-bearing-controllers (tech debt doc).
8. Budget corrected to 5.5-6.5 hrs L40S (was underestimated 3.5-4),
with per-track breakdown.
Spec now: 464 lines, 10 sections, covers 4 V7 tracks × ~10 hypotheses
+ 8 reward terms + 7 controllers + 5 exploration mechanisms = ~30
audit items total.
Critical fixes after honest review:
1. reward-audit measurement: split into 3 classes (additive / transform
/ sample-selector) — one-size-fits-all metric couldn't work for
PopArt normalization or CF flip mirroring
2. H1 detection: replace undefined "Sharpe/state" with concrete
forced-exploration protocol (epsilon=1.0 for 1 epoch at mid-train,
sample states × magnitudes, compare Sharpe-per-trade)
3. H4 scope: clarify this is the per-action-branch gradient (direction
vs magnitude head), NOT the per-loss-component (IQN/CQL/C51/Ens)
budget. Fix sketch updated to target the correct knob.
4. Validation gate: criterion B (multi-fold Sharpe + WinRate) is now
MANDATORY; criterion A gates are soft. Prevents "partial success"
hiding a policy-quality failure.
5. Trade-count gate: per-fold ≥100 on ≥5/6, not an average — 5 lean
folds pooled with 1 fat one should not pass.
Minor:
- Budget trimmed to 3.5–4 hrs L40S (was 6–10)
- L40S pool made explicit as target
- Phase 1 branches now pushed as wip/* for crash safety
- Hyperopt-params mismatch handling documented
Sub-project A of the production-readiness roadmap. Single-branch,
phase-gated: measurement substrate → parallel investigation tracks
(magnitude collapse + reward V7 audit) → single convergence commit →
6-fold × 50-epoch L40S validation.
Target criteria at close:
A (action dist): F_Half/F_Full each ≥15% on ≥4/6 folds, WinRate
55-65%, ≥100 trades/val window
B (multi-fold): Best Sharpe > 10 on ≥5/6 folds, WinRate > 55% on
≥5/6 folds
Greenfield checkpoint policy — any action-space or weight-shape change
is allowed. All confirmed bugs fix together in one Phase-2 commit.
V7 discipline gates inclusion (measurement confirms the bug), no
ranking across findings.
Out of scope for this spec: paper trading (sub-project C), canary
deployment (sub-project D), hyperopt re-runs, architectural changes
beyond action space.
Awaiting user review before invoking writing-plans.
Completes Phase 3 of the env-unification effort: experience_env_step (training),
backtest_env_step (val single-step), and backtest_env_step_batch (val batched)
all call the same __device__ helper `unified_env_step_core` in trade_physics.cuh.
Drift between train/val becomes structurally impossible at compile time — any
change to the canonical step applies atomically to both.
Three structural fixes were required to make the training port work:
1) TWO max_position params in unified_env_step_core
Training scales `max_position` by Var[Q] (Kelly-from-atoms variance sizing)
→ `effective_max_pos`. The original training kernel used this for
compute_target_position_4branch but the UNSCALED `max_position` for
apply_kelly_cap and execute_trade. Initial port collapsed both into one
helper arg, tightening Kelly cap aggressively → tiny positions → Q-values
converge → q_gap=0.00 collapse. Split into `max_position_target` (scaled)
and `max_position_physics` (unscaled). Val passes the same value for both.
2) Remove double hold_time update
Helper now owns `update_hold_time(...)` inside the step. Training's
inline `hold_time = update_hold_time(...)` at line ~1574 was a double-
increment. Fix: capture `saved_hold_time` BEFORE helper for the
patience-multiplier in segment reward, then trust the helper's in-place
update of hold_time. Segment-hold-time semantics preserved.
3) Remove triple Kelly-stat update
Helper calls `record_kelly_trade_outcome(...)`. Training ALSO had Kelly
stat updates inside the reversing_trade block (line 1558) and the
exiting_trade block (line 1638). TRIPLE update → win_count/loss_count/
sum_wins/sum_losses inflated 3× → Kelly cap tightened proportionally →
positions starved → Q-gap collapse. Fix: only the helper now touches
Kelly win/loss/sum_wins/sum_losses. Training still updates its own
sum_returns / sum_sq_returns (continuous-Kelly stats, distinct
buffers not touched by the helper).
Smoke test result (RTX 3050 Ti, 20 epochs, single-trial):
Before port: Best Sharpe ~15-25 (varies, range 10-31)
After broken port: Best Sharpe 1.17 (q_gap=0.00 collapse)
After fix: Best Sharpe 39.25 (HIGHEST ever recorded)
q_gap 0.00 → 0.27 (growing, healthy separation)
sharpe_ema trajectory: -5.3 → 12.7 → 26.9 (rising)
Multi-trial (partial visible): q_gap rising 0.00 → 0.86 by epoch 6,
sharpe_ema 15-25 consistently positive. Both tests passed.
Training kernel now has the CORE STEP in a single helper call — the
kernel body downstream continues to handle training-only concerns
(counterfactuals, plan_params, reward shaping via shaping_scale,
saboteur via exploration_scale, replay buffer writes).
Net −99 LOC across three kernels replaced by shared helper calls.
Files touched:
crates/ml/src/cuda_pipeline/trade_physics.cuh (+15/-5)
crates/ml/src/cuda_pipeline/backtest_env_kernel.cu (+53/-108 net)
crates/ml/src/cuda_pipeline/experience_kernels.cu (+48/-102 net)
docs/superpowers/specs/2026-04-21-unified-train-val-env-design.md
(Phase 2 documented as helper-extraction; Phase 3 kernel-deletion
remains deferred — both kernel entry points kept for their distinct
threading models; the physics is what's now truly unified.)
Verified: cargo check + smoke test pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A) TD-propagation diagnostic smoke test (sparse rewards)
- New smoke_tests/td_propagation.rs captures training_sharpe_ema across
20 epochs with micro_reward_scale=0 to answer: "can sparse-reward
Q-learning extract policy from trade-completion P&L alone?"
- Asserts: finite sharpe_ema, no monotonic degradation (tolerance 0.1
over first-3 vs last-3 epoch avg), q_gap > 0.05
- First run answered YES: val Sharpe peaks at +12.49 at epoch 8 with
pure sparse rewards, confirming the objective is learnable. The
remaining issue is stability/overfitting, not TD propagation.
B) Entropy → plasticity gate in training_loop
- D3/N3 shrink_perturb trigger now fires on `last_action_entropy < 0.3`
OR `health_value < 0.3` (OR semantics, 3 consecutive epochs).
- Catches action-collapse cases the generic health metric misses: Q-values
separate cleanly but argmax stays pinned to a single branch.
- tracing::info now logs both signals.
C) commitment_lambda coverage verified
- Almgren-Chriss sqrt market-impact already in compute_tx_cost
(trade_physics.cuh:168-171). commitment_lambda was pure duplication.
- Inventory doc updated: P2 marked satisfied, table row status updated.
Files touched:
- crates/ml/src/trainers/dqn/smoke_tests/mod.rs (+2)
- crates/ml/src/trainers/dqn/smoke_tests/td_propagation.rs (new)
- crates/ml/src/trainers/dqn/trainer/training_loop.rs (+15/-3)
- docs/superpowers/specs/2026-04-21-phase1-reward-inventory.md (+6/-3)
Smoke test PASSED locally (RTX 3050 Ti, 30.99s end-to-end).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 2 P1 relocations — the intents behind two deleted reward shaping
terms now live at the correct layer: diagnostic monitoring, not reward.
1. training_sharpe_ema (was w_dsr reward input):
Field already existed. Added to HEALTH_DIAG log line as `sharpe_ema`.
Available for early-stopping, model selection, and exploration gating
without perturbing the objective.
2. action_entropy (was position_entropy_weight reward):
Computed from summary.action_counts at GPU summary download —
Shannon entropy normalized to [0, 1] by ln(n_bins). One-epoch lag
is fine for a diagnostic. Added to HEALTH_DIAG as `action_entropy`.
Stored in new trainer field `last_action_entropy: Option<f32>`.
New HEALTH_DIAG suffix: `diag [sharpe_ema={:.3} action_entropy={:.2}]`.
Verified visible on E1 smoke test epoch 18/19 output.
Also: honest recalibration of Phase 2 results added to the inventory
doc. The earlier commit messages framed "-150 → -17 Sharpe" as a 5×
improvement; in absolute terms Sharpe -17 is still catastrophic (you'd
blow the account). Phase 2 stopped active capital destruction but did
not find alpha. Sharpe_raw per bar went from -0.39 (lot of loss per bar)
to -0.09 (little loss per bar) — still net losing. q_gap=0.17 is a
capacity metric (network CAN differentiate), not profitability.
Path to actual profitability (Sharpe > 0) remains:
- TD propagation verification (Task #8)
- Unified env kernel (Phase 3)
- Possibly a different objective entirely
- Richer features (42 market + 20 OFI may be information-starved)
The rule going forward: before celebrating future improvements, ask
whether the result *crosses zero* (profitable) or is just *less
negative* (still losing).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mass deletion of the "v7 gem" reward terms identified in the Phase 1
inventory as category errors — each one rewarded an outcome-adjacent
behavior instead of encoding the underlying physics, and each
empirically hurt validation metrics more than it helped training
stability.
Deleted from experience_kernels.cu and all plumbing (Rust configs,
launch args, hyperopt logs, TOML entries):
* order_credit_weight - reward redundant with compute_tx_cost
(order_type_idx already differentiates
fills by order type)
* risk_efficiency_weight - reward double-counted drawdown penalty
asymmetrically (only on winners)
* urgency_credit_weight - reward was vol-normalized unrealized P&L,
pure rename of core return
* commitment_lambda - triple-counted churn + tx_cost
* w_dsr - kernel wrote DSR EMA but no longer added
to reward (dead); removed the EMA
bookkeeping too
* dsr_eta - kernel arg for the deleted DSR EMA
* position_entropy_weight - rewarded action-bucket diversity
regardless of outcome; histogram buffer
+ zero-init removed too
* exit_timing_weight - already inactive (used raw_next future
price, comment-deleted earlier)
* ofi_reward_weight - dead plumbing; OFI already passed as
feature through state[OFI_START..]
* opportunity_cost_scale - penalized flat when Q-gap wide;
redundant with Q-values themselves
Kernel arg count: experience_env_step_batch shrank from ~55 to ~45 args.
Rust-side config surface reduced correspondingly.
Results on E1 smoke test (20-epoch):
BEFORE any Phase 2 work:
Val Sharpe -120 to -150, MaxDD 10-15%, Sharpe_raw -0.39
AFTER reward_noise + Kelly (both envs) + urgency + this sweep:
Val Sharpe -17 to -22 (7× better)
Val MaxDD 0.27% (40× better)
Val Sharpe_raw ~-0.09 (4× better)
Training Sharpe_raw ~0 (stabilized from ±20 swings)
Final q_gap 0.1712 (highest yet, collapse mechanism fine)
The extreme train-Sharpe swings (+17 one epoch, -13 next) were not
learning dynamics — they were shaping-term noise. Core reward (P&L +
drawdown + churn + holding + tx_cost + Kelly physics cap) gives training
metrics that actually reflect what the model does.
Inventory doc (docs/superpowers/specs/2026-04-21-phase1-reward-inventory.md)
extended with a "better-form taxonomy" section: every deleted gem has
a correct layer it belongs to (physics, feature, diagnostic, gradient-
level regularization — not reward). Kelly cap and Q-target smoothing
are already relocated; others are scheduled per the taxonomy's P1/P2/P3
priority list.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extends the reward inventory with a review pass that preserves the
*ideas* behind behavioral terms while deleting the wrong-level mechanics.
Gems (relocate, don't delete):
- Probabilistic fill model (real physics, not shaping) → apply in both
modes via shared Philox RNG + exploration_scale
- Kelly-as-physics-constraint → hard position-size cap in trade_physics,
not a soft reward
- Path quality via per-step drawdown integration → no separate term,
just run drawdown penalty every bar
- Saboteur relocation → both modes, scaled by exploration_scale (training
1.0, validation 0.5, diagnostic 0.0) — avoids creating a new clean-vs-
noisy mismatch in the opposite direction
Pearls:
- Sparse reward is fine if TD propagation works — diagnose before
deleting micro_reward_scale; measure cov(Q(s_entry,a), trade_return)
- Every behavioral term maps to one of four failure modes: double-count,
misplaced physics, wrong-level regularization, or compensating for a
downstream bug. None are semantically "neutral" shaping.
Novel architectural moves:
- Diagnostic-only entropy tracking (no reward, just HEALTH_DIAG
logging)
- Single exploration_scale scalar replacing all mode toggles
(feedback_no_feature_flags compliant)
- Layered reward with per-term budget caps — makes the train/val Sharpe
gap computable instead of uncomputable magic
- Q-target smoothing replacing reward_noise_scale (regularization at
gradient level, not reward level)
Updated disposition table: 4 DELETE, 2 MOVE, 2 RELOCATE-to-physics,
1 DIAGNOSE-then-DELETE, 4 delete-dead-plumbing.
Three relocations (Kelly cap, saboteur scaling, Q-target smoothing) can
land as independent PRs before the unified_env_kernel rewrite, shrinking
Phase 2's change surface.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Enumerates every term that contributes to training or validation reward,
classifies each as P&L-aligned (keeper) or behavioral (remover), with
source line references into experience_kernels.cu and backtest_env_kernel.cu.
Findings:
- 7 P&L-aligned terms to preserve in the unified env (core return,
tx_cost, drawdown, inventory, churn, capital floor, + vol normalization)
- 8 active behavioral training-only terms to delete:
* order_credit_weight (rewards theoretical limit-order savings,
double-counts actual tx_cost)
* risk_efficiency_weight (double-counts drawdown asymmetrically)
* urgency_credit_weight (redundant with core return)
* kelly_sizing_weight (rewards matching a formula, not outcomes)
* micro_reward_scale (OFI-momentum signal follower)
* commitment_lambda (triple-counts churn)
* reward_noise_scale (belongs at gradient level, not reward level)
* position_entropy_weight (changes what "optimal" means)
- 4 dormant/dead terms to clean up (w_dsr, exit_timing, ofi_reward,
opp_cost_scale — all plumbing with no kernel effect)
Biggest offenders by magnitude: urgency_credit and risk_efficiency can
dominate the core signal on a single positioned bar.
Phase 2 (unified_env_kernel implementation) can now proceed with a
concrete deletion list, not a judgment call.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Design document for the follow-up to the adaptive-learning-rootcause
session. Lays out the evidence, root cause, options considered, and
recommended path for closing the 70% structural gap between training
and validation Sharpe that remained after the distillation collapse
fix landed.
Key findings documented:
- Reward-shaping ablation (2026-04-20) closed ~30% of the gap; ~70%
remains architectural
- Two env kernels (experience_env_step vs backtest_env_step) have
drifted: spread scaling, fill model, saboteur noise, reward terms,
action selection, position dynamics all differ
- Hint from history: experience_kernels.cu:1418 comment "Regime-
adaptive scaling removed to eliminate train/eval mismatch" shows
someone aligned *some* things previously
Recommended path (Option C, "unified env with layered reward"):
- Single unified_env_step kernel replaces both
- Core reward = pure P&L; shaping is additive and P&L-units-aligned
- Validation = training with exploration_scale=0 AND shaping_scale=0
- Scale factors are pinned device-mapped scalars (same pattern used by
the distillation alpha fix)
Phased implementation plan with ~8-day budget and concrete success
criteria: validation Sharpe_raw within 0.05 of training Sharpe_raw by
epoch 30 on L40S production run.
Rejected alternatives: backtest-matches-training (hides real issue),
training-matches-backtest (regresses stability), two-environment with
divergence as metric (fallback only).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Layer 2 labeled as G2-G5, G5 explicit for gradient budget
- P2 spectral_gap_norm added as 7th component in composition (rebalanced weights)
- N6 ensemble oracle has explicit threshold (>0.8 triggers N3 immediately)
- N8 meta-Q wiring clarified: logged but not in composition until validated
- Files Changed deduplicated, paths qualified
- Success criteria: WinRate >55% (above random baseline ~25%)
- HEALTH_DIAG log line includes all 7 components
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fixes Q-value collapse via unified LearningHealth signal that senses
training health (6 components) and continuously adapts:
- CQL regularization (regime + health gated)
- Gradient budget (IQN/CQL/Ens/C51 dynamic allocation)
- Tau target EMA (health-coupled)
- Expected SARSA temperature (continuous, no hardcoded threshold)
Plus 4 pearls (PER priorities, spectral detection, gradient consistency,
adaptive gamma) and 8 novels (self-distillation, barrier loss, plasticity
injection, CF curriculum, information bottleneck, ensemble oracle,
contrarian override, meta-Q network).
Core principle: training hyperparameters are OUTPUTS of the temporal
pipeline, not static schedules. The system meta-learns its own settings.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Training and validation use different kernels with different state layouts,
making generalization impossible. Single state_layout.cuh + one assembly
function + STATE_DIM as compile-time constant.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Removed the legacy non-branching Sequential q_network and target_network
from DQNAgent. These were never used (branching+dueling always active)
but allocated VRAM and ran noise resets every step. -190 lines.
Config tuning for dense micro-reward system:
- n_steps: 5→1 (TD(0), micro-rewards cancel over n>1)
- tau: 0.007→0.01 (faster target tracking for TD(0))
- c51_alpha_max: 0.5→1.0 (full C51, PopArt handles normalization)
- curiosity_weight: 0.1→0.0 (dense micro-reward replaces curiosity)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
target_dim expansion: adds raw_open (OHLCV) and mid_price_open
(MBP-10 midpoint at bar formation) to fxcache targets. FXCACHE_VERSION
2→3 for auto-rebuild. Legacy v2 files handled with close-price fallback.
Spec v5 adds 3 pearls:
- Bar duration encoding in Mamba2 (continuous-time SSM awareness)
- Order book center of mass from all 10 MBP-10 levels (aggression signal)
- Retrospective hold quality bonus (teaches exit timing)
Plus: Hold action (4th direction), DSR Sharpe EMA fix, counterfactual
magnitude/order sign fix, MFT mid-price mark-to-market.
OFI embed MLP now 18→10 (was 16→8). Mamba2 width SH2+10 (was SH2+8).
Attention width D+10 (was D+8).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
target_dim 4→6: adds raw_open (OHLCV) and mid_price_open (MBP-10
best bid/ask midpoint at bar formation) to the targets buffer.
Micro-reward now uses real intra-bar move (close - open) / atr
instead of close-to-close approximation. Also adds spread cost
awareness via |open - mid| penalty. Requires fxcache recomputation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Major changes from v1:
- Fix OFI data pipeline (ofi_gpu never wired, indices 42→66)
- Expand PORTFOLIO_STRIDE 30→38 for prev-OFI storage
- Add Mamba2 d_h_history backward (2 new cuBLAS GEMMs)
- Expose attention d_input_scratch for gradient flow
- Pre-compute OFI deltas in experience collection (state[74..82])
- Lower rank normalization threshold to 1e-5 for micro-rewards
- Use close-to-close return instead of unavailable open_price
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Dense per-bar reward using order flow momentum × price confirmation.
OFI embedding MLP (16→8 via cuBLAS) feeds into Mamba2 history AND
attention input. Replaces sparse exit-only reward with continuous
temporal signal for the SSM and attention heads to learn from.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Single cuGraphExecLaunch per step. PER sampling, gather, training,
priority update ALL as child graph nodes in one parent. Direct-to-trainer
gather eliminates DtoD copies. GPU-side counters eliminate host writes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: CUfunction sharing between graphed children and ungraphed
outside-graph ops corrupts kernel state on Hopper → 3100ms adam replay.
Fix: capture EVERYTHING unconditionally in child graphs. Selectivity,
denoise, causal intervention, vaccine, Q-stats all become graphed children.
Zero ungraphed launches = zero CUfunction conflicts.
7 children, ~190 kernel nodes, one capture, one replay per step.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 1 results: 4.0s/step GPU compute (graph replay = ungraphed, cuBLAS
replaced manual matmul but feature count increased). CPU pipeline fully
async (0.1ms/step). Total ~718s/epoch — 100% GPU bound.
Updated Phase 2 targets: aux parallelism (2b) is highest priority —
80/177 kernels with 4 independent trainers. nsys profiling should run
first to validate SM vs memory-bound assumptions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Inventory penalty makes flat optimal — model learns to do nothing.
Instead: penalize inaction proportional to model's own predicted edge
(Q-value spread). Creates virtuous cycle: better temporal attention →
higher self-imposed penalty for missed trades → more trading on signal.
No hindsight bias (uses predicted edge, not actual price change).
Micro-reward (already exists) rewards correct positioning.
Churn penalty (new) prevents rapid flips.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
forward_child: 4 branch streams fork after h_s2, join before loss
aux_child: IQL/IQN/attention on parallel streams
Uses CU_STREAM_CAPTURE_MODE_RELAXED with fork-join events as graph edges
Existing branch_streams[4] + events in batched_forward.rs ready to activate
Target: <40s epochs (from <80s Phase 1)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Portfolio slots ps[15]-ps[19] are Kelly accumulators (active).
Plan params moved to ps[23]-ps[29]. PORTFOLIO_STRIDE grows 23→30.
Fixed asymmetry: scales profit_target only (not stop_loss).
Clarified epsilon mid-plan: overridden by direction lock (intended).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
