User correctly identified that CPU mirror function tests don't test
the production GPU code path. A bug shared between mirror and kernel
(translated identically wrong) would slip through. Mirror tests + a
single GPU bridge test were a weak compromise.
GPU-direct testing strategy:
- All Phase 0 kernel-correctness tests (0.A, 0.B, 0.C, 0.D, 0.F):
launch tiny test-only kernels with the SAME math the Phase 2
production kernel will use; assert properties of the output.
- Test 0.E (synthetic edge discovery): stays CPU. It tests an
ALGORITHMIC PROPERTY of Thompson exploration (does it discover
edge if edge exists?), not a kernel correctness property.
- All Phase 2 unit tests (2.A-2.D): GPU-direct against the
modified production kernel.
- Phase 0.F (real checkpoint extraction): unchanged — already GPU.
Local development uses RTX 3050 GPU (per memory user_dev_environment.md).
CI runs --ignored flag to skip GPU tests on CPU-only runners.
Time budget: Phase 0 was 1-2 days (CPU mirror); now 2-3 days
(GPU-direct, includes kernel wrapper setup half-day).
Other delta:
- Phase 0 deliverable file renamed: distributional_q.rs ->
distributional_q_tests.rs (no mirror functions, just tests +
kernel wrappers).
- Phase 2 unit tests rephrased to launch production kernel rather
than compare against CPU mirror.
- L1 verification gate runtime: seconds -> minutes (GPU launch
overhead per test).
The user's intuition was right: testing production directly is the
honest approach. Mirror was an optimization that traded correctness
for speed; with local GPU available the optimization isn't needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Self-review identified 8 major + 8 medium + 15 minor issues. All fixed:
MAJORS:
M1 Test 0.A: clarified — compare argmax(E[Q]), Boltzmann(E[Q]),
and Thompson sampling distributions; assertion is on relative
ordering across all three.
M2 IQN quantile count: replaced hardcoded `5` with N_IQN_QUANTILES
constant (defined per task #147 fixed-quantile design).
M3 "Converged checkpoint" definition: ≥60 epochs trained AND
val_sharpe stabilised (no >10% change over last 10 epochs).
Cites prior 60-epoch validation runs (task #80, train-7rgqd).
M4 R3 reframed: replaced "no issue" handwave with explicit
by-design tradeoff acknowledgment + cost analysis. Wasted
exploration is the cost of finding out whether edge exists.
M5 Test 0.D σ_long=0.05 justified: chosen to match expected order
of magnitude given typical |return| ~ 50bps; Phase 0.F
validates against real checkpoint.
M6 rng_ctr post-increment: clarified — matches existing pattern
at experience_kernels.cu:858 (no behaviour change).
M7 train_active_frac instrumentation: NEW Phase 2 deliverable —
existing HEALTH_DIAG only has val_active_frac, but L3 verifies
training-time active_frac. Spec now explicitly adds this
~10-line metrics.rs change as a Phase 2 deliverable.
M8 eps_dir cleanup code-level detail: explicit reference to
experience_kernels.cu lines 814-865; remove eps_dir from both
static EPS_FLOOR clamp AND adaptive boost block; verify
variable can be removed from kernel signature via grep.
MEDIUMS:
Med1 Current C51/IQN combination: clarified that compute_expected_q
blends per training schedule; Phase 2 replaces with explicit
0.5*E_C51 + 0.5*E_IQN equal weighting; Phase 0.F verifies.
Med2 Eval mode phrasing: "eval mode already sets eps=0 in existing
kernel" — no semantic override, factually correct.
Med3 Magnitude σ claim: clarified — magnitude branch likely has σ
bias in OPPOSITE direction (Full has larger σ; UCB would
prefer Full and worsen saturation). Empirical verification
deferred. Phase 0.F should also report per-magnitude σ.
Med4 Hierarchical sampling claim corrected: it's not about
balancing 50/50 (already 50/50). It's about decoupling
cluster-best decisions; clarified.
Med5 n_atoms vs N_IQN_QUANTILES: clarified — n_atoms variable per
config (currently 51); N_IQN_QUANTILES fixed at 5.
Med6 Conviction code: removed pseudo-code; references existing
implementation at experience_kernels.cu:1091; provides
implementation hint for E[Q] reuse.
Med7 Q-target propagation: clarified — uses full distribution
(C51 atom projection / IQN quantile regression), not just
E[Q]. Thompson modifies action selection only.
Med8 References: added Thompson 1933 (original), Bellemare 2017
(C51), Dabney 2018 (IQN) for theoretical foundations.
MINORS:
Min1 Date updated to 2026-04-27.
Min2-3 Argmax monotonic /2 simplified out — argmax(a+b) =
argmax((a+b)/2). Code clarity improved.
Min4 P(argmax picks Long) = 0 deterministic; reframed assertion.
Min5 Test 0.F structural assertions added: σ_C51[FLAT] < 0.01 ×
σ_C51[LONG]; same for IQN; E[Q_FLAT] > E[Q_LONG]; argmax
picks FLAT; Thompson P(LONG)+P(SHORT) ≥ 0.20.
Min6 -INFINITY → CUDART_INF_F (CUDA convention).
Min7 dir_idx scope: comment notes it's declared earlier in kernel.
Min8 action_select args: explicit — three buffers exist on GPU
but not currently passed; new params, no new buffers.
Min9 Phase 0 time math: 5 hours tests + 1 hour enumeration + 2
hours 0.F + (3 hours runtime if checkpoint training needed,
runs in parallel). Honest budget.
Min10 "Two-stream" → "5-Layer Gate" header.
Min11 Plan 5 reference uses full path consistently.
Min12 Plan B time budget: explicit 1 day if pass; 2-5 days if bug.
Min13 active_frac: clarified Long+Short combined, not per direction.
Min14 train_active_frac: now in Phase 2 deliverables (see M7).
Min15 "20 mechanisms" → 21, with sub-counts in section headers.
Spec now 615 lines, comprehensive coverage of:
- Pearl + theoretical foundation
- Problem statement (with bias-might-be-correct caveat)
- Architecture (Thompson at training, argmax at eval)
- Train vs eval distinct selectors with behavior-change disclosure
- Direction-only scope with magnitude σ-bias warning
- Conviction stays E[Q]-based (no Kelly cap jitter)
- Interaction matrix: 21 mechanisms in 3 categories
- 6 v2 enhancements documented + deferred
- Phase 0/1/2/3 with tests, exit gates, time budgets
- 5-layer verification + train_active_frac instrumentation
- 8 risks with mitigations + 5 stop conditions
- What v1 doesn't touch (referencing interaction matrix)
- References (Thompson 1933, Bellemare 2017, Dabney 2018, etc.)
- Aggregation contract (project-wide pearl, enforced)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per-user direction: every existing mechanism in the DQN system MUST be
explicitly considered for interaction with Thompson, and Thompson itself
MUST be examined for system-specific improvements beyond vanilla.
INTERACTION MATRIX (3 categories, 20 mechanisms):
Category 1 — Compose with Thompson (no change required):
Counterfactual reward, B.2 novelty bonus, PopArt, Saboteur,
Curiosity, NoisyNets/VSN, Distillation, CQL, Polyak target EMA,
HER, PER, Replay warm-start, Multi-fold validation harness.
Category 2 — Trivially adapt to Thompson (one-line changes):
D7/N7 contrarian sign flip (negate the SAMPLE), cosine epsilon
schedule (still applies to mag/ord/urg), per-sample epsilon (IQL
expectile gap), adaptive Boltzmann tau (still applies to mag/ord/urg).
Category 3 — Take precedence over Thompson (hard constraints):
Plan-based action lock (Thompson sample discarded if plan active),
per-magnitude Kelly cap, trail stop, capital floor breach.
Critical insights from the audit:
1. NoisyNets is ALREADY a form of training-time Thompson at the
parameter level. Output-space Thompson stacks on top —
total exploration = parameter-space ⊗ output-space (multiplicative).
2. Curiosity is ORTHOGONAL to Thompson — Thompson explores actions
whose Q is uncertain; curiosity explores states whose dynamics
are uncertain. Both axes desirable; no conflict.
3. Plan lock takes precedence; same as currently with Boltzmann.
OUTSIDE-THE-BOX v2 ENHANCEMENTS (deferred to follow-up specs):
v2.1 Triple-source Thompson (C51 + IQN + Ensemble) — incorporate
the existing ensemble Q-head as 3rd uncertainty source.
v2.2 Persistent Thompson (anti-churn for HFT) — bias sampling
toward current direction, ISV-driven; reduces tx_cost from
Long/Short oscillation across bars.
v2.3 CVaR-aware eval (risk-adjusted deployment) — eval picks
argmax(E[Q] − λ·CVaR_α[Q]); risk-aware decision making for
production with real capital.
v2.4 Information-Directed Sampling (Russo & Van Roy 2014) — picks
action minimizing regret²/info_gain; more efficient than
vanilla Thompson when learning saturates.
v2.5 Hierarchical Thompson on (trade vs no-trade) → (which
direction) — addresses 50/50 structural advantage of no-trade.
v2.6 Composition with curiosity-driven exploration — explicit
coupling beyond reward-side composition.
Each v2 enhancement gets its own spec/plan when prioritised. Vanilla
Thompson is v1; ships first; verified independently.
ALSO FIXED (from earlier self-review):
- Pearl claim softened: only the C51 Hold/Flat bias is directly
attributed; other historic bugs had different mechanisms.
- TFT entry removed from contract table — TFT is a Variable
Selection Network (feature processor), not a Q-head. Replaced
with generic "Future Q-head additions" placeholder.
- Eval direction = argmax E[Q] explicitly flagged as a behavior
change from current Boltzmann-with-tau (val_dir_dist will be
more concentrated than current).
- Phase 0.E budget reduced (1-2 hours, not 1 day) — synthetic
bandit is ~50 lines of Rust, not full RL training loop.
- Phase 0 enumerates existing checkpoints before training new one.
- Architecture diagram parenthesis fixed.
- Conviction implementation note: compute E[Q] once, reuse for
conviction AND eval-mode argmax — no redundant computation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User correctly challenged the "band-aid removal" framing. All fixes
shipped during the val-Flat-collapse investigation addressed real bugs
at their respective layers and should be preserved:
- Kelly cap warm-branch (0c9d1ee39): post-decision physics layer.
Thompson-independent. KEEP.
- Train Return display + Sharpe annualization (non-tau parts of
7a3d88646): display/metric layer. Thompson-independent. KEEP.
- Direction Boltzmann tau-floor (tau part of 7a3d88646) +
adaptive eps_dir floor (d54b49efc): gates inside direction-branch
action selection. Phase 2 replaces direction-branch action
selection wholesale (eps-greedy + Boltzmann → Thompson), so these
direction-only code paths become structurally unreachable.
The latter two are NOT band-aids being removed because Thompson is
better. They are dead code being cleaned up because Thompson replaces
the surrounding mechanism. Magnitude/order/urgency branches keep their
existing eps-greedy + Boltzmann + tau-floor + EPS_FLOOR paths intact.
Reframed Phase 3 deliverable: "direction-branch dead-code cleanup"
with explicit rationale (per feedback_no_legacy_aliases.md and
feedback_no_partial_refactor.md). 0.5 day budget instead of 1.
Also clarified eval action selection: argmax of (E[Q_C51]+E[Q_IQN])/2
is correct. Bellman backup is a Q-learning UPDATE rule, not an
action-selection rule. Once Q is learned, optimal policy is greedy
argmax of learned Q. Online Bellman lookahead at eval would require
a forward model of market dynamics — not available, not standard.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Revises the C51-bias spec after deeper review surfaced 12 design gaps,
of which 4 were critical:
1. Train-only vs train+eval ambiguity — UCB at eval would conflate
"model recommends Long" with "model is uncertain about Long",
inflating reported edge. CRITICAL for trading where eval drives
real capital decisions.
2. Thompson sampling is more principled than UCB:
- parameter-free (no κ to tune)
- uses distribution directly without scalar reduction
- naturally explore-exploit balanced via distribution shape
3. c51_alpha is the wrong blend weight (it's C51-vs-MSE-warmup, not
C51-vs-IQN). Equal-weight average of C51 and IQN samples is the
structural choice — no tuned blend weight needed.
4. The bias might be CORRECT BEHAVIOUR — model rationally choosing
Flat when no edge has been discovered. Phase 0 must include a
synthetic-edge test (controlled MDP with KNOWN positive Long
expected value) to verify Thompson can discover edge if it exists.
Other gaps fixed:
- Eval at argmax E[Q] (not Boltzmann, not Thompson)
- Pearl wording broadened to cover ensembles + future methods
- Ensemble Q-head added to aggregation contract table
- Explicit caveat: NEVER extend Thompson to magnitude branch (would
worsen existing magnitude saturation)
- Phase 0.F uses CONVERGED checkpoint (≥30 epochs), not 2-epoch run
- L4 long smoke (30 epochs, ~1 hour) added — Thompson edge discovery
needs longer feedback loop than 5 epochs
- Phase 3 explicitly removes eps-floor + tau-floor band-aids
(Thompson replaces direction Boltzmann; band-aids become dead code)
- Conviction stays E[Q]-based, not sample-based (avoid Kelly cap
jitter from stochastic samples)
Architecture (Thompson only, no UCB):
TRAINING: dir_idx = argmax(0.5 × (sample_C51(d) + sample_IQN(d)))
magnitude/order/urgency: existing Boltzmann + ε-greedy
EVAL: dir_idx = argmax(0.5 × (E[Q_C51] + E[Q_IQN]))
magnitude/order/urgency: existing Boltzmann (eval mode)
Direction-branch ε-greedy + Boltzmann are REMOVED — Thompson is the
exploration mechanism. No new GPU buffers; existing C51 atoms + IQN
quantiles passed to action_select.
5-7 days active work across 4 sub-plans; each gets its own
writing-plans cycle.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Designs the structural fix for the C51 expected-Q Hold/Flat bias exposed
by the Kelly val-Flat-collapse fix. The bias is a manifestation of a
deeper project-wide pearl:
"Distributional RL aggregation discards uncertainty;
action selection must restore it."
Any value head representing Q as a distribution (atoms, quantiles,
ensembles) MUST expose both E[Q] and σ(Q) to action selection. Boltzmann
on E[Q] alone produces structural bias toward low-variance actions
regardless of expected payoff — the C51+IQN Flat-attractor is one
instance of this lost-information pattern.
Fix: extract σ(Q) from BOTH C51 atoms (closed form) and IQN quantiles
(IQR/1.349), blend by loss-time weight, feed Q_eff = E[Q] + κ·σ
(κ=1.0 structural identity) to direction-branch Boltzmann ONLY.
4-phase implementation:
Phase 0 — TDD hypothesis verification (Rust mirror functions + 5 unit
tests including GPU integration on real checkpoint)
Phase 1 — Audit existing reward levers (B.2, CF, PopArt, Q-target)
via 6 unit tests; fix any bugs found
Phase 2 — UCB integration: new compute_q_with_uncertainty kernel,
modified action_select, Rust orchestration, project-wide
aggregation contract in dqn-wire-up-audit.md
Phase 3 — Verification per Plan 5 Task 5 multi-seed × multi-fold
5-layer verification gate; 8 risks with mitigations; 4 stop conditions
that halt execution and force redesign.
Existing band-aid fixes (Kelly cap, eps-floor, tau-floor) stay — they
address symptoms at different layers. UCB adds the missing aggregation
step that was the common root across all the symptoms.
Direction-branch only — magnitude/order/urgency don't have the
Flat-attractor (atom-mass collapse asymmetry).
5-7 days active work across 4 sub-plans; each gets its own
writing-plans cycle.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User pearl (2026-04-24): the DQN already has every primitive TLOB needs
(gpu_attention, batched_forward/backward, GpuLinear, cuBLASLt handles,
cuda_autograd). Port TLOB's Q/K/V + attention as a composition of
existing primitives. Random-init, trainable end-to-end from the DQN's
reward signal. Uniform with Mamba2, IQL, atoms/γ/τ/ε — all of which
already follow this pattern.
Eliminates:
- ONNX Runtime dependency (was already dead — stripped from ml-supervised)
- Separate supervised pretraining pipeline
- "Freeze vs fine-tune" false dichotomy
- Pretrained-checkpoint-file-not-found failure mode
Prerequisites for the cuBLAS-native design (all satisfied):
- gpu_attention.rs exists
- GpuLinear trainable layer exists
- cuda_autograd over cuBLAS exists
- MBP-10 data already in the DQN data pipeline
What was "BLOCKED on prerequisites" in the Task 6C audit referred to
the OLD pretrained-ONNX design. The cuBLAS-native design has all
prerequisites satisfied — Task 6C can proceed under the new scope.
Pearl captured in memory: pearl_tlob_no_pretraining.md. Generalises to
any future attention/state-space/Neural ODE module: port to cuBLAS,
random init, let the DQN teach it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User raised that the CPU-controller pattern (where CPU computes adaptive
values from ISV and writes back to ISV) violates the architectural
principle: GPU kernels should compute adaptive decisions; CPU-side code
is pure observation.
Replaces the AdaptiveController trait with read-only AdaptiveMonitor:
- No update() — CPU doesn't compute adaptive values.
- No write_output() — CPU doesn't write adaptive values to ISV.
- read() returns the GPU-computed value from ISV.
- diagnose() emits HEALTH_DIAG snapshot.
- observe() + fire_rate() track how often the GPU-computed value changes.
Reclassifies the 9 adaptive mechanisms:
- 6 reactive get GPU kernel + CPU monitor: atoms, gamma, kelly_cap, tau
(Polyak EMA), epsilon, grad_balancer (last is already GPU-driven).
- 3 static get ISV constructor-write, no monitor: cql_alpha,
conviction_floor, plan_threshold.
New ISV slots for GPU-written adaptive outputs (EPSILON_EFF, TAU_EFF,
GAMMA_EFF, KELLY_CAP_EFF) and CPU-born inputs (EPOCH_IDX, TOTAL_EPOCHS),
plus slots for the 3 static configs.
Rationale:
- Unified adaptive machinery, no CPU-side special cases.
- Per-sample granularity available (Expected SARSA τ in c51_loss_kernel
is already the exemplar — reads ISV q_gap + health per sample).
- Zero CPU→GPU config transfer in any path.
- ISV is single source of truth for every adaptive value.
Plan 1 Tasks 8-17 restructured:
- Task 8 creates AdaptiveMonitor trait (not AdaptiveController).
- Tasks 9, 10, 11, 13, 14, 17: GPU kernel + monitor pairs.
- Tasks 12, 15, 16: static ISV writes only.
Follow-up commits will revert Batch A (d76849f31) and Batch B (4189da563)
which implemented the old CPU-compute pattern, then re-implement under
this design.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Populated docs/dqn-wire-up-audit.md with every pub module and CUDA
kernel in the DQN path. Each entry classified Wired / Partial / Orphan /
Ghost / OUT-of-DQN-scope with the action plan linking to the plan+task
that resolves any non-Wired status.
No Orphan left unclassified. Orphans fall into three buckets:
1. Scheduled for wiring by a later Plan (gpu_statistics → Plan 2 D.2;
tlob_loader → Plan 2 D.8).
2. OUT-of-DQN-scope because supervised consumers exist (PPO kernels,
xLSTM, KAN trainable adapter, flash_attention, benchmarks).
3. Genuinely unused — escalated to user review in the task output,
not deleted autonomously (streaming_dbn_loader, unified_data_loader,
training/orchestrator, training_pipeline, inference_validator,
model_loader_integration, paper_trading/mod.rs,
portfolio_transformer, regime_detection/mod.rs).
Summary: 109 total modules/kernels, 74 wired, 7 partial, 11 orphan,
0 ghost, 17 OUT-of-DQN-scope.
Plan 1 Task 6. Spec §4.A.5, Invariant 2.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User raised a backward-compat concern: integer "schema version"
semantically implies a family of coexisting versions with upgrade
paths between them, inviting the forbidden pattern
(feedback_no_legacy_aliases.md, feedback_no_partial_refactor.md).
Replace with a compile-time structural fingerprint:
- ISV[0..2) stores a u64 FNV-1a hash of the slot layout (split across
two f32 lanes preserving raw bits).
- The hash is computed by a const fn over the slot list; any slot
change automatically updates the fingerprint. No human decides
"what version is this now".
- Checkpoint load is fail-fast only. Error message does NOT mention
migration as an option.
- Pre-commit hook rejects any `fn migrate_isv|upgrade_isv` to make
the no-migration rule structurally enforced (landed as part of
Plan 1 Task 5 hook extension).
- StateResetRegistry entry renamed ISV_SCHEMA_VERSION →
ISV_LAYOUT_FINGERPRINT (Task 2's landed code touched in Task 5's
commit per no-partial-refactor).
Updated:
- spec §4.A.2 (fingerprint design + rationale)
- spec §5 landing-order note (fingerprint auto-updates on layout change)
- Plan 1 architecture line
- Plan 1 Task 2 StateResetRegistry test + entry naming
- Plan 1 Task 5 — full rewrite of implementation steps
- Plan 1 exit criteria #6
- Plan 2 pre-plan verification (grep fingerprint constants, not version==0)
- Plan 2 Task 6D.1 (update fingerprint seed, not bump version)
- Plan 2 Task 6D exit criteria #7
- Plan 3 pre-plan verification
- docs/isv-slots.md ISV[0..2) row
No code changed. Plan 1 Task 5 is not yet implemented — this commit
realigns the spec + plans so the implementer subagent works from the
corrected design.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User-mandated addition:
Invariant 9 — No deferred work, no stubs, no TODO/FIXME
Every commit lands complete. No stubs (placeholder return values), no
TODO/FIXME/XXX/HACK/TBD markers, no half-finished implementations. If
it can't finish in this commit, it doesn't start. Stubs and deferrals
train the network on semantic emptiness — invisible in convergence
metrics but burn GPU time optimising against partly-fake signal.
Authority: feedback_no_stubs.md, feedback_no_todo_fixme.md,
feedback_no_quickfixes.md elevated to first-class spec enforcement.
Pre-commit hook greps for forbidden markers.
Part E decisions made concrete (per Invariant 9):
- E.1 TFT VSN: IN (full VSN extension)
- E.2 GRN: IN if A.5 audit finds absent (otherwise mark existing as canonical)
- E.3 Multi-quantile heads: IN (5/25/50/75/95 quantile decomposition)
- E.4 Encoder-decoder: IN (extends D.3 to full trunk/head separation)
- E.5 Interpretability: IN (attention-weight ISV diagnostics)
- E.6 Auxiliary heads: IN (next-return + regime-classification)
- xLSTM: OUT (redundant with Mamba2+TLOB; YAGNI)
- KAN: OUT (function approx not a bottleneck)
No "evaluate later" items. Every concept either lands or explicitly
doesn't, with rationale.
§8 decisions tightened:
- D.8 TLOB mode: decision criterion = per-step cost benchmark ≤ 5ms
- C.6 controller order: fixed in spec (atoms→gamma→Kelly→cql_alpha→
tau→epsilon→conviction_floor→plan_threshold→balancer)
- A.3 resource plan: auto-switch L40S→H100 at 12 GPU-hour threshold
- A.4 metric band init: [mean − 3σ, mean + 3σ] from last 3 good runs
No discretionary "we'll see" remain. All decisions data/order/budget/
statistic-driven.
Counter updates: "seven invariants" → "nine" in §3 header and §5
cleanup contract.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Self-review pass on the DQN v2 unified design spec (d13b53586). Fixes
applied inline:
Ambiguity fixes:
- §3 Invariant 3: explicit definition of "training step loop" (per-step
code in fused_training::step_fused and captured graph children; NOT
per-epoch / per-fold / per-checkpoint code).
- §2 Tier 1: "stable epochs" defined as epochs [warmup_end..run_end)
with warmup_end from B.3's seed-phase decay or explicit flag.
- §4 A.2: migration mechanism clarified as fail-fast initially; helpers
added only when a specific migration need arises (no speculative
scaffolding).
- §4 A.3: N=5, K=6 stated as DEFAULTS with ≥ MINIMUMS per Invariant 6;
resource budget flagged (~8-10 GPU-hours per validation pass).
- §4 B.3: "≥ 100K experiences" (minimum), warm-restart clarified as
runtime condition not feature flag.
- §4 C.3: adaptive KL threshold mechanism made explicit (second ISV
slot for threshold EMA, third for amplification multiplier).
- §4 C.6: cleanup timing explicit — old scaffolding removed in the SAME
commit that migrates its last consumer (no deferred pass).
- §4 D.1: DQN-path-specific scope clarified; validation via grad-norm
smoke + integration test.
- §4 D.6: plan_isv index naming made consistent with existing layout;
coordinated state-layout migration commit called out.
New Invariant 8 — Named dimensions, not indices:
Every dimension, slot, offset, or semantic position in a multi-field
buffer has a named constant. Raw numeric indices (`[0]`, `[6]`, `[23]`)
appear ONLY in the definition site. Named constants specified for ISV
slots (existing), portfolio state ps[0..30), plan_isv[0..7), plan_params
[0..6), state-vector offsets, branch indices (BRANCH_DIR/MAG/ORD/URG),
action sub-indices (DIR_SHORT/HOLD/LONG/FLAT, MAG_QUARTER/HALF/FULL).
Enforcement: audit pass during A.1/A.2, lint via grep for raw-index
access outside definition modules.
Landing order revision:
- Dependency graph explicit (A.2 before new ISV slots, D.1 before
D.2-4-8, state-layout changes in ONE coordinated commit).
- Spec decomposition note added — writing-plans may produce multiple
sequential plans rather than one monolithic plan.
New doc tracked by pre-commit: docs/dqn-named-dims.md (Invariant 8).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Unifies the Q-support range source across atom grid / warm-start quantile
clamp consumers via the ISV signal bus. One broadcast written at epoch
boundary from per-branch Q-stats EMAs, read by the two consumers that
previously held disagreeing ranges. Target observation: atom utilisation
≥40% (up from 11-15% on train-fpxnw).
Phase 0 — per-branch Q-stats kernel Rust plumbing:
* Load q_stats_per_branch_reduce alongside legacy q_stats_reduce
* Add per_branch_q_stats_pinned (28 f32 = 4 × 7, device-mapped)
* PerBranchQValueStats struct: [QValueStatsResult; 4]
* reduce_current_q_stats_per_branch launches the new kernel with the
four branch (off, size) pairs derived from config.branch_N_size
Phase 1 — ISV v-range plumbing (zero behavioural change at epoch 1):
* ISV_NETWORK_DIM=23 preserved for w_isv_fc1 sizing; ISV_TOTAL_DIM=31
allocates 8 additional slots for per-branch (centre, half-width)
* Slot constants V_CENTER_DIR..V_HALF_URG covering slots 23..30
* eval_q_mean_ema / eval_q_std_ema / eval_ema_initialized promoted
to [f32; 4] / [bool; 4]; scalar setters preserved for trajectory
backtracking (broadcast same value to all branches)
* Bootstrap at construction: centre=0, half=(v_max-v_min)/2 → the
byte-identical [config.v_min, config.v_max] span per branch before
any Q observations arrive
* reset_eval_v_range_state resets the 4 per-branch EMAs AND the 8 ISV
slots to bootstrap values; legacy eval_v_range_pinned[2] still reset
(deferred removal — spec Phase 3)
* update_eval_v_range reworked: signature takes PerBranchQValueStats and
per_branch_q_gaps. Maintains 4 independent adaptive-rate EMAs,
computes (centre, half) per branch with min_half_floor=0.1×(v_max-v_min)
and clamps to config bounds, writes 8 ISV slots. Branch-0 (direction)
centre±half is also mirrored into the legacy eval_v_range_pinned for
consumers that have not yet migrated to the per-branch bus.
Phase 2a/2b — atom grid per-branch v-range:
* adaptive_atom_positions kernel signature changed from
(v_min: float, v_max: float) to (branch_idx: int, isv_signals: float*);
reads centre/half from ISV slots 23+2·b, 24+2·b. Eliminates the f64→f32
ABI trap (spec Phase 2 side-effect) since the only per-branch range
path is now pointer-based.
* recompute_atom_positions passes branch_idx + isv_signals_dev_ptr per
branch; no scalar v_min/v_max arg remains.
Phase 2c — warm-start quantile clamp per-branch from ISV:
* warm_start_atom_positions reads per-branch (centre, half) from pinned
ISV host memory, clamps shared reward-quantile vector into each
branch's adaptive range before tiling into atom_positions_buf.
Bootstrap makes this equivalent to the pre-spec config.v_{min,max}
clamp until the first Q observation lands.
Deviations from spec:
* Phase 2d (per_sample_support_buf → [N, 4, 3]) NOT implemented. The
spec's premise was that per_sample_support is host-tiled from
eval_v_range, but the active path in this codebase has it filled by
iql_compute_per_sample_support (V(s)-centered, per-sample, already
adaptive) — orthogonal to the ISV bus. Migrating that kernel to
per-branch output would require rewriting iql_value_kernel +
iql_support_floor + C51/MSE loss kernel indexing in lockstep, which
the "no unrelated refactoring" constraint disallows. The loss-kernel
Bellman projection today uses V-centered bounds that are themselves
adaptive; the ISV v-range fix still lands the primary win (atom grid
+ warm-start agreement) without touching IQL.
Compile verified: cargo check -p ml + --workspace pass (SQLX_OFFLINE,
CARGO_INCREMENTAL=0, sccache). No TODO/FIXME/XXX introduced.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Diagnostic-only audit of all 22 ISV slots using train-mdh86 HEALTH_DIAG
(20 epochs, same oscillation pattern as train-rq6n8). Identifies three
dead writers (slots 2, 3, 4), one strongly anti-correlated signal
(slot 12 learning_health, r=-0.765 vs Sharpe), sample-0 bias in four
regime slots (8-11), and EMA-pollution risk on the Q-scale EMAs
(16, 21) that feed Kelly conviction + C51 bin weighting.
Produces a per-slot table with writer/reader citations and an ordered
fix list. Hypothesises the observed +34 → -67 Sharpe swing is driven
by health mis-reporting "improving" while policy diverges, q_dir_abs_ref
contamination collapsing Kelly, and the zero-writer TD-error EMA
disabling micro-reward regime awareness.
No code changes — reconnaissance only.
Compares the 51-feature TLOBFeatureExtractor (crates/ml-supervised/src/tlob/
features.rs + mbp10_feature_extractor.rs) against Foxhunt's 20-slot OFI
vector persisted in .fxcache.
Findings: of TLOB's 51 slots, 23 are placeholders (sine waves / hardcoded
constants), 12 duplicate existing OFI or 42-dim market features, and only
~10 carry genuinely novel information. Meanwhile the existing
MicrostructureState struct in ml-features already computes 10 features
(realized variance, Hawkes intensity, weighted book pressure, spread
dynamics, aggression ratio, queue-depletion asymmetry, order-count flux,
intra-bar momentum, regime score, OFI trajectory) that are dropped before
reaching fxcache — these dominate the TLOB novel candidates and require
zero new math.
Phase B plan (drafted inline): bump OFI_DIM 20→28, bump FXCACHE_VERSION
4→5, prioritize persisting the 10 Foxhunt-internal MicrostructureState
features before any TLOB-derived additions, sweep all hard-coded 18/20
constants across CUDA kernels + gpu_dqn_trainer.rs, and rely on the
ensure-fxcache Argo step for cache regeneration.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extends grad_decomp_kernel to snapshot the trunk tensor slice (tensors
0..4 = w_s1, b_s1, w_s2, b_s2) in addition to the existing direction +
magnitude branch slices (8..12 / 12..16). Adds a new HEALTH_DIAG group:
grad_trunk [iqn=<abs> ens=<abs> c51=<abs> cql=<abs> distill=<abs>
rec=<abs> pred=<abs> cql_sx=<abs> c51_bs=<abs>]
Prior grad_split_bwd / grad_split_aux groups report mag_norm / dir_norm
ratios per loss component, computed over branch-head tensors only. That
measurement range structurally reports 0.0000 for any loss component
that writes exclusively to the trunk — IQN and Ens in particular. This
caused the persistent misdiagnosis that IQN-to-trunk was not wired; the
prior scoping in /tmp/foxhunt_research/iqn-to-trunk-wiring-scoping.md
confirmed the wiring is live (apply_iqn_trunk_gradient at
gpu_dqn_trainer.rs:4882) and that the zero reading was a blind spot in
the measurement pipeline.
Smoke confirms the diagnostic: after iqn_readiness ramps up (late
epochs), grad_trunk reports iqn=100..381 (real trunk SAXPY amplitude),
ens=0.07..3.57, c51=2.46..8.91 (value-head dueling path contributes
through trunk), while cql/cql_sx/distill/rec/pred stay near-zero — a
clean diagnostic baseline.
Also fixes stale documentation at dual-distributional-c51-iqn-design.md
that claimed "IQN trains in isolation — its gradients don't flow back
to the shared trunk": reworded to reflect current wired state with
file:function citation and explicit iqn_readiness gating note. Updated
the "What Changes" table ("IQN training") and "Implementation Order"
(Phase 1 marked DONE) with the same citation.
Changes:
- grad_decomp_kernel.cu: per-component result slot 2 → 3 floats
(mag_norm, dir_norm, trunk_norm); extra __shared__ sum_trunk +
tree reduction; new grad_trunk_start/trunk_len kernel args.
- gpu_dqn_trainer.rs: pinned result buffer 18 → 27 floats; snapshot
now does two copy_f32 passes (trunk → dst[0..trunk_len), branch →
dst[trunk_len..]); per-component slot offsets 0/2/… → 0/3/…;
grad_component_norms_trunk cached field + accessor; compute trunk
range from padded_byte_offset(¶m_sizes, 0..4).
- fused_training.rs: grad_trunk_norms_by_component() + per-component
grad_trunk_*_abs() accessors.
- training_loop.rs: HEALTH_DIAG emits new grad_trunk group ordered
[iqn ens c51 cql distill rec pred cql_sx c51_bs]; extended doc
comment explaining the three groups' roles.
- design spec (Problem #1 + What Changes row + Implementation Order):
stale "IQN trains in isolation" replaced by current wired-state
description, cites gpu_dqn_trainer.rs:4882 and readiness ramp at
gpu_dqn_trainer.rs:4228-4243.
Pure diagnostic — no training dynamics change, no atomicAdd, no tuning
knobs, no TF32 changes. Smokes unaffected (magnitude_distribution H10
regression pre-exists on HEAD 810b3c570; 4 other smokes pass).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Task 2.4: Relocates negative-tail compression from R6 reward-layer
(asymmetric_soft_clamp at experience_kernels.cu:78-81) to C51 Bellman
target smoothing (c51_loss_kernel.cu::block_bellman_project_f).
Functionality preserved — same invariant, better location. Upper +10
cap kept inline as fminf(reward, 10.0f) for numerical safety.
Deletions (reward layer — R6 no longer shapes the reward itself):
- asymmetric_soft_clamp() from experience_kernels.cu:78-81 (no callers)
- Reward-layer clamp replaced with fminf(reward, 10.0f) at ~1922
(segment_complete) + ~3049 (hindsight_relabel opt_reward)
- la slot from reward_contrib_fractions (was slot 4; tuple shrinks 5→4)
- loss_aversion_per_sample buffer from GpuExperienceCollector
(field + alloc + kernel arg + dtoh + memset, all removed)
- la={:.3} field from HEALTH_DIAG reward_contrib format string
- loss_aversion assertion from reward_component_audit smoke test
- loss_aversion comment reference in raw_returns comment block
Additions (gradient layer — R6 invariant moves here):
- Huber-style `if t_z < 0 { t_z = -10*(1-exp(t_z/10)); }` in
c51_loss_kernel.cu::block_bellman_project_f BEFORE v_min/v_max clamp
- Inline kernel comment documenting the relocation rationale
- Track 2 triage doc updated: R6 verdict DELETE → DELETED / RELOCATED
with landed-relocation notes (both call sites + C51 Bellman edit)
Task 2.5 Bug #6: Stale `patience_mult` docstring at
experience_kernels.cu:1144 referenced the defunct R7 V8 reward (deleted
in Task 0.8). Rewrote the reward-shape docstring to reflect current
post-V7 / Task 0.8 reality (sparse = 2.0 * vol_normalized_return, capped
inline) and notes the R6 relocation. Per feedback_trust_code_not_docs.md.
Per feedback_no_functionality_removal.md: R6's invariant is RELOCATED,
not deleted. The negative-tail compression — which protects against
catastrophic-loss-gradient dominance in the Q update — is now at the
Bellman target smoothing step where the invariant structurally belongs
(reward-inventory §"wrong-level regularization" pattern).
Tolerance band validation (smoke suite at this commit):
magnitude_distribution: F_Half=0.150 F_Full=0.237 (≥0.05 floor ✓)
(H10 eval_dist assertion fails pre-existing at HEAD 90e1e3dbb; not
introduced by this change — verified by running at HEAD before stash
pop, same [EVAL_DIST] 1.000/0.000/0.000 collapse.)
reward_component_audit: cf_flip=0.584 trail=0.304 (cf_flip≥0.1 ✓, PASS)
controller_activity: [CTRL_FIRE] anti_lr=0.000 tau=0.000 gamma=0.000
clip=0.400 cql=0.000 cost=0.000 (PASS)
exploration_coverage: entropy @ep5=0.988 @ep20=0.985 (PASS)
multi_fold_convergence: Best Sharpe 81.54/38.82/84.18 (≥20 floor ✓)
best_val_metric 0.043/0.024/0.049 (baseline was 0.028/0.018/0.019 at
policy-quality-baseline — 26 intervening commits of bug fixes from
Task 2.5 bugs #1–#7 would account for persistent drift; within
run-to-run variance of HEAD-pre-change)
Scoping output from the Full=0% investigation (post-Task-2.2 smoke showed
eval_dist [eq=0.580 eh=0.420 ef=0.000] — Quarter+Half recovered,
Full still 0%).
Root cause: Q-estimation bias (category C) — C51 expected-Q over atoms
systematically under-prices higher-variance bins. Full has 4x Quarter's
PnL variance per experience_kernels.cu risk scaling. Bias is already
documented at experience_kernels.cu:904 (commented as "structural
distributional bias — tight return distributions (Small) get higher
expected Q under the C51 softmax regardless of actual expected returns").
Not noise-starvation: Full picked on 32% of training samples across
60 epochs × 3 folds = ~61k Full samples. Ample data.
Proposed Task 2.X = Rank 1 + Rank 2 combined (~90 LOC):
Rank 1: per-bin variance weighting in C51 branch loss — amplify
Full's gradient by sqrt(var_f / mean_var)
Rank 2: Q-spread regularization — lambda * ReLU(target_spread -
Q_spread)^2 penalty preventing degenerate fixpoint
Alternative fixes ranked but not recommended:
Rank 3 per-magnitude reward shaping (~80 LOC) — only if data-disfavor
is confirmed at L40S
Rank 4 magnitude curriculum (~20 LOC) — rejected (noise-starvation ruled out)
Rank 5 state-vector enrichment (~150 LOC) — premature
Prerequisite: ~40 LOC of per-magnitude win-rate + realized-variance
instrumentation to confirm category C vs A at L40S scale.
L40S gating recommendation was:
ef >= 0.15 at L40S → Task 2.X not needed (smoke artefact)
0.05 <= ef < 0.15 → Task 2.X scoped but optional
ef < 0.05 → Task 2.X required
Per feedback_fix_aggressively.md (landed this session): we ship Task 2.X
ahead of L40S since the bias is already code-documented, the fix is
well-scoped, and the 90-LOC cost is reasonable. L40S validation (Task
2.8) will verify the fixed state end-to-end instead of the un-fixed one.
No deletions anywhere per feedback_no_functionality_removal.md.
Per feedback_no_functionality_removal.md: R5 was originally scoped as
DELETE in the Phase 2 plan and the Track 2 triage because
reward_contrib[3] = 0.000 across 60 / 60 smoke epochs and
dqn-smoketest.toml sets micro_reward_scale = 0.0. Re-examination during
Phase 2 Task 2.3 rejected the DELETE path:
- dqn-production.toml already sets micro_reward_scale = 0.1, so R5 is
load-bearing in production, not dead code. The 0.0 value in smoke is
deliberate test-isolation (td_propagation / magnitude_distribution /
reward_component_audit all want the sparse-reward TD path isolated).
- The state-vector OFI block at state[SL_OFI_START..SL_OFI_START+SL_OFI_DIM)
= [42..62) provides representation features for the encoder (policy
side). R5 is a per-bar reward gradient on the critic (critic side).
Different mechanisms — production deploys both together.
- R5 also reads PREV_MID (retrospective hold quality), which is NOT in
the state vector. That signal exists only in the kernel branch.
Changes — pure documentation, no behavior change:
- experience_kernels.cu: ~30-line comment block at the R5 wiring site
(~L1915) documenting the parameter-not-flag status, production vs
smoke values, why state-vector OFI is complementary not redundant,
and the feedback_no_functionality_removal.md seal.
- experience_kernels.cu: fix stale kernel-signature comment that claimed
OFI was at state[66..74). Correct range is [42..62) per state_layout.cuh.
- config.rs: extend DQNHyperparameters::micro_reward_scale docstring and
add a comment at the Default impl pointing back at the kernel site.
- gpu_experience_collector.rs: extend reward_contrib_fractions docstring
to mark the micro=0.000 slot as a SEMANTIC value when the loaded profile
has micro_reward_scale=0.0, not a wiring regression.
- track2-triage.md: R5 verdict changed from DELETE to FIX-documented-disable
with the rationale above; "Proposed Phase 2 changes" section 1 and
"Next Track 2 steps" updated accordingly.
Smoke tests: 3 / 4 pass (reward_component_audit, controller_activity,
exploration_coverage). magnitude_distribution is failing on baseline
HEAD c0fee5a9b as well — pre-existing H10 magnitude-head collapse
regression (eval dist_mag ef=0.000, eh=0.000), unrelated to R5.
Verified via git-stash round-trip: baseline fails, changes do not
introduce new failures.
Closes Track 2 R5 verdict (originally DELETE, now FIX-documented-disable).
V7 audit of the 7 adaptive controllers named in spec §5.3 against the
baseline controller_activity smoke run (3 folds × 20 epochs = 60 epochs,
intervention-based fire detection per commit ed4b30b49):
* C1 anti_lr: DIAGNOSTIC (0/60; wiring surprise — fire detector reads
base scheduler, not post-anti_lr LR, flagged for Phase 2 fix)
* C2 adaptive tau: DIAGNOSTIC (2/60; fires at fold boundaries only)
* C3 adaptive gamma: DIAGNOSTIC (1/60; pinned to floor by health-coupled
correction; re-measure at L40S when health is real)
* C4 adaptive grad_clip: CANDIDATE FOR DELETE pending ablation (12/60,
20% — above diagnostic, below load-bearing; needs Track 3 Step 2)
* C5 cql_alpha: DIAGNOSTIC (2/60; regime_stability gate uninformative
at smoke scale)
* C6 cost_anneal: DIAGNOSTIC by design (deterministic sigmoid; fire_cost
hard-coded false at training_loop.rs:2475)
* C7 base LR scheduler: DIAGNOSTIC by construction (pure open-loop, not
in the closed-loop controller battery)
Cross-controller finding: no controller fires in > 50% of epochs, so
the policy is not currently held on the rails by adaptive intervention
at smoke scale. However, C2 / C3 / C5 are structurally inert here —
their trigger signals (health, regime_stability) are degenerate at
smoke scale, so their 0-ish fire rates are lower bounds, not
representative. Promote to final after L40S validation.
Wiring surprise (anti_lr): fire_lr detection reads lr_scheduler.get_lr()
*before* the anti_lr multiplier applies. Under smoke Constant LR this
reports 0 correctly-for-the-wrong-reason; under L40S Cosine/Linear it
will false-positive on pure decay drift. Recommend detecting via the
anti_mult != 1.0 branch directly (training_loop.rs:2936–2942) for an
unambiguous intervention semantic matching C4/C6.
Follow-ups flagged for Phase 2:
- Fix C1 fire-detection wiring before L40S re-run
- Run C4 grad-clip ablation (25 min, gates the final C4 verdict)
- Optional: reset fire_counts per fold to remove C2/C5 boundary artefact
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Preliminary triage of spec §5.1 hypotheses H1–H10 using the Phase 0
baseline capture on RTX 3050 Ti. L40S validation pending per plan.
Verdicts:
H1 PENDING (needs forced-exploration instrumentation not yet wired)
H2 REJECTED var_scale=0.96 across 19/20 epochs; Var[Q] inactive at smoke scale
H3 INCONCLUSIVE kelly degenerate (insufficient win/loss counts at smoke scale)
H4 CONFIRMED grad_ratio_mag_dir=0.0000 across 20/20 epochs (threshold <0.1)
H5 REJECTED ent_mag stays ≥0.98 throughout; no bootstrap collapse
H6 REJECTED Full fire rate (0) is lower than Quarter fire rate, not higher
H7 REJECTED vsn and sigma symmetric between mag and dir branches
H8 REJECTED target-net drift equal (mag=dir=0.001)
H9 PENDING (same instrumentation gap as H1)
H10 CONFIRMED training ent_mag=0.98, eval F_Quarter=100%
Synthesis: H4 is the root cause. Magnitude branch receives ~0 gradient →
weights stay near init → three magnitude Q-values near-identical → argmax
picks bin 0 (Quarter) on ties → H10 manifests at eval time. H2, H5, H7,
H8 all ruled out as contributors.
Proposed Phase 2 priority: fix H4 (gradient-flow path into magnitude head
— likely per-component advantage weighting or direction-conditioning of
w_b1fc) + H10 (Q-margin argmax + stochastic eval rollouts as safety net).
Phase 1 next: validate preliminary verdicts on L40S, instrument
per-component gradient decomposition for magnitude, proceed with
Tracks 2/3/4 in parallel.
All <PENDING> markers replaced with real values from a clean 20-epoch
magnitude_distribution smoke run + 3-fold multi_fold_convergence run
on RTX 3050 Ti at HEAD 0472b9730.
All 5 Phase 0 smoke tests PASS:
magnitude_distribution — Quarter=0.596 Half=0.150 Full=0.255
reward_component_audit — cf_flip=0.619 trail=0.290 la=0.013
controller_activity — all 6 controllers < 20% firing
exploration_coverage — ent_mag @ep5=0.987 @ep20=0.985
multi_fold_convergence — 3/3 folds, Best Sharpe 51 / 39 / 87
This commit closes Phase 0. The next commit tags it as
policy-quality-baseline — the reference state against which Phase 2
interventions are measured.
Phase 0 instrumentation is complete (Tasks 0.4 / 0.5 / 0.8 / 0.10 / 0.16
shipped this session; 0.6 partial; 0.3 partial). This scaffold encodes
the full metric set with explicit <PENDING …> markers per row so the
GPU-run capture can fill in values without forgetting any HEALTH_DIAG
field.
The policy-quality-baseline tag is intentionally NOT applied yet —
tagging requires real captured values, not the scaffold. Runner workflow:
for t in magnitude_distribution reward_component_audit \\
controller_activity exploration_coverage multi_fold_convergence; do
SQLX_OFFLINE=true CUBLAS_WORKSPACE_CONFIG=:4096:8 \\
FOXHUNT_TEST_DATA=test_data/futures-baseline \\
cargo test -p ml --release --lib -- \$t --ignored --nocapture 2>&1 | tee out_\$t.log
done
Then edit this doc, replace markers with values, amend, and tag.
User preference: everything lands on main, ~500 LOC is small enough
that feature-branch isolation isn't needed.
Spec changes:
* §2.3 outcome paths: drop "unmerged branch" vocabulary
* §3 architecture: main-branch timeline, no feat/, no wip/
* §4 Phase 0: commits on main, tag policy-quality-baseline at end
* §5 Phase 1: experiments are worktree edits reverted after, only
triage docs commit (to main)
* §6 Phase 2: commits land on main, author chooses granularity,
each leaves smokes green
* §7.3 outcome handling: iterate on main, baseline tag for rollback
* §7.4 closing: tag policy-quality-v1, update status
Plan changes:
* Task 0.1: verify-starting-state (not branch creation)
* Task 1.x: temp-edit-then-revert (not wip branches)
* Task 1.5: triage docs commit directly to main
* Phase 4: tag + close (not merge + cleanup)
* Rollback: one git reset --hard command
Addressed functional concerns from third review:
1. Surrogate-noise gate was statistically weak — "random Sharpe ≤ 0.5×
trained Sharpe on same val data" could be satisfied by sample
noise. Replaced with percentile-based test:
* Pool all 6 val folds (~600 trades) for statistical power.
* Run N=30 surrogate random-action rollouts (matched marginal
action distribution — so difference is state-action mapping,
not action frequency).
* Assert trained pooled Sharpe > 95th-percentile of surrogate
distribution. Explicit false-positive control.
2. Escalation path ("scope exceeds sub-project A v2") was hand-wavy.
Now concrete: failure after 3 iterations → paused, not closed →
triage spec opened with evidence summary + diagnosis (policy/
reward/state/architecture/data bottleneck) + explicit decision
(continue on feat/policy-quality or fork feat/policy-quality-v2).
Triage itself is a full brainstorming cycle.
3. HEALTH_DIAG §4.2 Track 1 fields only covered H1–H5 — H6–H10
needed their own detection signals. Added:
- trail_fire_rate_{quarter,half,full} (H6)
- hold_time_at_exit_{quarter,half,full} (H6)
- vsn_mask_{magnitude,direction} (H7)
- noisy_sigma_{mag_head,dir_head} (H7)
- target_drift_{mag_head,dir_head} (H8)
- action_dist_eval_{quarter,half,full} (H10)
Fixed inconsistencies between sections after the rev-2 feature-branch
refactor (sections had gotten out of sync):
1. §2 (success criteria) rewritten — was stale from rev 1. Now has
clean mandatory/soft split matching §7.2, corrects the trade-count
gate (per-fold not averaged), references the surrogate-noise gate.
New §2.3 enumerates the outcome paths.
2. §4 and §6 no longer say "commit on main" — both land on
feat/policy-quality per the feature-branch architecture. Phase 4
merges to main at §7.4.
3. §7.3 outcome handling rewritten — was written for single-branch
model ("Phase 2 commit is NOT reverted"). New wording matches
feature-branch semantics: failing mandatory = don't merge; failing
soft = merge + follow-up.
4. §6 smoke-tests rule now lists all six Phase 0 smokes, not just
Track 1/2.
5. §7.4 Phase-4 merge-strategy added — --no-ff merge commit, tag
policy-quality-v1, cleanup sequence documented.
6. §5.5 conflict-resolution added — tracks can reach contradictory
conclusions (e.g. T1 says fix, T2 says delete). Resolution rules:
mandatory-gate dominance, simplification wins, escalation.
7. §4.3 baseline metrics now committed to the feature branch (not
"investigation-only"), matching the crash-safety discipline.
8. Budget / risk register aligned — 5.5–6.5 hrs plan + 1 buffer,
hard-capped at 3 Phase-3 validation runs.
If H9 confirms and Phase 2 removes the magnitude branch, F_Half/F_Full
gates become vacuous. Substitute: direction bin distribution healthy
(F_Short + F_Long each >= 20% of non-Flat actions). Preserves the
'model actually trades' intent of Criterion A.
Addresses critique from second self-review:
1. SCOPE GAP fixed: spec now covers all 4 V7 categories the user asked
for. Added Track 3 (controllers) and Track 4 (exploration) with their
own smoke tests (controller_activity, exploration_coverage) and
HEALTH_DIAG fields. Previous version only had action-space + reward.
2. H1-H5 expanded to H1-H10 with 5 new codebase-specific hypotheses:
- H6: Regime-adaptive trailing stop closing Full-positions early
- H7: VSN / NoisyNets masking magnitude features
- H8: Target tau too slow for magnitude propagation
- H9: Data genuinely favors Quarter (no bug, delete magnitude branch)
- H10: Entropy regularization + argmax tie ordering bias
3. H3 REFRAMED from "Kelly bug" to design critique — Kelly as hard
multiplier vs soft gate. Fix sketch updated to offer replacement.
4. Feature branch + tag strategy replaces "single branch" containment.
feat/policy-quality isolates work from main until Phase 4 merge via
PR. git tag policy-quality-baseline gives named rollback anchor.
5. SURROGATE-NOISE CHECK added as MANDATORY validation gate — catches
look-ahead/leakage bugs that would otherwise produce false-positive
edges. Permanent smoke artifact.
6. Validation gate restructured: B + surrogate-noise MANDATORY;
A criteria soft; controller-activity soft. Partial success path
explicit per-gate.
7. Risk register expanded — added scenarios for B-mandatory failure
(feat branch stays unmerged), all-10-hypotheses-REJECTED (fall back
to H9 delete-branch), surrogate-noise failure (halt spec, open
diagnostic spec), multi-load-bearing-controllers (tech debt doc).
8. Budget corrected to 5.5-6.5 hrs L40S (was underestimated 3.5-4),
with per-track breakdown.
Spec now: 464 lines, 10 sections, covers 4 V7 tracks × ~10 hypotheses
+ 8 reward terms + 7 controllers + 5 exploration mechanisms = ~30
audit items total.
Critical fixes after honest review:
1. reward-audit measurement: split into 3 classes (additive / transform
/ sample-selector) — one-size-fits-all metric couldn't work for
PopArt normalization or CF flip mirroring
2. H1 detection: replace undefined "Sharpe/state" with concrete
forced-exploration protocol (epsilon=1.0 for 1 epoch at mid-train,
sample states × magnitudes, compare Sharpe-per-trade)
3. H4 scope: clarify this is the per-action-branch gradient (direction
vs magnitude head), NOT the per-loss-component (IQN/CQL/C51/Ens)
budget. Fix sketch updated to target the correct knob.
4. Validation gate: criterion B (multi-fold Sharpe + WinRate) is now
MANDATORY; criterion A gates are soft. Prevents "partial success"
hiding a policy-quality failure.
5. Trade-count gate: per-fold ≥100 on ≥5/6, not an average — 5 lean
folds pooled with 1 fat one should not pass.
Minor:
- Budget trimmed to 3.5–4 hrs L40S (was 6–10)
- L40S pool made explicit as target
- Phase 1 branches now pushed as wip/* for crash safety
- Hyperopt-params mismatch handling documented
Sub-project A of the production-readiness roadmap. Single-branch,
phase-gated: measurement substrate → parallel investigation tracks
(magnitude collapse + reward V7 audit) → single convergence commit →
6-fold × 50-epoch L40S validation.
Target criteria at close:
A (action dist): F_Half/F_Full each ≥15% on ≥4/6 folds, WinRate
55-65%, ≥100 trades/val window
B (multi-fold): Best Sharpe > 10 on ≥5/6 folds, WinRate > 55% on
≥5/6 folds
Greenfield checkpoint policy — any action-space or weight-shape change
is allowed. All confirmed bugs fix together in one Phase-2 commit.
V7 discipline gates inclusion (measurement confirms the bug), no
ranking across findings.
Out of scope for this spec: paper trading (sub-project C), canary
deployment (sub-project D), hyperopt re-runs, architectural changes
beyond action space.
Awaiting user review before invoking writing-plans.
Completes Phase 3 of the env-unification effort: experience_env_step (training),
backtest_env_step (val single-step), and backtest_env_step_batch (val batched)
all call the same __device__ helper `unified_env_step_core` in trade_physics.cuh.
Drift between train/val becomes structurally impossible at compile time — any
change to the canonical step applies atomically to both.
Three structural fixes were required to make the training port work:
1) TWO max_position params in unified_env_step_core
Training scales `max_position` by Var[Q] (Kelly-from-atoms variance sizing)
→ `effective_max_pos`. The original training kernel used this for
compute_target_position_4branch but the UNSCALED `max_position` for
apply_kelly_cap and execute_trade. Initial port collapsed both into one
helper arg, tightening Kelly cap aggressively → tiny positions → Q-values
converge → q_gap=0.00 collapse. Split into `max_position_target` (scaled)
and `max_position_physics` (unscaled). Val passes the same value for both.
2) Remove double hold_time update
Helper now owns `update_hold_time(...)` inside the step. Training's
inline `hold_time = update_hold_time(...)` at line ~1574 was a double-
increment. Fix: capture `saved_hold_time` BEFORE helper for the
patience-multiplier in segment reward, then trust the helper's in-place
update of hold_time. Segment-hold-time semantics preserved.
3) Remove triple Kelly-stat update
Helper calls `record_kelly_trade_outcome(...)`. Training ALSO had Kelly
stat updates inside the reversing_trade block (line 1558) and the
exiting_trade block (line 1638). TRIPLE update → win_count/loss_count/
sum_wins/sum_losses inflated 3× → Kelly cap tightened proportionally →
positions starved → Q-gap collapse. Fix: only the helper now touches
Kelly win/loss/sum_wins/sum_losses. Training still updates its own
sum_returns / sum_sq_returns (continuous-Kelly stats, distinct
buffers not touched by the helper).
Smoke test result (RTX 3050 Ti, 20 epochs, single-trial):
Before port: Best Sharpe ~15-25 (varies, range 10-31)
After broken port: Best Sharpe 1.17 (q_gap=0.00 collapse)
After fix: Best Sharpe 39.25 (HIGHEST ever recorded)
q_gap 0.00 → 0.27 (growing, healthy separation)
sharpe_ema trajectory: -5.3 → 12.7 → 26.9 (rising)
Multi-trial (partial visible): q_gap rising 0.00 → 0.86 by epoch 6,
sharpe_ema 15-25 consistently positive. Both tests passed.
Training kernel now has the CORE STEP in a single helper call — the
kernel body downstream continues to handle training-only concerns
(counterfactuals, plan_params, reward shaping via shaping_scale,
saboteur via exploration_scale, replay buffer writes).
Net −99 LOC across three kernels replaced by shared helper calls.
Files touched:
crates/ml/src/cuda_pipeline/trade_physics.cuh (+15/-5)
crates/ml/src/cuda_pipeline/backtest_env_kernel.cu (+53/-108 net)
crates/ml/src/cuda_pipeline/experience_kernels.cu (+48/-102 net)
docs/superpowers/specs/2026-04-21-unified-train-val-env-design.md
(Phase 2 documented as helper-extraction; Phase 3 kernel-deletion
remains deferred — both kernel entry points kept for their distinct
threading models; the physics is what's now truly unified.)
Verified: cargo check + smoke test pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A) TD-propagation diagnostic smoke test (sparse rewards)
- New smoke_tests/td_propagation.rs captures training_sharpe_ema across
20 epochs with micro_reward_scale=0 to answer: "can sparse-reward
Q-learning extract policy from trade-completion P&L alone?"
- Asserts: finite sharpe_ema, no monotonic degradation (tolerance 0.1
over first-3 vs last-3 epoch avg), q_gap > 0.05
- First run answered YES: val Sharpe peaks at +12.49 at epoch 8 with
pure sparse rewards, confirming the objective is learnable. The
remaining issue is stability/overfitting, not TD propagation.
B) Entropy → plasticity gate in training_loop
- D3/N3 shrink_perturb trigger now fires on `last_action_entropy < 0.3`
OR `health_value < 0.3` (OR semantics, 3 consecutive epochs).
- Catches action-collapse cases the generic health metric misses: Q-values
separate cleanly but argmax stays pinned to a single branch.
- tracing::info now logs both signals.
C) commitment_lambda coverage verified
- Almgren-Chriss sqrt market-impact already in compute_tx_cost
(trade_physics.cuh:168-171). commitment_lambda was pure duplication.
- Inventory doc updated: P2 marked satisfied, table row status updated.
Files touched:
- crates/ml/src/trainers/dqn/smoke_tests/mod.rs (+2)
- crates/ml/src/trainers/dqn/smoke_tests/td_propagation.rs (new)
- crates/ml/src/trainers/dqn/trainer/training_loop.rs (+15/-3)
- docs/superpowers/specs/2026-04-21-phase1-reward-inventory.md (+6/-3)
Smoke test PASSED locally (RTX 3050 Ti, 30.99s end-to-end).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 2 P1 relocations — the intents behind two deleted reward shaping
terms now live at the correct layer: diagnostic monitoring, not reward.
1. training_sharpe_ema (was w_dsr reward input):
Field already existed. Added to HEALTH_DIAG log line as `sharpe_ema`.
Available for early-stopping, model selection, and exploration gating
without perturbing the objective.
2. action_entropy (was position_entropy_weight reward):
Computed from summary.action_counts at GPU summary download —
Shannon entropy normalized to [0, 1] by ln(n_bins). One-epoch lag
is fine for a diagnostic. Added to HEALTH_DIAG as `action_entropy`.
Stored in new trainer field `last_action_entropy: Option<f32>`.
New HEALTH_DIAG suffix: `diag [sharpe_ema={:.3} action_entropy={:.2}]`.
Verified visible on E1 smoke test epoch 18/19 output.
Also: honest recalibration of Phase 2 results added to the inventory
doc. The earlier commit messages framed "-150 → -17 Sharpe" as a 5×
improvement; in absolute terms Sharpe -17 is still catastrophic (you'd
blow the account). Phase 2 stopped active capital destruction but did
not find alpha. Sharpe_raw per bar went from -0.39 (lot of loss per bar)
to -0.09 (little loss per bar) — still net losing. q_gap=0.17 is a
capacity metric (network CAN differentiate), not profitability.
Path to actual profitability (Sharpe > 0) remains:
- TD propagation verification (Task #8)
- Unified env kernel (Phase 3)
- Possibly a different objective entirely
- Richer features (42 market + 20 OFI may be information-starved)
The rule going forward: before celebrating future improvements, ask
whether the result *crosses zero* (profitable) or is just *less
negative* (still losing).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mass deletion of the "v7 gem" reward terms identified in the Phase 1
inventory as category errors — each one rewarded an outcome-adjacent
behavior instead of encoding the underlying physics, and each
empirically hurt validation metrics more than it helped training
stability.
Deleted from experience_kernels.cu and all plumbing (Rust configs,
launch args, hyperopt logs, TOML entries):
* order_credit_weight - reward redundant with compute_tx_cost
(order_type_idx already differentiates
fills by order type)
* risk_efficiency_weight - reward double-counted drawdown penalty
asymmetrically (only on winners)
* urgency_credit_weight - reward was vol-normalized unrealized P&L,
pure rename of core return
* commitment_lambda - triple-counted churn + tx_cost
* w_dsr - kernel wrote DSR EMA but no longer added
to reward (dead); removed the EMA
bookkeeping too
* dsr_eta - kernel arg for the deleted DSR EMA
* position_entropy_weight - rewarded action-bucket diversity
regardless of outcome; histogram buffer
+ zero-init removed too
* exit_timing_weight - already inactive (used raw_next future
price, comment-deleted earlier)
* ofi_reward_weight - dead plumbing; OFI already passed as
feature through state[OFI_START..]
* opportunity_cost_scale - penalized flat when Q-gap wide;
redundant with Q-values themselves
Kernel arg count: experience_env_step_batch shrank from ~55 to ~45 args.
Rust-side config surface reduced correspondingly.
Results on E1 smoke test (20-epoch):
BEFORE any Phase 2 work:
Val Sharpe -120 to -150, MaxDD 10-15%, Sharpe_raw -0.39
AFTER reward_noise + Kelly (both envs) + urgency + this sweep:
Val Sharpe -17 to -22 (7× better)
Val MaxDD 0.27% (40× better)
Val Sharpe_raw ~-0.09 (4× better)
Training Sharpe_raw ~0 (stabilized from ±20 swings)
Final q_gap 0.1712 (highest yet, collapse mechanism fine)
The extreme train-Sharpe swings (+17 one epoch, -13 next) were not
learning dynamics — they were shaping-term noise. Core reward (P&L +
drawdown + churn + holding + tx_cost + Kelly physics cap) gives training
metrics that actually reflect what the model does.
Inventory doc (docs/superpowers/specs/2026-04-21-phase1-reward-inventory.md)
extended with a "better-form taxonomy" section: every deleted gem has
a correct layer it belongs to (physics, feature, diagnostic, gradient-
level regularization — not reward). Kelly cap and Q-target smoothing
are already relocated; others are scheduled per the taxonomy's P1/P2/P3
priority list.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extends the reward inventory with a review pass that preserves the
*ideas* behind behavioral terms while deleting the wrong-level mechanics.
Gems (relocate, don't delete):
- Probabilistic fill model (real physics, not shaping) → apply in both
modes via shared Philox RNG + exploration_scale
- Kelly-as-physics-constraint → hard position-size cap in trade_physics,
not a soft reward
- Path quality via per-step drawdown integration → no separate term,
just run drawdown penalty every bar
- Saboteur relocation → both modes, scaled by exploration_scale (training
1.0, validation 0.5, diagnostic 0.0) — avoids creating a new clean-vs-
noisy mismatch in the opposite direction
Pearls:
- Sparse reward is fine if TD propagation works — diagnose before
deleting micro_reward_scale; measure cov(Q(s_entry,a), trade_return)
- Every behavioral term maps to one of four failure modes: double-count,
misplaced physics, wrong-level regularization, or compensating for a
downstream bug. None are semantically "neutral" shaping.
Novel architectural moves:
- Diagnostic-only entropy tracking (no reward, just HEALTH_DIAG
logging)
- Single exploration_scale scalar replacing all mode toggles
(feedback_no_feature_flags compliant)
- Layered reward with per-term budget caps — makes the train/val Sharpe
gap computable instead of uncomputable magic
- Q-target smoothing replacing reward_noise_scale (regularization at
gradient level, not reward level)
Updated disposition table: 4 DELETE, 2 MOVE, 2 RELOCATE-to-physics,
1 DIAGNOSE-then-DELETE, 4 delete-dead-plumbing.
Three relocations (Kelly cap, saboteur scaling, Q-target smoothing) can
land as independent PRs before the unified_env_kernel rewrite, shrinking
Phase 2's change surface.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Enumerates every term that contributes to training or validation reward,
classifies each as P&L-aligned (keeper) or behavioral (remover), with
source line references into experience_kernels.cu and backtest_env_kernel.cu.
Findings:
- 7 P&L-aligned terms to preserve in the unified env (core return,
tx_cost, drawdown, inventory, churn, capital floor, + vol normalization)
- 8 active behavioral training-only terms to delete:
* order_credit_weight (rewards theoretical limit-order savings,
double-counts actual tx_cost)
* risk_efficiency_weight (double-counts drawdown asymmetrically)
* urgency_credit_weight (redundant with core return)
* kelly_sizing_weight (rewards matching a formula, not outcomes)
* micro_reward_scale (OFI-momentum signal follower)
* commitment_lambda (triple-counts churn)
* reward_noise_scale (belongs at gradient level, not reward level)
* position_entropy_weight (changes what "optimal" means)
- 4 dormant/dead terms to clean up (w_dsr, exit_timing, ofi_reward,
opp_cost_scale — all plumbing with no kernel effect)
Biggest offenders by magnitude: urgency_credit and risk_efficiency can
dominate the core signal on a single positioned bar.
Phase 2 (unified_env_kernel implementation) can now proceed with a
concrete deletion list, not a judgment call.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Design document for the follow-up to the adaptive-learning-rootcause
session. Lays out the evidence, root cause, options considered, and
recommended path for closing the 70% structural gap between training
and validation Sharpe that remained after the distillation collapse
fix landed.
Key findings documented:
- Reward-shaping ablation (2026-04-20) closed ~30% of the gap; ~70%
remains architectural
- Two env kernels (experience_env_step vs backtest_env_step) have
drifted: spread scaling, fill model, saboteur noise, reward terms,
action selection, position dynamics all differ
- Hint from history: experience_kernels.cu:1418 comment "Regime-
adaptive scaling removed to eliminate train/eval mismatch" shows
someone aligned *some* things previously
Recommended path (Option C, "unified env with layered reward"):
- Single unified_env_step kernel replaces both
- Core reward = pure P&L; shaping is additive and P&L-units-aligned
- Validation = training with exploration_scale=0 AND shaping_scale=0
- Scale factors are pinned device-mapped scalars (same pattern used by
the distillation alpha fix)
Phased implementation plan with ~8-day budget and concrete success
criteria: validation Sharpe_raw within 0.05 of training Sharpe_raw by
epoch 30 on L40S production run.
Rejected alternatives: backtest-matches-training (hides real issue),
training-matches-backtest (regresses stability), two-environment with
divergence as metric (fallback only).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Layer 2 labeled as G2-G5, G5 explicit for gradient budget
- P2 spectral_gap_norm added as 7th component in composition (rebalanced weights)
- N6 ensemble oracle has explicit threshold (>0.8 triggers N3 immediately)
- N8 meta-Q wiring clarified: logged but not in composition until validated
- Files Changed deduplicated, paths qualified
- Success criteria: WinRate >55% (above random baseline ~25%)
- HEALTH_DIAG log line includes all 7 components
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fixes Q-value collapse via unified LearningHealth signal that senses
training health (6 components) and continuously adapts:
- CQL regularization (regime + health gated)
- Gradient budget (IQN/CQL/Ens/C51 dynamic allocation)
- Tau target EMA (health-coupled)
- Expected SARSA temperature (continuous, no hardcoded threshold)
Plus 4 pearls (PER priorities, spectral detection, gradient consistency,
adaptive gamma) and 8 novels (self-distillation, barrier loss, plasticity
injection, CF curriculum, information bottleneck, ensemble oracle,
contrarian override, meta-Q network).
Core principle: training hyperparameters are OUTPUTS of the temporal
pipeline, not static schedules. The system meta-learns its own settings.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Training and validation use different kernels with different state layouts,
making generalization impossible. Single state_layout.cuh + one assembly
function + STATE_DIM as compile-time constant.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Removed the legacy non-branching Sequential q_network and target_network
from DQNAgent. These were never used (branching+dueling always active)
but allocated VRAM and ran noise resets every step. -190 lines.
Config tuning for dense micro-reward system:
- n_steps: 5→1 (TD(0), micro-rewards cancel over n>1)
- tau: 0.007→0.01 (faster target tracking for TD(0))
- c51_alpha_max: 0.5→1.0 (full C51, PopArt handles normalization)
- curiosity_weight: 0.1→0.0 (dense micro-reward replaces curiosity)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
target_dim expansion: adds raw_open (OHLCV) and mid_price_open
(MBP-10 midpoint at bar formation) to fxcache targets. FXCACHE_VERSION
2→3 for auto-rebuild. Legacy v2 files handled with close-price fallback.
Spec v5 adds 3 pearls:
- Bar duration encoding in Mamba2 (continuous-time SSM awareness)
- Order book center of mass from all 10 MBP-10 levels (aggression signal)
- Retrospective hold quality bonus (teaches exit timing)
Plus: Hold action (4th direction), DSR Sharpe EMA fix, counterfactual
magnitude/order sign fix, MFT mid-price mark-to-market.
OFI embed MLP now 18→10 (was 16→8). Mamba2 width SH2+10 (was SH2+8).
Attention width D+10 (was D+8).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
target_dim 4→6: adds raw_open (OHLCV) and mid_price_open (MBP-10
best bid/ask midpoint at bar formation) to the targets buffer.
Micro-reward now uses real intra-bar move (close - open) / atr
instead of close-to-close approximation. Also adds spread cost
awareness via |open - mid| penalty. Requires fxcache recomputation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Major changes from v1:
- Fix OFI data pipeline (ofi_gpu never wired, indices 42→66)
- Expand PORTFOLIO_STRIDE 30→38 for prev-OFI storage
- Add Mamba2 d_h_history backward (2 new cuBLAS GEMMs)
- Expose attention d_input_scratch for gradient flow
- Pre-compute OFI deltas in experience collection (state[74..82])
- Lower rank normalization threshold to 1e-5 for micro-rewards
- Use close-to-close return instead of unavailable open_price
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Dense per-bar reward using order flow momentum × price confirmation.
OFI embedding MLP (16→8 via cuBLAS) feeds into Mamba2 history AND
attention input. Replaces sparse exit-only reward with continuous
temporal signal for the SSM and attention heads to learn from.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
