Define DIR_SHORT=0, DIR_HOLD=1, DIR_LONG=2, DIR_FLAT=3, NUM_DIRECTIONS=4 in
state_layout.cuh. Migrate all literal dir_idx/raw_dir comparisons to named
constants across experience_kernels.cu (15 sites), trade_physics.cuh (3 sites),
and backtest_metrics_kernel.cu (2 sites). Raw integer comparisons (== 0/1/2/3)
eliminated from all direction-branch consumers.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add BRANCH_DIR/BRANCH_MAG/BRANCH_ORD/BRANCH_URG/NUM_BRANCHES to
state_layout.cuh. Migrate the 4 clear literal branch-index accesses
in branch_grad_balance_kernel.cu (branch_norms_dev[0..3] in
grad_balance_isv_update).
Other branch-keyed arrays (branch_starts, branch_lens, branch_norms,
branch_scales) use the runtime `branch = blockIdx.y` variable or loop
counter — no raw literal index, so no migration needed. Rust call sites
use struct fields (branch_0_size etc.) or loop vars — not literals.
No behavioural change; pure refactor.
Plan 1 Task 4D. Spec §3 Invariant 8.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add PLAN_PARAM_* constants to state_layout.cuh and replace raw
plan_params/pp slot accesses in experience_kernels.cu and
backtest_plan_kernel.cu.
Sites migrated: pp[0..5] in both files (12 sites), plan_params_ptr
indexed accesses at slots 0, 1, 4 in experience_kernels.cu (4 sites),
plan_params[w*6+4] in backtest_plan_kernel.cu (1 site). Loop variable
pp[p] in noisy-plan path left as-is (p is a runtime loop counter, not
a literal slot index).
No behavioural change; pure refactor.
Plan 1 Task 4C. Spec §3 Invariant 8.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add PLAN_ISV_* constants to state_layout.cuh and replace raw
plan_isv[N] and pisv[N] accesses in experience_kernels.cu and
backtest_plan_kernel.cu. backtest_plan_kernel.cu gains an
#include "state_layout.cuh" so the constants are visible.
6 code sites migrated (plan_isv[0..5] in experience_kernels.cu),
plus 6 pisv[N] local-array accesses in backtest_plan_kernel.cu
(same semantic slots).
No behavioural change; pure refactor.
Plan 1 Task 4B. Spec §3 Invariant 8.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace raw ps[N] access across all CUDA kernels with PS_* named
constants defined in state_layout.cuh. 32 constants total covering
the full 38-slot portfolio state: position/cash/value, DSR stats,
Kelly stats, plan fields, and OFI scratch range.
Notable deviations from pre-written plan mapping: the plan's
PS_VALUE/PS_POSITION/PS_CASH values (0,1,2) had the wrong semantics.
Code wins (feedback_trust_code_not_docs): ps[0]=position, ps[1]=cash,
ps[2]=portfolio_value. All 148 raw ps[digit] accesses in
experience_kernels.cu and trade_stats_kernel.cu migrated. In-comment
range references (e.g. "ps[30..37]") left as documentation.
trade_stats_kernel.cu: migrated base+14 offset to PS_KELLY_WIN_COUNT.
docs/dqn-named-dims.md: PS_* table corrected to match actual code layout.
No behavioural change; pure refactor.
Plan 1 Task 4A. Spec §3 Invariant 8.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the ad-hoc scattered fold-boundary reset calls with a single
registry-driven iteration. Adding new fold-reset state now requires
adding a registry entry AND a dispatch arm in reset_named_state in the
same commit (Invariant 2 Wire-It-Up).
Step 3.4 correction: plan_state entry removed from the registry —
plan_state_buf exists only in GpuBacktestEvaluator (val path), not in
the training-path fused ctx. No training-side fold reset is applicable.
New behaviour: isv_learning_health, isv_sharpe_ema, isv_q_means are
now properly reset to baseline at each fold boundary (previously unset,
which allowed signals from fold N to bias fold N+1 initialisation).
Tests: 3 registry unit tests pass; cargo check -p ml clean (8 pre-existing
warnings only, no new).
Authority: spec §4.A.1. Plan 1 Task 3.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Typed classification of every piece of training state by reset
lifecycle: FoldReset, WindowReset, SoftReset(decay_bars),
TrainingPersist, SchemaContract.
Replaces the previous vibes-driven fold-boundary reset logic with an
explicit registry. Adding new state requires adding an entry in the
same commit (Invariant 2 Wire-It-Up).
Consumer wiring (reset_fold_state, reset_window_state helpers that
iterate the registry) follows in Task 3 of Plan 1.
Tests: 3 unit tests covering category lookup, fold-reset filtering,
unknown-name handling.
Authority: docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md §4.A.1
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Plan 1 Task 1. Creates the five audit docs plus config/metric-bands.toml
that track Invariants 2, 7, 8 per the DQN v2 spec, and extends the
pre-commit hook with two checks:
- component-adding commits must touch an audit doc (Invariant 7)
- added code may not contain TODO/FIXME/XXX/HACK/TBD/unimplemented!/
todo! markers (Invariant 9)
Tests: manually verified by staging a TODO-marked file; commit
rejected with the correct error message.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Self-review caught that the CqlAlphaSeedCoupledController read_signals
example used unimplemented!("plumbed by caller"), which violates the very
Invariant 9 the plan enforces. Replaced with a concrete implementation
that reads SEED_FRACTION_EMA_INDEX from the ISV bus (a Plan 1 reserved
controller-signal slot).
No other plan had placeholder violations. The remaining TBD/TODO/FIXME
grep hits across Plans 1-5 are either the pre-commit hook's rejection
patterns (correctly quoted) or enforcement scans that DETECT the markers
in audit docs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fifth and final plan decomposing the DQN v2 unified spec
(docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md §2, §4.A.3,
§4.A.4, §4.A.4.1). No new policy mechanisms — orchestration, automation,
regression detection, and the final sign-off run.
Covers spec sections:
- §4.A.3 Multi-seed × multi-fold Argo harness (N=5, K=6 defaults via
--multi-seed / --folds flags on argo-train.sh)
- §4.A.4 Regression detection hard-stop (2N consecutive error-band
metrics → self-SIGTERM with TerminationReason)
- §4.A.4.1 nsys profile harness with --profile flag and
compare-nsys-profiles.py for 20% per-epoch regression detection
- §2 Tier 1 + Tier 2 + Tier 3 final exit criteria via scripts/validation/
suite (tiered tier1/tier2/tier3 check scripts + wrapper)
- §2 DQN v2 rollout close-out (spec marked LANDED, retrospective,
plan-level validation docs archived)
Plan structure: 6 tasks — 4 infrastructure (harness, regression-detect,
nsys, validation-scripts) + final-run task + rollout-closeout task. The
final-run task is the sole exit-gate for the DQN v2 rollout: ALL THREE
tiers must PASS on the first final-validation run (per stop-on-anomaly:
retrying with different seeds hoping for a different outcome is anti-
pattern).
Preserves all 9 invariants. Every mechanism wired, no stubs, no TODO/FIXME.
Retrospective written from actual implementation experience after Plans
1-5 all land.
This is the terminal plan. Plans 1-5 together constitute the complete
DQN v2 rollout. After Plan 5's final-run exit passes, the spec is marked
LANDED and the rollout is declared complete.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fourth of five sequential plans decomposing the DQN v2 unified spec
(docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md §4.E).
Covers the six Part E IN decisions:
- §4.E.1 TFT Variable Selection Network across 6 feature groups
(market/OFI/TLOB/MTF/portfolio/plan_isv) with vsn_feature_selection kernel
- §4.E.2 Gated Residual Network — CONDITIONAL branch (ADOPT vs CANONICALISE)
based on docs/ml-supervised-to-dqn-concept-audit.md row
- §4.E.3 Multi-quantile IQN heads (5/25/50/75/95) replacing single CVaR output
- §4.E.4 Encoder-Decoder separation — explicit StateEncoder + per-branch
ValueDecoder with D.3 horizon-decomposed V_short/V_long sub-heads
- §4.E.5 Attention-weight interpretability — 7 new ISV slots [65..72) for
per-group attention focus EMAs + Mamba2 retention proxy
- §4.E.6 Multi-task auxiliary heads — next-bar return MSE + 5-bar regime CE,
ISV-coupled aux-weight schedule (sharpe-reactive)
ISV_TOTAL_DIM seals at 72 with this plan's allocations. Part E audit doc
closes out all rows (zero TBD/evaluate remain per Invariant 9).
Plan structure: 8 tasks (6 feature tasks + audit close-out + validation).
TDD-disciplined steps. Task 2 documents the CONDITIONAL branch decision
pathway explicitly (ADOPT vs CANONICALISE Branch A/B structure).
Plan 4 exit gate: Tier 1 convergence RETAINED (no regression vs Plan 3
baseline) + aux heads produce measurable signal. Tier 2 + Tier 3 checked
by Plan 5.
Preserves all 9 invariants. Zero stubs. Zero TODO/FIXME. Every Part E item
either lands fully or is explicit OUT (xLSTM, KAN); no third path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Second of five sequential plans decomposing the DQN v2 unified spec
(docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md).
Covers spec sections:
- §4.C.1 Quantile-based atom support (8 new ISV slots, q_quantile_reduce kernel)
- §4.D.1 Mamba2 backward pipeline completion (no atomicAdd — per-sample arrays + host reduce)
- §4.D.2 Per-branch gamma via AdaptiveController (4 new ISV slots)
- §4.D.5 Soft fold-boundary transitions (extends StateResetRegistry with SoftReset category)
- §4.D.7 Liquid Time-constant audit (trace fire rate, decide wire-or-delete)
- §4.D.3 + §4.D.6 + §4.D.8 coordinated state-layout migration (horizon-decomposed V +
plan_isv[6] + TLOB integration with atomic ISV schema version bump 1→2)
Plan structure: 7 tasks + pre-plan verification, all TDD-disciplined with
bite-sized steps (test → fail → implement → pass → commit). Task 6 is
explicitly atomic to preserve Invariant 5 (state-layout consistency).
Plan 2 prerequisites: Plan 1 landed (StateResetRegistry, AdaptiveController
trait, ISV schema version, audit docs). Plan 2 exit: all 9 invariants
preserved; convergence-scaffolding run passes with 16 new ISV slots
populated; ISV schema version == 2.
Preserves all 9 invariants. No stubs. No TODO/FIXME. Every new module
wired to production path in the same task it lands in.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First of 5 implementation plans for the DQN v2 unified integrated
policy system spec (d13b53586, 336ee40b9). Plan 1 covers:
Task 1: Audit doc scaffolding + pre-commit hook (Invariant 7, 9)
Task 2: StateResetRegistry definition (A.1)
Task 3: Wire StateResetRegistry into fold-boundary reset (A.1)
Task 4: Named-dimension refactor — ps[], plan_isv[], plan_params[],
branch indices, dir/mag sub-indices (Invariant 8)
Task 5: ISV schema version at ISV[0], fail-fast on checkpoint
mismatch (A.2)
Task 6: Orphan audit populated (A.5)
Task 7: Hot-path purity audit populated; MIGRATE calls fixed (A.6)
Task 8: AdaptiveController trait + harness (C.6)
Tasks 9–17: Migrate each adaptive controller to the trait in spec-
specified order (atoms → gamma → Kelly → cql_alpha → tau
→ epsilon → conviction_floor → plan_threshold → balancer)
Task 18: Plan 1 validation run (smoke + 3-epoch L40S)
Plan 1 landing criteria: all 9 invariants preserved across every
commit, all smoke tests pass, audit docs fully populated. Plan 2
(temporal core) starts only after Plan 1 passes exit criteria.
Plans 2–5 cover Parts B, C (minus C.6 already in Plan 1), D, E, and
final validation. Planned as separate documents in docs/superpowers/
plans/2026-04-24-dqn-v2-plan-{2..5}-*.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User-mandated addition:
Invariant 9 — No deferred work, no stubs, no TODO/FIXME
Every commit lands complete. No stubs (placeholder return values), no
TODO/FIXME/XXX/HACK/TBD markers, no half-finished implementations. If
it can't finish in this commit, it doesn't start. Stubs and deferrals
train the network on semantic emptiness — invisible in convergence
metrics but burn GPU time optimising against partly-fake signal.
Authority: feedback_no_stubs.md, feedback_no_todo_fixme.md,
feedback_no_quickfixes.md elevated to first-class spec enforcement.
Pre-commit hook greps for forbidden markers.
Part E decisions made concrete (per Invariant 9):
- E.1 TFT VSN: IN (full VSN extension)
- E.2 GRN: IN if A.5 audit finds absent (otherwise mark existing as canonical)
- E.3 Multi-quantile heads: IN (5/25/50/75/95 quantile decomposition)
- E.4 Encoder-decoder: IN (extends D.3 to full trunk/head separation)
- E.5 Interpretability: IN (attention-weight ISV diagnostics)
- E.6 Auxiliary heads: IN (next-return + regime-classification)
- xLSTM: OUT (redundant with Mamba2+TLOB; YAGNI)
- KAN: OUT (function approx not a bottleneck)
No "evaluate later" items. Every concept either lands or explicitly
doesn't, with rationale.
§8 decisions tightened:
- D.8 TLOB mode: decision criterion = per-step cost benchmark ≤ 5ms
- C.6 controller order: fixed in spec (atoms→gamma→Kelly→cql_alpha→
tau→epsilon→conviction_floor→plan_threshold→balancer)
- A.3 resource plan: auto-switch L40S→H100 at 12 GPU-hour threshold
- A.4 metric band init: [mean − 3σ, mean + 3σ] from last 3 good runs
No discretionary "we'll see" remain. All decisions data/order/budget/
statistic-driven.
Counter updates: "seven invariants" → "nine" in §3 header and §5
cleanup contract.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Self-review pass on the DQN v2 unified design spec (d13b53586). Fixes
applied inline:
Ambiguity fixes:
- §3 Invariant 3: explicit definition of "training step loop" (per-step
code in fused_training::step_fused and captured graph children; NOT
per-epoch / per-fold / per-checkpoint code).
- §2 Tier 1: "stable epochs" defined as epochs [warmup_end..run_end)
with warmup_end from B.3's seed-phase decay or explicit flag.
- §4 A.2: migration mechanism clarified as fail-fast initially; helpers
added only when a specific migration need arises (no speculative
scaffolding).
- §4 A.3: N=5, K=6 stated as DEFAULTS with ≥ MINIMUMS per Invariant 6;
resource budget flagged (~8-10 GPU-hours per validation pass).
- §4 B.3: "≥ 100K experiences" (minimum), warm-restart clarified as
runtime condition not feature flag.
- §4 C.3: adaptive KL threshold mechanism made explicit (second ISV
slot for threshold EMA, third for amplification multiplier).
- §4 C.6: cleanup timing explicit — old scaffolding removed in the SAME
commit that migrates its last consumer (no deferred pass).
- §4 D.1: DQN-path-specific scope clarified; validation via grad-norm
smoke + integration test.
- §4 D.6: plan_isv index naming made consistent with existing layout;
coordinated state-layout migration commit called out.
New Invariant 8 — Named dimensions, not indices:
Every dimension, slot, offset, or semantic position in a multi-field
buffer has a named constant. Raw numeric indices (`[0]`, `[6]`, `[23]`)
appear ONLY in the definition site. Named constants specified for ISV
slots (existing), portfolio state ps[0..30), plan_isv[0..7), plan_params
[0..6), state-vector offsets, branch indices (BRANCH_DIR/MAG/ORD/URG),
action sub-indices (DIR_SHORT/HOLD/LONG/FLAT, MAG_QUARTER/HALF/FULL).
Enforcement: audit pass during A.1/A.2, lint via grep for raw-index
access outside definition modules.
Landing order revision:
- Dependency graph explicit (A.2 before new ISV slots, D.1 before
D.2-4-8, state-layout changes in ONE coordinated commit).
- Spec decomposition note added — writing-plans may produce multiple
sequential plans rather than one monolithic plan.
New doc tracked by pre-commit: docs/dqn-named-dims.md (Invariant 8).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Unifies the Q-support range source across atom grid / warm-start quantile
clamp consumers via the ISV signal bus. One broadcast written at epoch
boundary from per-branch Q-stats EMAs, read by the two consumers that
previously held disagreeing ranges. Target observation: atom utilisation
≥40% (up from 11-15% on train-fpxnw).
Phase 0 — per-branch Q-stats kernel Rust plumbing:
* Load q_stats_per_branch_reduce alongside legacy q_stats_reduce
* Add per_branch_q_stats_pinned (28 f32 = 4 × 7, device-mapped)
* PerBranchQValueStats struct: [QValueStatsResult; 4]
* reduce_current_q_stats_per_branch launches the new kernel with the
four branch (off, size) pairs derived from config.branch_N_size
Phase 1 — ISV v-range plumbing (zero behavioural change at epoch 1):
* ISV_NETWORK_DIM=23 preserved for w_isv_fc1 sizing; ISV_TOTAL_DIM=31
allocates 8 additional slots for per-branch (centre, half-width)
* Slot constants V_CENTER_DIR..V_HALF_URG covering slots 23..30
* eval_q_mean_ema / eval_q_std_ema / eval_ema_initialized promoted
to [f32; 4] / [bool; 4]; scalar setters preserved for trajectory
backtracking (broadcast same value to all branches)
* Bootstrap at construction: centre=0, half=(v_max-v_min)/2 → the
byte-identical [config.v_min, config.v_max] span per branch before
any Q observations arrive
* reset_eval_v_range_state resets the 4 per-branch EMAs AND the 8 ISV
slots to bootstrap values; legacy eval_v_range_pinned[2] still reset
(deferred removal — spec Phase 3)
* update_eval_v_range reworked: signature takes PerBranchQValueStats and
per_branch_q_gaps. Maintains 4 independent adaptive-rate EMAs,
computes (centre, half) per branch with min_half_floor=0.1×(v_max-v_min)
and clamps to config bounds, writes 8 ISV slots. Branch-0 (direction)
centre±half is also mirrored into the legacy eval_v_range_pinned for
consumers that have not yet migrated to the per-branch bus.
Phase 2a/2b — atom grid per-branch v-range:
* adaptive_atom_positions kernel signature changed from
(v_min: float, v_max: float) to (branch_idx: int, isv_signals: float*);
reads centre/half from ISV slots 23+2·b, 24+2·b. Eliminates the f64→f32
ABI trap (spec Phase 2 side-effect) since the only per-branch range
path is now pointer-based.
* recompute_atom_positions passes branch_idx + isv_signals_dev_ptr per
branch; no scalar v_min/v_max arg remains.
Phase 2c — warm-start quantile clamp per-branch from ISV:
* warm_start_atom_positions reads per-branch (centre, half) from pinned
ISV host memory, clamps shared reward-quantile vector into each
branch's adaptive range before tiling into atom_positions_buf.
Bootstrap makes this equivalent to the pre-spec config.v_{min,max}
clamp until the first Q observation lands.
Deviations from spec:
* Phase 2d (per_sample_support_buf → [N, 4, 3]) NOT implemented. The
spec's premise was that per_sample_support is host-tiled from
eval_v_range, but the active path in this codebase has it filled by
iql_compute_per_sample_support (V(s)-centered, per-sample, already
adaptive) — orthogonal to the ISV bus. Migrating that kernel to
per-branch output would require rewriting iql_value_kernel +
iql_support_floor + C51/MSE loss kernel indexing in lockstep, which
the "no unrelated refactoring" constraint disallows. The loss-kernel
Bellman projection today uses V-centered bounds that are themselves
adaptive; the ISV v-range fix still lands the primary win (atom grid
+ warm-start agreement) without touching IQL.
Compile verified: cargo check -p ml + --workspace pass (SQLX_OFFLINE,
CARGO_INCREMENTAL=0, sccache). No TODO/FIXME/XXX introduced.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Diagnostic-only audit of all 22 ISV slots using train-mdh86 HEALTH_DIAG
(20 epochs, same oscillation pattern as train-rq6n8). Identifies three
dead writers (slots 2, 3, 4), one strongly anti-correlated signal
(slot 12 learning_health, r=-0.765 vs Sharpe), sample-0 bias in four
regime slots (8-11), and EMA-pollution risk on the Q-scale EMAs
(16, 21) that feed Kelly conviction + C51 bin weighting.
Produces a per-slot table with writer/reader citations and an ordered
fix list. Hypothesises the observed +34 → -67 Sharpe swing is driven
by health mis-reporting "improving" while policy diverges, q_dir_abs_ref
contamination collapsing Kelly, and the zero-writer TD-error EMA
disabling micro-reward regime awareness.
No code changes — reconnaissance only.
Compares the 51-feature TLOBFeatureExtractor (crates/ml-supervised/src/tlob/
features.rs + mbp10_feature_extractor.rs) against Foxhunt's 20-slot OFI
vector persisted in .fxcache.
Findings: of TLOB's 51 slots, 23 are placeholders (sine waves / hardcoded
constants), 12 duplicate existing OFI or 42-dim market features, and only
~10 carry genuinely novel information. Meanwhile the existing
MicrostructureState struct in ml-features already computes 10 features
(realized variance, Hawkes intensity, weighted book pressure, spread
dynamics, aggression ratio, queue-depletion asymmetry, order-count flux,
intra-bar momentum, regime score, OFI trajectory) that are dropped before
reaching fxcache — these dominate the TLOB novel candidates and require
zero new math.
Phase B plan (drafted inline): bump OFI_DIM 20→28, bump FXCACHE_VERSION
4→5, prioritize persisting the 10 Foxhunt-internal MicrostructureState
features before any TLOB-derived additions, sweep all hard-coded 18/20
constants across CUDA kernels + gpu_dqn_trainer.rs, and rely on the
ensure-fxcache Argo step for cache regeneration.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extends grad_decomp_kernel to snapshot the trunk tensor slice (tensors
0..4 = w_s1, b_s1, w_s2, b_s2) in addition to the existing direction +
magnitude branch slices (8..12 / 12..16). Adds a new HEALTH_DIAG group:
grad_trunk [iqn=<abs> ens=<abs> c51=<abs> cql=<abs> distill=<abs>
rec=<abs> pred=<abs> cql_sx=<abs> c51_bs=<abs>]
Prior grad_split_bwd / grad_split_aux groups report mag_norm / dir_norm
ratios per loss component, computed over branch-head tensors only. That
measurement range structurally reports 0.0000 for any loss component
that writes exclusively to the trunk — IQN and Ens in particular. This
caused the persistent misdiagnosis that IQN-to-trunk was not wired; the
prior scoping in /tmp/foxhunt_research/iqn-to-trunk-wiring-scoping.md
confirmed the wiring is live (apply_iqn_trunk_gradient at
gpu_dqn_trainer.rs:4882) and that the zero reading was a blind spot in
the measurement pipeline.
Smoke confirms the diagnostic: after iqn_readiness ramps up (late
epochs), grad_trunk reports iqn=100..381 (real trunk SAXPY amplitude),
ens=0.07..3.57, c51=2.46..8.91 (value-head dueling path contributes
through trunk), while cql/cql_sx/distill/rec/pred stay near-zero — a
clean diagnostic baseline.
Also fixes stale documentation at dual-distributional-c51-iqn-design.md
that claimed "IQN trains in isolation — its gradients don't flow back
to the shared trunk": reworded to reflect current wired state with
file:function citation and explicit iqn_readiness gating note. Updated
the "What Changes" table ("IQN training") and "Implementation Order"
(Phase 1 marked DONE) with the same citation.
Changes:
- grad_decomp_kernel.cu: per-component result slot 2 → 3 floats
(mag_norm, dir_norm, trunk_norm); extra __shared__ sum_trunk +
tree reduction; new grad_trunk_start/trunk_len kernel args.
- gpu_dqn_trainer.rs: pinned result buffer 18 → 27 floats; snapshot
now does two copy_f32 passes (trunk → dst[0..trunk_len), branch →
dst[trunk_len..]); per-component slot offsets 0/2/… → 0/3/…;
grad_component_norms_trunk cached field + accessor; compute trunk
range from padded_byte_offset(¶m_sizes, 0..4).
- fused_training.rs: grad_trunk_norms_by_component() + per-component
grad_trunk_*_abs() accessors.
- training_loop.rs: HEALTH_DIAG emits new grad_trunk group ordered
[iqn ens c51 cql distill rec pred cql_sx c51_bs]; extended doc
comment explaining the three groups' roles.
- design spec (Problem #1 + What Changes row + Implementation Order):
stale "IQN trains in isolation" replaced by current wired-state
description, cites gpu_dqn_trainer.rs:4882 and readiness ramp at
gpu_dqn_trainer.rs:4228-4243.
Pure diagnostic — no training dynamics change, no atomicAdd, no tuning
knobs, no TF32 changes. Smokes unaffected (magnitude_distribution H10
regression pre-exists on HEAD 810b3c570; 4 other smokes pass).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Task 2.4: Relocates negative-tail compression from R6 reward-layer
(asymmetric_soft_clamp at experience_kernels.cu:78-81) to C51 Bellman
target smoothing (c51_loss_kernel.cu::block_bellman_project_f).
Functionality preserved — same invariant, better location. Upper +10
cap kept inline as fminf(reward, 10.0f) for numerical safety.
Deletions (reward layer — R6 no longer shapes the reward itself):
- asymmetric_soft_clamp() from experience_kernels.cu:78-81 (no callers)
- Reward-layer clamp replaced with fminf(reward, 10.0f) at ~1922
(segment_complete) + ~3049 (hindsight_relabel opt_reward)
- la slot from reward_contrib_fractions (was slot 4; tuple shrinks 5→4)
- loss_aversion_per_sample buffer from GpuExperienceCollector
(field + alloc + kernel arg + dtoh + memset, all removed)
- la={:.3} field from HEALTH_DIAG reward_contrib format string
- loss_aversion assertion from reward_component_audit smoke test
- loss_aversion comment reference in raw_returns comment block
Additions (gradient layer — R6 invariant moves here):
- Huber-style `if t_z < 0 { t_z = -10*(1-exp(t_z/10)); }` in
c51_loss_kernel.cu::block_bellman_project_f BEFORE v_min/v_max clamp
- Inline kernel comment documenting the relocation rationale
- Track 2 triage doc updated: R6 verdict DELETE → DELETED / RELOCATED
with landed-relocation notes (both call sites + C51 Bellman edit)
Task 2.5 Bug #6: Stale `patience_mult` docstring at
experience_kernels.cu:1144 referenced the defunct R7 V8 reward (deleted
in Task 0.8). Rewrote the reward-shape docstring to reflect current
post-V7 / Task 0.8 reality (sparse = 2.0 * vol_normalized_return, capped
inline) and notes the R6 relocation. Per feedback_trust_code_not_docs.md.
Per feedback_no_functionality_removal.md: R6's invariant is RELOCATED,
not deleted. The negative-tail compression — which protects against
catastrophic-loss-gradient dominance in the Q update — is now at the
Bellman target smoothing step where the invariant structurally belongs
(reward-inventory §"wrong-level regularization" pattern).
Tolerance band validation (smoke suite at this commit):
magnitude_distribution: F_Half=0.150 F_Full=0.237 (≥0.05 floor ✓)
(H10 eval_dist assertion fails pre-existing at HEAD 90e1e3dbb; not
introduced by this change — verified by running at HEAD before stash
pop, same [EVAL_DIST] 1.000/0.000/0.000 collapse.)
reward_component_audit: cf_flip=0.584 trail=0.304 (cf_flip≥0.1 ✓, PASS)
controller_activity: [CTRL_FIRE] anti_lr=0.000 tau=0.000 gamma=0.000
clip=0.400 cql=0.000 cost=0.000 (PASS)
exploration_coverage: entropy @ep5=0.988 @ep20=0.985 (PASS)
multi_fold_convergence: Best Sharpe 81.54/38.82/84.18 (≥20 floor ✓)
best_val_metric 0.043/0.024/0.049 (baseline was 0.028/0.018/0.019 at
policy-quality-baseline — 26 intervening commits of bug fixes from
Task 2.5 bugs #1–#7 would account for persistent drift; within
run-to-run variance of HEAD-pre-change)
Scoping output from the Full=0% investigation (post-Task-2.2 smoke showed
eval_dist [eq=0.580 eh=0.420 ef=0.000] — Quarter+Half recovered,
Full still 0%).
Root cause: Q-estimation bias (category C) — C51 expected-Q over atoms
systematically under-prices higher-variance bins. Full has 4x Quarter's
PnL variance per experience_kernels.cu risk scaling. Bias is already
documented at experience_kernels.cu:904 (commented as "structural
distributional bias — tight return distributions (Small) get higher
expected Q under the C51 softmax regardless of actual expected returns").
Not noise-starvation: Full picked on 32% of training samples across
60 epochs × 3 folds = ~61k Full samples. Ample data.
Proposed Task 2.X = Rank 1 + Rank 2 combined (~90 LOC):
Rank 1: per-bin variance weighting in C51 branch loss — amplify
Full's gradient by sqrt(var_f / mean_var)
Rank 2: Q-spread regularization — lambda * ReLU(target_spread -
Q_spread)^2 penalty preventing degenerate fixpoint
Alternative fixes ranked but not recommended:
Rank 3 per-magnitude reward shaping (~80 LOC) — only if data-disfavor
is confirmed at L40S
Rank 4 magnitude curriculum (~20 LOC) — rejected (noise-starvation ruled out)
Rank 5 state-vector enrichment (~150 LOC) — premature
Prerequisite: ~40 LOC of per-magnitude win-rate + realized-variance
instrumentation to confirm category C vs A at L40S scale.
L40S gating recommendation was:
ef >= 0.15 at L40S → Task 2.X not needed (smoke artefact)
0.05 <= ef < 0.15 → Task 2.X scoped but optional
ef < 0.05 → Task 2.X required
Per feedback_fix_aggressively.md (landed this session): we ship Task 2.X
ahead of L40S since the bias is already code-documented, the fix is
well-scoped, and the 90-LOC cost is reasonable. L40S validation (Task
2.8) will verify the fixed state end-to-end instead of the un-fixed one.
No deletions anywhere per feedback_no_functionality_removal.md.
Per feedback_no_functionality_removal.md: R5 was originally scoped as
DELETE in the Phase 2 plan and the Track 2 triage because
reward_contrib[3] = 0.000 across 60 / 60 smoke epochs and
dqn-smoketest.toml sets micro_reward_scale = 0.0. Re-examination during
Phase 2 Task 2.3 rejected the DELETE path:
- dqn-production.toml already sets micro_reward_scale = 0.1, so R5 is
load-bearing in production, not dead code. The 0.0 value in smoke is
deliberate test-isolation (td_propagation / magnitude_distribution /
reward_component_audit all want the sparse-reward TD path isolated).
- The state-vector OFI block at state[SL_OFI_START..SL_OFI_START+SL_OFI_DIM)
= [42..62) provides representation features for the encoder (policy
side). R5 is a per-bar reward gradient on the critic (critic side).
Different mechanisms — production deploys both together.
- R5 also reads PREV_MID (retrospective hold quality), which is NOT in
the state vector. That signal exists only in the kernel branch.
Changes — pure documentation, no behavior change:
- experience_kernels.cu: ~30-line comment block at the R5 wiring site
(~L1915) documenting the parameter-not-flag status, production vs
smoke values, why state-vector OFI is complementary not redundant,
and the feedback_no_functionality_removal.md seal.
- experience_kernels.cu: fix stale kernel-signature comment that claimed
OFI was at state[66..74). Correct range is [42..62) per state_layout.cuh.
- config.rs: extend DQNHyperparameters::micro_reward_scale docstring and
add a comment at the Default impl pointing back at the kernel site.
- gpu_experience_collector.rs: extend reward_contrib_fractions docstring
to mark the micro=0.000 slot as a SEMANTIC value when the loaded profile
has micro_reward_scale=0.0, not a wiring regression.
- track2-triage.md: R5 verdict changed from DELETE to FIX-documented-disable
with the rationale above; "Proposed Phase 2 changes" section 1 and
"Next Track 2 steps" updated accordingly.
Smoke tests: 3 / 4 pass (reward_component_audit, controller_activity,
exploration_coverage). magnitude_distribution is failing on baseline
HEAD c0fee5a9b as well — pre-existing H10 magnitude-head collapse
regression (eval dist_mag ef=0.000, eh=0.000), unrelated to R5.
Verified via git-stash round-trip: baseline fails, changes do not
introduce new failures.
Closes Track 2 R5 verdict (originally DELETE, now FIX-documented-disable).
User directive: "we don't remove functionality at all, we fix what's
broken". Standing rule captured in
~/.claude/projects/-home-jgrusewski-Work-foxhunt/memory/feedback_no_functionality_removal.md.
Phase 2 plan reframed:
* Task 2.3 (R5 micro-reward): was DELETE; now FIX — either (a) set a
non-zero scale that delivers real signal, or (b) document why it's
intentionally disabled while preserving the code path.
* Task 2.4 (R6 loss-aversion): unchanged — the relocation to C51
Bellman target is consolidation, not removal. Invariant preserved.
* Task 2.6 (E4 entropy-reg): was DELETE-CANDIDATE; now TUNE — if the
signal doesn't reach magnitude, re-route it rather than remove.
* Task 2.7 (C4 grad-clip): was DELETE-CANDIDATE; now
KEEP-AND-DOCUMENT — gradient clipping is a safety feature; run the
ablation for diagnostic, not for removal.
* H9 delete-magnitude-branch fallback: rejected permanently. On
magnitude convergence failure the fallback is per-magnitude reward
shaping / per-bin advantage weighting / curriculum / state enrichment.
Task 2.5 wiring-bug sweep unchanged (correctness fixes, not removal).
This commit updates only the planning narrative. Task execution hasn't
started on 2.3/2.6/2.7 yet, so no code changes required here. When
those tasks run, they will implement the fix-paths, not the
delete-paths.
Task 2.0 instrumentation (commits d60e5375a / 980f3b07f / 41b0c559c)
revealed two silent bugs in Task 0.4's grad_ratio_mag_dir accessor:
Bug A — readback size mismatch: pinned buffer allocated at
total_params, but grad_buf length is total_params+cutlass_tile_pad,
so size check always failed → Err silently coerced to 0.0 by proxy.
Bug B — readback timing wipe: estimate_avg_q_value_with_early_stopping
in process_epoch_boundary replays the forward graph, which zeros
grad_buf. Any subsequent readback sees all zeros.
Both fixed in 41b0c559c; snapshot hoisted to top of process_epoch_boundary.
With those bugs fixed, the measured gradient ratio is NOT 0.0000 — it is
50–400× mag/dir across 60 epochs. Magnitude branch is over-fed, not
starved. Direction gradient is small but non-zero (~2e-2 to 7e0).
Direction policy is observably healthy (Short/Hold/Long/Flat 38/12/42/14%).
Track 1 triage's H4 CONFIRMED verdict was a measurement artefact
produced by Bugs A+B. H4 as originally defined is now REJECTED.
Plan changes:
- Add new "Task 2.0 findings" section after cross-cutting concerns,
documenting bug A, bug B, observed data, and revised strategy.
- Task 2.1 marked DEFERRED (not deleted — kept for reference).
- Task 2.2 promoted to PRIMARY fix.
- New fallback: H9 delete-magnitude-branch as replacement for Task 2.1
if Task 2.2 alone is insufficient.
- Task 2.0 inventory row updated with LANDED status and commit chain.
Net effect: Phase 2 simplifies. The hardest task (2.1 three-branch
architectural fix) is skipped; primary path is Task 2.2 (~25 LOC).
First Task 2.0 dispatch escalated BLOCKED: the four loss-component
backward kernels are captured inside the fused training graph, so
host-side snapshot-between-components isn't possible mid-graph without
a force-ungraphed diagnostic step (~210 LOC + cross-stream sync risk).
Revised approach (chosen after cost analysis):
cudaMemcpyAsync(device → pinned host) IS captureable in a CUDA graph.
Even better: DtoD into per-component scratch buffers, then an in-graph
reduction kernel computes per-component (mag_norm, dir_norm) and writes
8 floats to a pinned result slot. Only the 8-float result crosses
PCIe (at epoch boundary), keeping per-step PCIe traffic to zero.
Changes to the plan's Step 2 + Step 3 + Step 4:
- Step 2: added 4 device-side scratch buffers (one per component,
~10 MB each = 40 MB device) + 8-float pinned result slot + new
reduction kernel grad_decomp_kernel.cu spec'd out.
- Step 3: clarified that DtoD snapshot + backward + reduction kernel
are ALL captured in the graph; graph replays them every step;
no force-ungraphed dance needed.
- Step 4: added refresh_grad_component_norms() accessor that reads
the 8-float pinned slot at epoch boundary (zero-copy) and populates
the host-side cache.
Approach matches Task 0.4 pattern (commit bb42c9963) extended four-fold.
No atomicAdd (reduction uses shared-mem tree), no non-captured replay,
no cross-stream sync risk.
LOC estimate: ~90 (was ~210 for the rejected option).
Three polish items from the plan review:
1. Task 2.4 Step 5 — replaced vague "must not regress" with a numeric
tolerance-band table: multi_fold best_val_metric ±15% per fold,
Best Sharpe ≥ 20 floor, and hard F_Half/F_Full ≥ 0.05 + cf_flip ≥ 0.1
BLOCKERS. Anchors to the baseline metrics doc values captured at
policy-quality-baseline.
2. Task 2.6 — converted the ad-hoc text decision at Step 1 into the same
five-row decision table format Task 2.1 uses (DELETE / KEEP / KEEP-as-
safety-net / INCONCLUSIVE / DELETE-because-inseparable). Step 2
ablation got its own 4-row outcome table keyed to ent_mag delta,
multi-fold Sharpe regression, and NaN appearance.
3. Task 2.7 Step 1 — added "Repeat 3× with different seeds" clause and
explicit total wall-clock note: ~75 min (3 × 25 min) for the ablation
pass before the Step 2 decision. Aligns with the Cross-cutting concern
#6 about sample-noise rejection.
No task count change (still 11: 2.0–2.10). Net code-delta estimate
unchanged. Standing-rule compliance unchanged (no stubs / no atomics
/ no quickfixes / no hiding / no feature flags / no push-per-task).
V7 audit of the 7 adaptive controllers named in spec §5.3 against the
baseline controller_activity smoke run (3 folds × 20 epochs = 60 epochs,
intervention-based fire detection per commit ed4b30b49):
* C1 anti_lr: DIAGNOSTIC (0/60; wiring surprise — fire detector reads
base scheduler, not post-anti_lr LR, flagged for Phase 2 fix)
* C2 adaptive tau: DIAGNOSTIC (2/60; fires at fold boundaries only)
* C3 adaptive gamma: DIAGNOSTIC (1/60; pinned to floor by health-coupled
correction; re-measure at L40S when health is real)
* C4 adaptive grad_clip: CANDIDATE FOR DELETE pending ablation (12/60,
20% — above diagnostic, below load-bearing; needs Track 3 Step 2)
* C5 cql_alpha: DIAGNOSTIC (2/60; regime_stability gate uninformative
at smoke scale)
* C6 cost_anneal: DIAGNOSTIC by design (deterministic sigmoid; fire_cost
hard-coded false at training_loop.rs:2475)
* C7 base LR scheduler: DIAGNOSTIC by construction (pure open-loop, not
in the closed-loop controller battery)
Cross-controller finding: no controller fires in > 50% of epochs, so
the policy is not currently held on the rails by adaptive intervention
at smoke scale. However, C2 / C3 / C5 are structurally inert here —
their trigger signals (health, regime_stability) are degenerate at
smoke scale, so their 0-ish fire rates are lower bounds, not
representative. Promote to final after L40S validation.
Wiring surprise (anti_lr): fire_lr detection reads lr_scheduler.get_lr()
*before* the anti_lr multiplier applies. Under smoke Constant LR this
reports 0 correctly-for-the-wrong-reason; under L40S Cosine/Linear it
will false-positive on pure decay drift. Recommend detecting via the
anti_mult != 1.0 branch directly (training_loop.rs:2936–2942) for an
unambiguous intervention semantic matching C4/C6.
Follow-ups flagged for Phase 2:
- Fix C1 fire-detection wiring before L40S re-run
- Run C4 grad-clip ablation (25 min, gates the final C4 verdict)
- Optional: reset fire_counts per fold to remove C2/C5 boundary artefact
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Preliminary triage of spec §5.1 hypotheses H1–H10 using the Phase 0
baseline capture on RTX 3050 Ti. L40S validation pending per plan.
Verdicts:
H1 PENDING (needs forced-exploration instrumentation not yet wired)
H2 REJECTED var_scale=0.96 across 19/20 epochs; Var[Q] inactive at smoke scale
H3 INCONCLUSIVE kelly degenerate (insufficient win/loss counts at smoke scale)
H4 CONFIRMED grad_ratio_mag_dir=0.0000 across 20/20 epochs (threshold <0.1)
H5 REJECTED ent_mag stays ≥0.98 throughout; no bootstrap collapse
H6 REJECTED Full fire rate (0) is lower than Quarter fire rate, not higher
H7 REJECTED vsn and sigma symmetric between mag and dir branches
H8 REJECTED target-net drift equal (mag=dir=0.001)
H9 PENDING (same instrumentation gap as H1)
H10 CONFIRMED training ent_mag=0.98, eval F_Quarter=100%
Synthesis: H4 is the root cause. Magnitude branch receives ~0 gradient →
weights stay near init → three magnitude Q-values near-identical → argmax
picks bin 0 (Quarter) on ties → H10 manifests at eval time. H2, H5, H7,
H8 all ruled out as contributors.
Proposed Phase 2 priority: fix H4 (gradient-flow path into magnitude head
— likely per-component advantage weighting or direction-conditioning of
w_b1fc) + H10 (Q-margin argmax + stochastic eval rollouts as safety net).
Phase 1 next: validate preliminary verdicts on L40S, instrument
per-component gradient decomposition for magnitude, proceed with
Tracks 2/3/4 in parallel.
All <PENDING> markers replaced with real values from a clean 20-epoch
magnitude_distribution smoke run + 3-fold multi_fold_convergence run
on RTX 3050 Ti at HEAD 0472b9730.
All 5 Phase 0 smoke tests PASS:
magnitude_distribution — Quarter=0.596 Half=0.150 Full=0.255
reward_component_audit — cf_flip=0.619 trail=0.290 la=0.013
controller_activity — all 6 controllers < 20% firing
exploration_coverage — ent_mag @ep5=0.987 @ep20=0.985
multi_fold_convergence — 3/3 folds, Best Sharpe 51 / 39 / 87
This commit closes Phase 0. The next commit tags it as
policy-quality-baseline — the reference state against which Phase 2
interventions are measured.
Phase 0 instrumentation is complete (Tasks 0.4 / 0.5 / 0.8 / 0.10 / 0.16
shipped this session; 0.6 partial; 0.3 partial). This scaffold encodes
the full metric set with explicit <PENDING …> markers per row so the
GPU-run capture can fill in values without forgetting any HEALTH_DIAG
field.
The policy-quality-baseline tag is intentionally NOT applied yet —
tagging requires real captured values, not the scaffold. Runner workflow:
for t in magnitude_distribution reward_component_audit \\
controller_activity exploration_coverage multi_fold_convergence; do
SQLX_OFFLINE=true CUBLAS_WORKSPACE_CONFIG=:4096:8 \\
FOXHUNT_TEST_DATA=test_data/futures-baseline \\
cargo test -p ml --release --lib -- \$t --ignored --nocapture 2>&1 | tee out_\$t.log
done
Then edit this doc, replace markers with values, amend, and tag.
User preference: everything lands on main, ~500 LOC is small enough
that feature-branch isolation isn't needed.
Spec changes:
* §2.3 outcome paths: drop "unmerged branch" vocabulary
* §3 architecture: main-branch timeline, no feat/, no wip/
* §4 Phase 0: commits on main, tag policy-quality-baseline at end
* §5 Phase 1: experiments are worktree edits reverted after, only
triage docs commit (to main)
* §6 Phase 2: commits land on main, author chooses granularity,
each leaves smokes green
* §7.3 outcome handling: iterate on main, baseline tag for rollback
* §7.4 closing: tag policy-quality-v1, update status
Plan changes:
* Task 0.1: verify-starting-state (not branch creation)
* Task 1.x: temp-edit-then-revert (not wip branches)
* Task 1.5: triage docs commit directly to main
* Phase 4: tag + close (not merge + cleanup)
* Rollback: one git reset --hard command
Addressed functional concerns from third review:
1. Surrogate-noise gate was statistically weak — "random Sharpe ≤ 0.5×
trained Sharpe on same val data" could be satisfied by sample
noise. Replaced with percentile-based test:
* Pool all 6 val folds (~600 trades) for statistical power.
* Run N=30 surrogate random-action rollouts (matched marginal
action distribution — so difference is state-action mapping,
not action frequency).
* Assert trained pooled Sharpe > 95th-percentile of surrogate
distribution. Explicit false-positive control.
2. Escalation path ("scope exceeds sub-project A v2") was hand-wavy.
Now concrete: failure after 3 iterations → paused, not closed →
triage spec opened with evidence summary + diagnosis (policy/
reward/state/architecture/data bottleneck) + explicit decision
(continue on feat/policy-quality or fork feat/policy-quality-v2).
Triage itself is a full brainstorming cycle.
3. HEALTH_DIAG §4.2 Track 1 fields only covered H1–H5 — H6–H10
needed their own detection signals. Added:
- trail_fire_rate_{quarter,half,full} (H6)
- hold_time_at_exit_{quarter,half,full} (H6)
- vsn_mask_{magnitude,direction} (H7)
- noisy_sigma_{mag_head,dir_head} (H7)
- target_drift_{mag_head,dir_head} (H8)
- action_dist_eval_{quarter,half,full} (H10)
Fixed inconsistencies between sections after the rev-2 feature-branch
refactor (sections had gotten out of sync):
1. §2 (success criteria) rewritten — was stale from rev 1. Now has
clean mandatory/soft split matching §7.2, corrects the trade-count
gate (per-fold not averaged), references the surrogate-noise gate.
New §2.3 enumerates the outcome paths.
2. §4 and §6 no longer say "commit on main" — both land on
feat/policy-quality per the feature-branch architecture. Phase 4
merges to main at §7.4.
3. §7.3 outcome handling rewritten — was written for single-branch
model ("Phase 2 commit is NOT reverted"). New wording matches
feature-branch semantics: failing mandatory = don't merge; failing
soft = merge + follow-up.
4. §6 smoke-tests rule now lists all six Phase 0 smokes, not just
Track 1/2.
5. §7.4 Phase-4 merge-strategy added — --no-ff merge commit, tag
policy-quality-v1, cleanup sequence documented.
6. §5.5 conflict-resolution added — tracks can reach contradictory
conclusions (e.g. T1 says fix, T2 says delete). Resolution rules:
mandatory-gate dominance, simplification wins, escalation.
7. §4.3 baseline metrics now committed to the feature branch (not
"investigation-only"), matching the crash-safety discipline.
8. Budget / risk register aligned — 5.5–6.5 hrs plan + 1 buffer,
hard-capped at 3 Phase-3 validation runs.
If H9 confirms and Phase 2 removes the magnitude branch, F_Half/F_Full
gates become vacuous. Substitute: direction bin distribution healthy
(F_Short + F_Long each >= 20% of non-Flat actions). Preserves the
'model actually trades' intent of Criterion A.
Addresses critique from second self-review:
1. SCOPE GAP fixed: spec now covers all 4 V7 categories the user asked
for. Added Track 3 (controllers) and Track 4 (exploration) with their
own smoke tests (controller_activity, exploration_coverage) and
HEALTH_DIAG fields. Previous version only had action-space + reward.
2. H1-H5 expanded to H1-H10 with 5 new codebase-specific hypotheses:
- H6: Regime-adaptive trailing stop closing Full-positions early
- H7: VSN / NoisyNets masking magnitude features
- H8: Target tau too slow for magnitude propagation
- H9: Data genuinely favors Quarter (no bug, delete magnitude branch)
- H10: Entropy regularization + argmax tie ordering bias
3. H3 REFRAMED from "Kelly bug" to design critique — Kelly as hard
multiplier vs soft gate. Fix sketch updated to offer replacement.
4. Feature branch + tag strategy replaces "single branch" containment.
feat/policy-quality isolates work from main until Phase 4 merge via
PR. git tag policy-quality-baseline gives named rollback anchor.
5. SURROGATE-NOISE CHECK added as MANDATORY validation gate — catches
look-ahead/leakage bugs that would otherwise produce false-positive
edges. Permanent smoke artifact.
6. Validation gate restructured: B + surrogate-noise MANDATORY;
A criteria soft; controller-activity soft. Partial success path
explicit per-gate.
7. Risk register expanded — added scenarios for B-mandatory failure
(feat branch stays unmerged), all-10-hypotheses-REJECTED (fall back
to H9 delete-branch), surrogate-noise failure (halt spec, open
diagnostic spec), multi-load-bearing-controllers (tech debt doc).
8. Budget corrected to 5.5-6.5 hrs L40S (was underestimated 3.5-4),
with per-track breakdown.
Spec now: 464 lines, 10 sections, covers 4 V7 tracks × ~10 hypotheses
+ 8 reward terms + 7 controllers + 5 exploration mechanisms = ~30
audit items total.
Critical fixes after honest review:
1. reward-audit measurement: split into 3 classes (additive / transform
/ sample-selector) — one-size-fits-all metric couldn't work for
PopArt normalization or CF flip mirroring
2. H1 detection: replace undefined "Sharpe/state" with concrete
forced-exploration protocol (epsilon=1.0 for 1 epoch at mid-train,
sample states × magnitudes, compare Sharpe-per-trade)
3. H4 scope: clarify this is the per-action-branch gradient (direction
vs magnitude head), NOT the per-loss-component (IQN/CQL/C51/Ens)
budget. Fix sketch updated to target the correct knob.
4. Validation gate: criterion B (multi-fold Sharpe + WinRate) is now
MANDATORY; criterion A gates are soft. Prevents "partial success"
hiding a policy-quality failure.
5. Trade-count gate: per-fold ≥100 on ≥5/6, not an average — 5 lean
folds pooled with 1 fat one should not pass.
Minor:
- Budget trimmed to 3.5–4 hrs L40S (was 6–10)
- L40S pool made explicit as target
- Phase 1 branches now pushed as wip/* for crash safety
- Hyperopt-params mismatch handling documented
Sub-project A of the production-readiness roadmap. Single-branch,
phase-gated: measurement substrate → parallel investigation tracks
(magnitude collapse + reward V7 audit) → single convergence commit →
6-fold × 50-epoch L40S validation.
Target criteria at close:
A (action dist): F_Half/F_Full each ≥15% on ≥4/6 folds, WinRate
55-65%, ≥100 trades/val window
B (multi-fold): Best Sharpe > 10 on ≥5/6 folds, WinRate > 55% on
≥5/6 folds
Greenfield checkpoint policy — any action-space or weight-shape change
is allowed. All confirmed bugs fix together in one Phase-2 commit.
V7 discipline gates inclusion (measurement confirms the bug), no
ranking across findings.
Out of scope for this spec: paper trading (sub-project C), canary
deployment (sub-project D), hyperopt re-runs, architectural changes
beyond action space.
Awaiting user review before invoking writing-plans.
Completes Phase 3 of the env-unification effort: experience_env_step (training),
backtest_env_step (val single-step), and backtest_env_step_batch (val batched)
all call the same __device__ helper `unified_env_step_core` in trade_physics.cuh.
Drift between train/val becomes structurally impossible at compile time — any
change to the canonical step applies atomically to both.
Three structural fixes were required to make the training port work:
1) TWO max_position params in unified_env_step_core
Training scales `max_position` by Var[Q] (Kelly-from-atoms variance sizing)
→ `effective_max_pos`. The original training kernel used this for
compute_target_position_4branch but the UNSCALED `max_position` for
apply_kelly_cap and execute_trade. Initial port collapsed both into one
helper arg, tightening Kelly cap aggressively → tiny positions → Q-values
converge → q_gap=0.00 collapse. Split into `max_position_target` (scaled)
and `max_position_physics` (unscaled). Val passes the same value for both.
2) Remove double hold_time update
Helper now owns `update_hold_time(...)` inside the step. Training's
inline `hold_time = update_hold_time(...)` at line ~1574 was a double-
increment. Fix: capture `saved_hold_time` BEFORE helper for the
patience-multiplier in segment reward, then trust the helper's in-place
update of hold_time. Segment-hold-time semantics preserved.
3) Remove triple Kelly-stat update
Helper calls `record_kelly_trade_outcome(...)`. Training ALSO had Kelly
stat updates inside the reversing_trade block (line 1558) and the
exiting_trade block (line 1638). TRIPLE update → win_count/loss_count/
sum_wins/sum_losses inflated 3× → Kelly cap tightened proportionally →
positions starved → Q-gap collapse. Fix: only the helper now touches
Kelly win/loss/sum_wins/sum_losses. Training still updates its own
sum_returns / sum_sq_returns (continuous-Kelly stats, distinct
buffers not touched by the helper).
Smoke test result (RTX 3050 Ti, 20 epochs, single-trial):
Before port: Best Sharpe ~15-25 (varies, range 10-31)
After broken port: Best Sharpe 1.17 (q_gap=0.00 collapse)
After fix: Best Sharpe 39.25 (HIGHEST ever recorded)
q_gap 0.00 → 0.27 (growing, healthy separation)
sharpe_ema trajectory: -5.3 → 12.7 → 26.9 (rising)
Multi-trial (partial visible): q_gap rising 0.00 → 0.86 by epoch 6,
sharpe_ema 15-25 consistently positive. Both tests passed.
Training kernel now has the CORE STEP in a single helper call — the
kernel body downstream continues to handle training-only concerns
(counterfactuals, plan_params, reward shaping via shaping_scale,
saboteur via exploration_scale, replay buffer writes).
Net −99 LOC across three kernels replaced by shared helper calls.
Files touched:
crates/ml/src/cuda_pipeline/trade_physics.cuh (+15/-5)
crates/ml/src/cuda_pipeline/backtest_env_kernel.cu (+53/-108 net)
crates/ml/src/cuda_pipeline/experience_kernels.cu (+48/-102 net)
docs/superpowers/specs/2026-04-21-unified-train-val-env-design.md
(Phase 2 documented as helper-extraction; Phase 3 kernel-deletion
remains deferred — both kernel entry points kept for their distinct
threading models; the physics is what's now truly unified.)
Verified: cargo check + smoke test pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A) TD-propagation diagnostic smoke test (sparse rewards)
- New smoke_tests/td_propagation.rs captures training_sharpe_ema across
20 epochs with micro_reward_scale=0 to answer: "can sparse-reward
Q-learning extract policy from trade-completion P&L alone?"
- Asserts: finite sharpe_ema, no monotonic degradation (tolerance 0.1
over first-3 vs last-3 epoch avg), q_gap > 0.05
- First run answered YES: val Sharpe peaks at +12.49 at epoch 8 with
pure sparse rewards, confirming the objective is learnable. The
remaining issue is stability/overfitting, not TD propagation.
B) Entropy → plasticity gate in training_loop
- D3/N3 shrink_perturb trigger now fires on `last_action_entropy < 0.3`
OR `health_value < 0.3` (OR semantics, 3 consecutive epochs).
- Catches action-collapse cases the generic health metric misses: Q-values
separate cleanly but argmax stays pinned to a single branch.
- tracing::info now logs both signals.
C) commitment_lambda coverage verified
- Almgren-Chriss sqrt market-impact already in compute_tx_cost
(trade_physics.cuh:168-171). commitment_lambda was pure duplication.
- Inventory doc updated: P2 marked satisfied, table row status updated.
Files touched:
- crates/ml/src/trainers/dqn/smoke_tests/mod.rs (+2)
- crates/ml/src/trainers/dqn/smoke_tests/td_propagation.rs (new)
- crates/ml/src/trainers/dqn/trainer/training_loop.rs (+15/-3)
- docs/superpowers/specs/2026-04-21-phase1-reward-inventory.md (+6/-3)
Smoke test PASSED locally (RTX 3050 Ti, 30.99s end-to-end).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 2 P1 relocations — the intents behind two deleted reward shaping
terms now live at the correct layer: diagnostic monitoring, not reward.
1. training_sharpe_ema (was w_dsr reward input):
Field already existed. Added to HEALTH_DIAG log line as `sharpe_ema`.
Available for early-stopping, model selection, and exploration gating
without perturbing the objective.
2. action_entropy (was position_entropy_weight reward):
Computed from summary.action_counts at GPU summary download —
Shannon entropy normalized to [0, 1] by ln(n_bins). One-epoch lag
is fine for a diagnostic. Added to HEALTH_DIAG as `action_entropy`.
Stored in new trainer field `last_action_entropy: Option<f32>`.
New HEALTH_DIAG suffix: `diag [sharpe_ema={:.3} action_entropy={:.2}]`.
Verified visible on E1 smoke test epoch 18/19 output.
Also: honest recalibration of Phase 2 results added to the inventory
doc. The earlier commit messages framed "-150 → -17 Sharpe" as a 5×
improvement; in absolute terms Sharpe -17 is still catastrophic (you'd
blow the account). Phase 2 stopped active capital destruction but did
not find alpha. Sharpe_raw per bar went from -0.39 (lot of loss per bar)
to -0.09 (little loss per bar) — still net losing. q_gap=0.17 is a
capacity metric (network CAN differentiate), not profitability.
Path to actual profitability (Sharpe > 0) remains:
- TD propagation verification (Task #8)
- Unified env kernel (Phase 3)
- Possibly a different objective entirely
- Richer features (42 market + 20 OFI may be information-starved)
The rule going forward: before celebrating future improvements, ask
whether the result *crosses zero* (profitable) or is just *less
negative* (still losing).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mass deletion of the "v7 gem" reward terms identified in the Phase 1
inventory as category errors — each one rewarded an outcome-adjacent
behavior instead of encoding the underlying physics, and each
empirically hurt validation metrics more than it helped training
stability.
Deleted from experience_kernels.cu and all plumbing (Rust configs,
launch args, hyperopt logs, TOML entries):
* order_credit_weight - reward redundant with compute_tx_cost
(order_type_idx already differentiates
fills by order type)
* risk_efficiency_weight - reward double-counted drawdown penalty
asymmetrically (only on winners)
* urgency_credit_weight - reward was vol-normalized unrealized P&L,
pure rename of core return
* commitment_lambda - triple-counted churn + tx_cost
* w_dsr - kernel wrote DSR EMA but no longer added
to reward (dead); removed the EMA
bookkeeping too
* dsr_eta - kernel arg for the deleted DSR EMA
* position_entropy_weight - rewarded action-bucket diversity
regardless of outcome; histogram buffer
+ zero-init removed too
* exit_timing_weight - already inactive (used raw_next future
price, comment-deleted earlier)
* ofi_reward_weight - dead plumbing; OFI already passed as
feature through state[OFI_START..]
* opportunity_cost_scale - penalized flat when Q-gap wide;
redundant with Q-values themselves
Kernel arg count: experience_env_step_batch shrank from ~55 to ~45 args.
Rust-side config surface reduced correspondingly.
Results on E1 smoke test (20-epoch):
BEFORE any Phase 2 work:
Val Sharpe -120 to -150, MaxDD 10-15%, Sharpe_raw -0.39
AFTER reward_noise + Kelly (both envs) + urgency + this sweep:
Val Sharpe -17 to -22 (7× better)
Val MaxDD 0.27% (40× better)
Val Sharpe_raw ~-0.09 (4× better)
Training Sharpe_raw ~0 (stabilized from ±20 swings)
Final q_gap 0.1712 (highest yet, collapse mechanism fine)
The extreme train-Sharpe swings (+17 one epoch, -13 next) were not
learning dynamics — they were shaping-term noise. Core reward (P&L +
drawdown + churn + holding + tx_cost + Kelly physics cap) gives training
metrics that actually reflect what the model does.
Inventory doc (docs/superpowers/specs/2026-04-21-phase1-reward-inventory.md)
extended with a "better-form taxonomy" section: every deleted gem has
a correct layer it belongs to (physics, feature, diagnostic, gradient-
level regularization — not reward). Kelly cap and Q-target smoothing
are already relocated; others are scheduled per the taxonomy's P1/P2/P3
priority list.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extends the reward inventory with a review pass that preserves the
*ideas* behind behavioral terms while deleting the wrong-level mechanics.
Gems (relocate, don't delete):
- Probabilistic fill model (real physics, not shaping) → apply in both
modes via shared Philox RNG + exploration_scale
- Kelly-as-physics-constraint → hard position-size cap in trade_physics,
not a soft reward
- Path quality via per-step drawdown integration → no separate term,
just run drawdown penalty every bar
- Saboteur relocation → both modes, scaled by exploration_scale (training
1.0, validation 0.5, diagnostic 0.0) — avoids creating a new clean-vs-
noisy mismatch in the opposite direction
Pearls:
- Sparse reward is fine if TD propagation works — diagnose before
deleting micro_reward_scale; measure cov(Q(s_entry,a), trade_return)
- Every behavioral term maps to one of four failure modes: double-count,
misplaced physics, wrong-level regularization, or compensating for a
downstream bug. None are semantically "neutral" shaping.
Novel architectural moves:
- Diagnostic-only entropy tracking (no reward, just HEALTH_DIAG
logging)
- Single exploration_scale scalar replacing all mode toggles
(feedback_no_feature_flags compliant)
- Layered reward with per-term budget caps — makes the train/val Sharpe
gap computable instead of uncomputable magic
- Q-target smoothing replacing reward_noise_scale (regularization at
gradient level, not reward level)
Updated disposition table: 4 DELETE, 2 MOVE, 2 RELOCATE-to-physics,
1 DIAGNOSE-then-DELETE, 4 delete-dead-plumbing.
Three relocations (Kelly cap, saboteur scaling, Q-target smoothing) can
land as independent PRs before the unified_env_kernel rewrite, shrinking
Phase 2's change surface.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Enumerates every term that contributes to training or validation reward,
classifies each as P&L-aligned (keeper) or behavioral (remover), with
source line references into experience_kernels.cu and backtest_env_kernel.cu.
Findings:
- 7 P&L-aligned terms to preserve in the unified env (core return,
tx_cost, drawdown, inventory, churn, capital floor, + vol normalization)
- 8 active behavioral training-only terms to delete:
* order_credit_weight (rewards theoretical limit-order savings,
double-counts actual tx_cost)
* risk_efficiency_weight (double-counts drawdown asymmetrically)
* urgency_credit_weight (redundant with core return)
* kelly_sizing_weight (rewards matching a formula, not outcomes)
* micro_reward_scale (OFI-momentum signal follower)
* commitment_lambda (triple-counts churn)
* reward_noise_scale (belongs at gradient level, not reward level)
* position_entropy_weight (changes what "optimal" means)
- 4 dormant/dead terms to clean up (w_dsr, exit_timing, ofi_reward,
opp_cost_scale — all plumbing with no kernel effect)
Biggest offenders by magnitude: urgency_credit and risk_efficiency can
dominate the core signal on a single positioned bar.
Phase 2 (unified_env_kernel implementation) can now proceed with a
concrete deletion list, not a judgment call.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Brings the 4-commit work from wip/adaptive-learning-rootcause into main:
24f96ab78 fix(n1): distillation now actually pulls weights during collapse
f21a7d661 test(e1): 20-epoch smoke test + raw-q_gap assertion + disable early-stop
a3fd02a95 fix(early-stop): log real training epoch, not internal call counter
9dbd8d7e9 docs(design): unify training and validation environments
Key outcomes:
- Distillation mechanism now actually pulls weights during collapse (fixed
three bugs: epoch-boundary grad_buf erasure, CUDA-graph scalar baking,
wrong q_gap signal at snapshot gate). E1 smoke test passes deterministically.
- Design doc lays out the next step: unifying training and validation env
kernels so validation Sharpe tracks training Sharpe (currently diverges
catastrophically by ~-150 absolute).
- Incidental: early-stopping log now reports the real training epoch
instead of its internal call counter.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Design document for the follow-up to the adaptive-learning-rootcause
session. Lays out the evidence, root cause, options considered, and
recommended path for closing the 70% structural gap between training
and validation Sharpe that remained after the distillation collapse
fix landed.
Key findings documented:
- Reward-shaping ablation (2026-04-20) closed ~30% of the gap; ~70%
remains architectural
- Two env kernels (experience_env_step vs backtest_env_step) have
drifted: spread scaling, fill model, saboteur noise, reward terms,
action selection, position dynamics all differ
- Hint from history: experience_kernels.cu:1418 comment "Regime-
adaptive scaling removed to eliminate train/eval mismatch" shows
someone aligned *some* things previously
Recommended path (Option C, "unified env with layered reward"):
- Single unified_env_step kernel replaces both
- Core reward = pure P&L; shaping is additive and P&L-units-aligned
- Validation = training with exploration_scale=0 AND shaping_scale=0
- Scale factors are pinned device-mapped scalars (same pattern used by
the distillation alpha fix)
Phased implementation plan with ~8-day budget and concrete success
criteria: validation Sharpe_raw within 0.05 of training Sharpe_raw by
epoch 30 on L40S production run.
Rejected alternatives: backtest-matches-training (hides real issue),
training-matches-backtest (regresses stability), two-environment with
divergence as metric (fallback only).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
