Phase 0 instrumentation is complete (Tasks 0.4 / 0.5 / 0.8 / 0.10 / 0.16
shipped this session; 0.6 partial; 0.3 partial). This scaffold encodes
the full metric set with explicit <PENDING …> markers per row so the
GPU-run capture can fill in values without forgetting any HEALTH_DIAG
field.
The policy-quality-baseline tag is intentionally NOT applied yet —
tagging requires real captured values, not the scaffold. Runner workflow:
for t in magnitude_distribution reward_component_audit \\
controller_activity exploration_coverage multi_fold_convergence; do
SQLX_OFFLINE=true CUBLAS_WORKSPACE_CONFIG=:4096:8 \\
FOXHUNT_TEST_DATA=test_data/futures-baseline \\
cargo test -p ml --release --lib -- \$t --ignored --nocapture 2>&1 | tee out_\$t.log
done
Then edit this doc, replace markers with values, amend, and tag.
User preference: everything lands on main, ~500 LOC is small enough
that feature-branch isolation isn't needed.
Spec changes:
* §2.3 outcome paths: drop "unmerged branch" vocabulary
* §3 architecture: main-branch timeline, no feat/, no wip/
* §4 Phase 0: commits on main, tag policy-quality-baseline at end
* §5 Phase 1: experiments are worktree edits reverted after, only
triage docs commit (to main)
* §6 Phase 2: commits land on main, author chooses granularity,
each leaves smokes green
* §7.3 outcome handling: iterate on main, baseline tag for rollback
* §7.4 closing: tag policy-quality-v1, update status
Plan changes:
* Task 0.1: verify-starting-state (not branch creation)
* Task 1.x: temp-edit-then-revert (not wip branches)
* Task 1.5: triage docs commit directly to main
* Phase 4: tag + close (not merge + cleanup)
* Rollback: one git reset --hard command
Addressed functional concerns from third review:
1. Surrogate-noise gate was statistically weak — "random Sharpe ≤ 0.5×
trained Sharpe on same val data" could be satisfied by sample
noise. Replaced with percentile-based test:
* Pool all 6 val folds (~600 trades) for statistical power.
* Run N=30 surrogate random-action rollouts (matched marginal
action distribution — so difference is state-action mapping,
not action frequency).
* Assert trained pooled Sharpe > 95th-percentile of surrogate
distribution. Explicit false-positive control.
2. Escalation path ("scope exceeds sub-project A v2") was hand-wavy.
Now concrete: failure after 3 iterations → paused, not closed →
triage spec opened with evidence summary + diagnosis (policy/
reward/state/architecture/data bottleneck) + explicit decision
(continue on feat/policy-quality or fork feat/policy-quality-v2).
Triage itself is a full brainstorming cycle.
3. HEALTH_DIAG §4.2 Track 1 fields only covered H1–H5 — H6–H10
needed their own detection signals. Added:
- trail_fire_rate_{quarter,half,full} (H6)
- hold_time_at_exit_{quarter,half,full} (H6)
- vsn_mask_{magnitude,direction} (H7)
- noisy_sigma_{mag_head,dir_head} (H7)
- target_drift_{mag_head,dir_head} (H8)
- action_dist_eval_{quarter,half,full} (H10)
Fixed inconsistencies between sections after the rev-2 feature-branch
refactor (sections had gotten out of sync):
1. §2 (success criteria) rewritten — was stale from rev 1. Now has
clean mandatory/soft split matching §7.2, corrects the trade-count
gate (per-fold not averaged), references the surrogate-noise gate.
New §2.3 enumerates the outcome paths.
2. §4 and §6 no longer say "commit on main" — both land on
feat/policy-quality per the feature-branch architecture. Phase 4
merges to main at §7.4.
3. §7.3 outcome handling rewritten — was written for single-branch
model ("Phase 2 commit is NOT reverted"). New wording matches
feature-branch semantics: failing mandatory = don't merge; failing
soft = merge + follow-up.
4. §6 smoke-tests rule now lists all six Phase 0 smokes, not just
Track 1/2.
5. §7.4 Phase-4 merge-strategy added — --no-ff merge commit, tag
policy-quality-v1, cleanup sequence documented.
6. §5.5 conflict-resolution added — tracks can reach contradictory
conclusions (e.g. T1 says fix, T2 says delete). Resolution rules:
mandatory-gate dominance, simplification wins, escalation.
7. §4.3 baseline metrics now committed to the feature branch (not
"investigation-only"), matching the crash-safety discipline.
8. Budget / risk register aligned — 5.5–6.5 hrs plan + 1 buffer,
hard-capped at 3 Phase-3 validation runs.
If H9 confirms and Phase 2 removes the magnitude branch, F_Half/F_Full
gates become vacuous. Substitute: direction bin distribution healthy
(F_Short + F_Long each >= 20% of non-Flat actions). Preserves the
'model actually trades' intent of Criterion A.
Addresses critique from second self-review:
1. SCOPE GAP fixed: spec now covers all 4 V7 categories the user asked
for. Added Track 3 (controllers) and Track 4 (exploration) with their
own smoke tests (controller_activity, exploration_coverage) and
HEALTH_DIAG fields. Previous version only had action-space + reward.
2. H1-H5 expanded to H1-H10 with 5 new codebase-specific hypotheses:
- H6: Regime-adaptive trailing stop closing Full-positions early
- H7: VSN / NoisyNets masking magnitude features
- H8: Target tau too slow for magnitude propagation
- H9: Data genuinely favors Quarter (no bug, delete magnitude branch)
- H10: Entropy regularization + argmax tie ordering bias
3. H3 REFRAMED from "Kelly bug" to design critique — Kelly as hard
multiplier vs soft gate. Fix sketch updated to offer replacement.
4. Feature branch + tag strategy replaces "single branch" containment.
feat/policy-quality isolates work from main until Phase 4 merge via
PR. git tag policy-quality-baseline gives named rollback anchor.
5. SURROGATE-NOISE CHECK added as MANDATORY validation gate — catches
look-ahead/leakage bugs that would otherwise produce false-positive
edges. Permanent smoke artifact.
6. Validation gate restructured: B + surrogate-noise MANDATORY;
A criteria soft; controller-activity soft. Partial success path
explicit per-gate.
7. Risk register expanded — added scenarios for B-mandatory failure
(feat branch stays unmerged), all-10-hypotheses-REJECTED (fall back
to H9 delete-branch), surrogate-noise failure (halt spec, open
diagnostic spec), multi-load-bearing-controllers (tech debt doc).
8. Budget corrected to 5.5-6.5 hrs L40S (was underestimated 3.5-4),
with per-track breakdown.
Spec now: 464 lines, 10 sections, covers 4 V7 tracks × ~10 hypotheses
+ 8 reward terms + 7 controllers + 5 exploration mechanisms = ~30
audit items total.
Critical fixes after honest review:
1. reward-audit measurement: split into 3 classes (additive / transform
/ sample-selector) — one-size-fits-all metric couldn't work for
PopArt normalization or CF flip mirroring
2. H1 detection: replace undefined "Sharpe/state" with concrete
forced-exploration protocol (epsilon=1.0 for 1 epoch at mid-train,
sample states × magnitudes, compare Sharpe-per-trade)
3. H4 scope: clarify this is the per-action-branch gradient (direction
vs magnitude head), NOT the per-loss-component (IQN/CQL/C51/Ens)
budget. Fix sketch updated to target the correct knob.
4. Validation gate: criterion B (multi-fold Sharpe + WinRate) is now
MANDATORY; criterion A gates are soft. Prevents "partial success"
hiding a policy-quality failure.
5. Trade-count gate: per-fold ≥100 on ≥5/6, not an average — 5 lean
folds pooled with 1 fat one should not pass.
Minor:
- Budget trimmed to 3.5–4 hrs L40S (was 6–10)
- L40S pool made explicit as target
- Phase 1 branches now pushed as wip/* for crash safety
- Hyperopt-params mismatch handling documented
Sub-project A of the production-readiness roadmap. Single-branch,
phase-gated: measurement substrate → parallel investigation tracks
(magnitude collapse + reward V7 audit) → single convergence commit →
6-fold × 50-epoch L40S validation.
Target criteria at close:
A (action dist): F_Half/F_Full each ≥15% on ≥4/6 folds, WinRate
55-65%, ≥100 trades/val window
B (multi-fold): Best Sharpe > 10 on ≥5/6 folds, WinRate > 55% on
≥5/6 folds
Greenfield checkpoint policy — any action-space or weight-shape change
is allowed. All confirmed bugs fix together in one Phase-2 commit.
V7 discipline gates inclusion (measurement confirms the bug), no
ranking across findings.
Out of scope for this spec: paper trading (sub-project C), canary
deployment (sub-project D), hyperopt re-runs, architectural changes
beyond action space.
Awaiting user review before invoking writing-plans.
Completes Phase 3 of the env-unification effort: experience_env_step (training),
backtest_env_step (val single-step), and backtest_env_step_batch (val batched)
all call the same __device__ helper `unified_env_step_core` in trade_physics.cuh.
Drift between train/val becomes structurally impossible at compile time — any
change to the canonical step applies atomically to both.
Three structural fixes were required to make the training port work:
1) TWO max_position params in unified_env_step_core
Training scales `max_position` by Var[Q] (Kelly-from-atoms variance sizing)
→ `effective_max_pos`. The original training kernel used this for
compute_target_position_4branch but the UNSCALED `max_position` for
apply_kelly_cap and execute_trade. Initial port collapsed both into one
helper arg, tightening Kelly cap aggressively → tiny positions → Q-values
converge → q_gap=0.00 collapse. Split into `max_position_target` (scaled)
and `max_position_physics` (unscaled). Val passes the same value for both.
2) Remove double hold_time update
Helper now owns `update_hold_time(...)` inside the step. Training's
inline `hold_time = update_hold_time(...)` at line ~1574 was a double-
increment. Fix: capture `saved_hold_time` BEFORE helper for the
patience-multiplier in segment reward, then trust the helper's in-place
update of hold_time. Segment-hold-time semantics preserved.
3) Remove triple Kelly-stat update
Helper calls `record_kelly_trade_outcome(...)`. Training ALSO had Kelly
stat updates inside the reversing_trade block (line 1558) and the
exiting_trade block (line 1638). TRIPLE update → win_count/loss_count/
sum_wins/sum_losses inflated 3× → Kelly cap tightened proportionally →
positions starved → Q-gap collapse. Fix: only the helper now touches
Kelly win/loss/sum_wins/sum_losses. Training still updates its own
sum_returns / sum_sq_returns (continuous-Kelly stats, distinct
buffers not touched by the helper).
Smoke test result (RTX 3050 Ti, 20 epochs, single-trial):
Before port: Best Sharpe ~15-25 (varies, range 10-31)
After broken port: Best Sharpe 1.17 (q_gap=0.00 collapse)
After fix: Best Sharpe 39.25 (HIGHEST ever recorded)
q_gap 0.00 → 0.27 (growing, healthy separation)
sharpe_ema trajectory: -5.3 → 12.7 → 26.9 (rising)
Multi-trial (partial visible): q_gap rising 0.00 → 0.86 by epoch 6,
sharpe_ema 15-25 consistently positive. Both tests passed.
Training kernel now has the CORE STEP in a single helper call — the
kernel body downstream continues to handle training-only concerns
(counterfactuals, plan_params, reward shaping via shaping_scale,
saboteur via exploration_scale, replay buffer writes).
Net −99 LOC across three kernels replaced by shared helper calls.
Files touched:
crates/ml/src/cuda_pipeline/trade_physics.cuh (+15/-5)
crates/ml/src/cuda_pipeline/backtest_env_kernel.cu (+53/-108 net)
crates/ml/src/cuda_pipeline/experience_kernels.cu (+48/-102 net)
docs/superpowers/specs/2026-04-21-unified-train-val-env-design.md
(Phase 2 documented as helper-extraction; Phase 3 kernel-deletion
remains deferred — both kernel entry points kept for their distinct
threading models; the physics is what's now truly unified.)
Verified: cargo check + smoke test pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A) TD-propagation diagnostic smoke test (sparse rewards)
- New smoke_tests/td_propagation.rs captures training_sharpe_ema across
20 epochs with micro_reward_scale=0 to answer: "can sparse-reward
Q-learning extract policy from trade-completion P&L alone?"
- Asserts: finite sharpe_ema, no monotonic degradation (tolerance 0.1
over first-3 vs last-3 epoch avg), q_gap > 0.05
- First run answered YES: val Sharpe peaks at +12.49 at epoch 8 with
pure sparse rewards, confirming the objective is learnable. The
remaining issue is stability/overfitting, not TD propagation.
B) Entropy → plasticity gate in training_loop
- D3/N3 shrink_perturb trigger now fires on `last_action_entropy < 0.3`
OR `health_value < 0.3` (OR semantics, 3 consecutive epochs).
- Catches action-collapse cases the generic health metric misses: Q-values
separate cleanly but argmax stays pinned to a single branch.
- tracing::info now logs both signals.
C) commitment_lambda coverage verified
- Almgren-Chriss sqrt market-impact already in compute_tx_cost
(trade_physics.cuh:168-171). commitment_lambda was pure duplication.
- Inventory doc updated: P2 marked satisfied, table row status updated.
Files touched:
- crates/ml/src/trainers/dqn/smoke_tests/mod.rs (+2)
- crates/ml/src/trainers/dqn/smoke_tests/td_propagation.rs (new)
- crates/ml/src/trainers/dqn/trainer/training_loop.rs (+15/-3)
- docs/superpowers/specs/2026-04-21-phase1-reward-inventory.md (+6/-3)
Smoke test PASSED locally (RTX 3050 Ti, 30.99s end-to-end).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 2 P1 relocations — the intents behind two deleted reward shaping
terms now live at the correct layer: diagnostic monitoring, not reward.
1. training_sharpe_ema (was w_dsr reward input):
Field already existed. Added to HEALTH_DIAG log line as `sharpe_ema`.
Available for early-stopping, model selection, and exploration gating
without perturbing the objective.
2. action_entropy (was position_entropy_weight reward):
Computed from summary.action_counts at GPU summary download —
Shannon entropy normalized to [0, 1] by ln(n_bins). One-epoch lag
is fine for a diagnostic. Added to HEALTH_DIAG as `action_entropy`.
Stored in new trainer field `last_action_entropy: Option<f32>`.
New HEALTH_DIAG suffix: `diag [sharpe_ema={:.3} action_entropy={:.2}]`.
Verified visible on E1 smoke test epoch 18/19 output.
Also: honest recalibration of Phase 2 results added to the inventory
doc. The earlier commit messages framed "-150 → -17 Sharpe" as a 5×
improvement; in absolute terms Sharpe -17 is still catastrophic (you'd
blow the account). Phase 2 stopped active capital destruction but did
not find alpha. Sharpe_raw per bar went from -0.39 (lot of loss per bar)
to -0.09 (little loss per bar) — still net losing. q_gap=0.17 is a
capacity metric (network CAN differentiate), not profitability.
Path to actual profitability (Sharpe > 0) remains:
- TD propagation verification (Task #8)
- Unified env kernel (Phase 3)
- Possibly a different objective entirely
- Richer features (42 market + 20 OFI may be information-starved)
The rule going forward: before celebrating future improvements, ask
whether the result *crosses zero* (profitable) or is just *less
negative* (still losing).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mass deletion of the "v7 gem" reward terms identified in the Phase 1
inventory as category errors — each one rewarded an outcome-adjacent
behavior instead of encoding the underlying physics, and each
empirically hurt validation metrics more than it helped training
stability.
Deleted from experience_kernels.cu and all plumbing (Rust configs,
launch args, hyperopt logs, TOML entries):
* order_credit_weight - reward redundant with compute_tx_cost
(order_type_idx already differentiates
fills by order type)
* risk_efficiency_weight - reward double-counted drawdown penalty
asymmetrically (only on winners)
* urgency_credit_weight - reward was vol-normalized unrealized P&L,
pure rename of core return
* commitment_lambda - triple-counted churn + tx_cost
* w_dsr - kernel wrote DSR EMA but no longer added
to reward (dead); removed the EMA
bookkeeping too
* dsr_eta - kernel arg for the deleted DSR EMA
* position_entropy_weight - rewarded action-bucket diversity
regardless of outcome; histogram buffer
+ zero-init removed too
* exit_timing_weight - already inactive (used raw_next future
price, comment-deleted earlier)
* ofi_reward_weight - dead plumbing; OFI already passed as
feature through state[OFI_START..]
* opportunity_cost_scale - penalized flat when Q-gap wide;
redundant with Q-values themselves
Kernel arg count: experience_env_step_batch shrank from ~55 to ~45 args.
Rust-side config surface reduced correspondingly.
Results on E1 smoke test (20-epoch):
BEFORE any Phase 2 work:
Val Sharpe -120 to -150, MaxDD 10-15%, Sharpe_raw -0.39
AFTER reward_noise + Kelly (both envs) + urgency + this sweep:
Val Sharpe -17 to -22 (7× better)
Val MaxDD 0.27% (40× better)
Val Sharpe_raw ~-0.09 (4× better)
Training Sharpe_raw ~0 (stabilized from ±20 swings)
Final q_gap 0.1712 (highest yet, collapse mechanism fine)
The extreme train-Sharpe swings (+17 one epoch, -13 next) were not
learning dynamics — they were shaping-term noise. Core reward (P&L +
drawdown + churn + holding + tx_cost + Kelly physics cap) gives training
metrics that actually reflect what the model does.
Inventory doc (docs/superpowers/specs/2026-04-21-phase1-reward-inventory.md)
extended with a "better-form taxonomy" section: every deleted gem has
a correct layer it belongs to (physics, feature, diagnostic, gradient-
level regularization — not reward). Kelly cap and Q-target smoothing
are already relocated; others are scheduled per the taxonomy's P1/P2/P3
priority list.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extends the reward inventory with a review pass that preserves the
*ideas* behind behavioral terms while deleting the wrong-level mechanics.
Gems (relocate, don't delete):
- Probabilistic fill model (real physics, not shaping) → apply in both
modes via shared Philox RNG + exploration_scale
- Kelly-as-physics-constraint → hard position-size cap in trade_physics,
not a soft reward
- Path quality via per-step drawdown integration → no separate term,
just run drawdown penalty every bar
- Saboteur relocation → both modes, scaled by exploration_scale (training
1.0, validation 0.5, diagnostic 0.0) — avoids creating a new clean-vs-
noisy mismatch in the opposite direction
Pearls:
- Sparse reward is fine if TD propagation works — diagnose before
deleting micro_reward_scale; measure cov(Q(s_entry,a), trade_return)
- Every behavioral term maps to one of four failure modes: double-count,
misplaced physics, wrong-level regularization, or compensating for a
downstream bug. None are semantically "neutral" shaping.
Novel architectural moves:
- Diagnostic-only entropy tracking (no reward, just HEALTH_DIAG
logging)
- Single exploration_scale scalar replacing all mode toggles
(feedback_no_feature_flags compliant)
- Layered reward with per-term budget caps — makes the train/val Sharpe
gap computable instead of uncomputable magic
- Q-target smoothing replacing reward_noise_scale (regularization at
gradient level, not reward level)
Updated disposition table: 4 DELETE, 2 MOVE, 2 RELOCATE-to-physics,
1 DIAGNOSE-then-DELETE, 4 delete-dead-plumbing.
Three relocations (Kelly cap, saboteur scaling, Q-target smoothing) can
land as independent PRs before the unified_env_kernel rewrite, shrinking
Phase 2's change surface.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Enumerates every term that contributes to training or validation reward,
classifies each as P&L-aligned (keeper) or behavioral (remover), with
source line references into experience_kernels.cu and backtest_env_kernel.cu.
Findings:
- 7 P&L-aligned terms to preserve in the unified env (core return,
tx_cost, drawdown, inventory, churn, capital floor, + vol normalization)
- 8 active behavioral training-only terms to delete:
* order_credit_weight (rewards theoretical limit-order savings,
double-counts actual tx_cost)
* risk_efficiency_weight (double-counts drawdown asymmetrically)
* urgency_credit_weight (redundant with core return)
* kelly_sizing_weight (rewards matching a formula, not outcomes)
* micro_reward_scale (OFI-momentum signal follower)
* commitment_lambda (triple-counts churn)
* reward_noise_scale (belongs at gradient level, not reward level)
* position_entropy_weight (changes what "optimal" means)
- 4 dormant/dead terms to clean up (w_dsr, exit_timing, ofi_reward,
opp_cost_scale — all plumbing with no kernel effect)
Biggest offenders by magnitude: urgency_credit and risk_efficiency can
dominate the core signal on a single positioned bar.
Phase 2 (unified_env_kernel implementation) can now proceed with a
concrete deletion list, not a judgment call.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Brings the 4-commit work from wip/adaptive-learning-rootcause into main:
24f96ab78 fix(n1): distillation now actually pulls weights during collapse
f21a7d661 test(e1): 20-epoch smoke test + raw-q_gap assertion + disable early-stop
a3fd02a95 fix(early-stop): log real training epoch, not internal call counter
9dbd8d7e9 docs(design): unify training and validation environments
Key outcomes:
- Distillation mechanism now actually pulls weights during collapse (fixed
three bugs: epoch-boundary grad_buf erasure, CUDA-graph scalar baking,
wrong q_gap signal at snapshot gate). E1 smoke test passes deterministically.
- Design doc lays out the next step: unifying training and validation env
kernels so validation Sharpe tracks training Sharpe (currently diverges
catastrophically by ~-150 absolute).
- Incidental: early-stopping log now reports the real training epoch
instead of its internal call counter.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Design document for the follow-up to the adaptive-learning-rootcause
session. Lays out the evidence, root cause, options considered, and
recommended path for closing the 70% structural gap between training
and validation Sharpe that remained after the distillation collapse
fix landed.
Key findings documented:
- Reward-shaping ablation (2026-04-20) closed ~30% of the gap; ~70%
remains architectural
- Two env kernels (experience_env_step vs backtest_env_step) have
drifted: spread scaling, fill model, saboteur noise, reward terms,
action selection, position dynamics all differ
- Hint from history: experience_kernels.cu:1418 comment "Regime-
adaptive scaling removed to eliminate train/eval mismatch" shows
someone aligned *some* things previously
Recommended path (Option C, "unified env with layered reward"):
- Single unified_env_step kernel replaces both
- Core reward = pure P&L; shaping is additive and P&L-units-aligned
- Validation = training with exploration_scale=0 AND shaping_scale=0
- Scale factors are pinned device-mapped scalars (same pattern used by
the distillation alpha fix)
Phased implementation plan with ~8-day budget and concrete success
criteria: validation Sharpe_raw within 0.05 of training Sharpe_raw by
epoch 30 on L40S production run.
Rejected alternatives: backtest-matches-training (hides real issue),
training-matches-backtest (regresses stability), two-environment with
divergence as metric (fallback only).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per user directive, ran rg scans for PINMEM, ROMEM, LOCKHOT, BORROW,
CPURO and populated the scoreboard with concrete file:line findings.
PINMEM: 10 new findings (PINMEM-011..019, plus rescan confirmed 001-010)
- Biggest surface: gpu_experience_collector.rs (7 htod sites)
- High priority: gpu_iqn_head.rs dtod_copy for iqn_rewards/iqn_dones
(PINMEM-013, score 25.0, switch to async-dtod, E=1)
ROMEM: 5 new findings (004 expanded, 005-008 added)
- ROMEM-004 now covers ~40 cuBLAS/cuBLASLt/cuDNN workspace casts across
shared_cublas_handle.rs, gpu_iql_trainer.rs, gpu_iqn_head.rs,
gpu_curiosity_trainer.rs, cublaslt_debug.rs — bulk false-positive
candidates (FFI convention, not actual RO writes)
- ROMEM-007 adds 6 more device-mapped pinned write sites (same pattern
as ROMEM-001/002 — benign cuMemHostAllocMapped)
LOCKHOT: 5 new findings (006-010)
- LOCKHOT-006/008: tokio::sync::Mutex<PPO> and RwLock<TLOBTransformer>
held across .await — deadlock risk. High priority.
- LOCKHOT-007: Arc<Mutex<VecDeque<f64>>> history locks per step
BORROW: 1 new finding (002)
- BORROW-002: RefCell<PPO> × 2 in validation/ppo_adapter.rs —
documented single-threaded, needs invariant verified
CPURO: deferred — .len()/.shape() scan returns hundreds of mostly-Vec
matches; left CPURO-000 task for next iter to classify per-site.
Scoreboard now has ~40 open findings across 12 active categories.
Expanding the scanner surface per user directive "all should be
addressed". Four distinct root causes for CPU-side violations around
GPU-owned data — each with its own scan command, fix hierarchy, and
learned-patterns entry.
- CPURO: CPU reads of GPU-resident data (sizes, reductions) that
force implicit sync. Cache at construction, keep stats on device.
- ROMEM: `*(ptr as *mut T)` writes where ptr came from a const /
read-only source. CUDA mapped-memory flags matter; cuBLAS/cuDNN
workspace casts are benign FFI.
- LOCKHOT: Mutex/RwLock on per-step path. High-value hits already
visible: Mutex<GpuDropout>, Mutex<Option<DropoutScheduler>> in
network.rs (every forward pass), Arc<Mutex<NStepBuffer>> in dqn.rs.
Never hold tokio::sync::RwLock across .await.
- BORROW: shared-&T promoted to &mut T via unsafe ptr casts or
UnsafeCell/RefCell. Real example shipped: gpu_replay_buffer.rs:690
changes CudaSlice<i32> → CudaSlice<u32> through raw-ptr cast.
Scoreboard seeded with 10 findings. Top scores:
- ROMEM-001 (25.0) - size_pinned mapped write, verify allocation flag
- ROMEM-004 (25.0) - cuBLAS workspace casts, likely false-positive bulk
- ROMEM-002 (15.0) - init-time pinned mapped writes
- LOCKHOT-001/002/003 (8.3) - dropout + nstep_buffer locks on hot path
- BORROW-001 (8.3) - CudaSlice element-type aliasing
CPURO is seeded with a scan-task (CPURO-000) to populate per-site
findings in iter 9 — too many Vec::len() false positives to list upfront.
Per user directive: htod/dtod transfers on the training hot path that
bounce through pageable host memory are a top-priority cleanup target.
Pinned memory (cuMemHostAlloc) + memcpy_htod_async enables DMA + stream
overlap. The codebase already has the pattern (size_pinned,
rng_step_pinned in gpu_replay_buffer.rs) — extend it to every per-step
transfer.
Prompt changes:
- New PINMEM category in §3 table (severity 5)
- Scan command excludes tests/benches/examples/smoke_tests
- New §6 learned pattern entry with 3-tier fix hierarchy:
(1) eliminate via on-device compute → (2) stage through pinned →
(3) async for dtod
- Added PINMEM to category ID allowlist in §2
Scoreboard seed: 10 initial findings from an iter-9 scan. Top-scoring:
PINMEM-003 (scratch_f32 scalar, E=1, score=25) and PINMEM-004 (init
scalars, score=15). Hot-path candidates include NoisyNet epsilon
refresh, PER replay-buffer inserts, target-net sync, PPO rollout.
Per user directive from iter 7 ("no deferred tasks, solve properly"),
the Needs-human review escape hatch is closed. Updated:
- Step 6c: replaced "move to Needs human review" with split-into-
sub-findings guidance; every finding must be solved.
- Step 6e: cargo-check failure now requires diagnosis + retry, not
deferral. Pre-existing external errors go to Known external state.
- Step 8 summary line: dropped the "deferred j" counter.
- Rule 7: similar rephrasing.
- New §6: learned patterns from iters 1-8 (dead flag chains,
aspirational config, dishonest fallbacks, inference-side flag flips,
accounting-only values, orphan files, hidden GPU sync).
Ralph re-feeds this file verbatim, so the next iteration picks up the
updated rules automatically.
- Layer 2 labeled as G2-G5, G5 explicit for gradient budget
- P2 spectral_gap_norm added as 7th component in composition (rebalanced weights)
- N6 ensemble oracle has explicit threshold (>0.8 triggers N3 immediately)
- N8 meta-Q wiring clarified: logged but not in composition until validated
- Files Changed deduplicated, paths qualified
- Success criteria: WinRate >55% (above random baseline ~25%)
- HEALTH_DIAG log line includes all 7 components
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fixes Q-value collapse via unified LearningHealth signal that senses
training health (6 components) and continuously adapts:
- CQL regularization (regime + health gated)
- Gradient budget (IQN/CQL/Ens/C51 dynamic allocation)
- Tau target EMA (health-coupled)
- Expected SARSA temperature (continuous, no hardcoded threshold)
Plus 4 pearls (PER priorities, spectral detection, gradient consistency,
adaptive gamma) and 8 novels (self-distillation, barrier loss, plasticity
injection, CF curriculum, information bottleneck, ensemble oracle,
contrarian override, meta-Q network).
Core principle: training hyperparameters are OUTPUTS of the temporal
pipeline, not static schedules. The system meta-learns its own settings.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Training and validation use different kernels with different state layouts,
making generalization impossible. Single state_layout.cuh + one assembly
function + STATE_DIM as compile-time constant.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Removed the legacy non-branching Sequential q_network and target_network
from DQNAgent. These were never used (branching+dueling always active)
but allocated VRAM and ran noise resets every step. -190 lines.
Config tuning for dense micro-reward system:
- n_steps: 5→1 (TD(0), micro-rewards cancel over n>1)
- tau: 0.007→0.01 (faster target tracking for TD(0))
- c51_alpha_max: 0.5→1.0 (full C51, PopArt handles normalization)
- curiosity_weight: 0.1→0.0 (dense micro-reward replaces curiosity)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
target_dim expansion: adds raw_open (OHLCV) and mid_price_open
(MBP-10 midpoint at bar formation) to fxcache targets. FXCACHE_VERSION
2→3 for auto-rebuild. Legacy v2 files handled with close-price fallback.
Spec v5 adds 3 pearls:
- Bar duration encoding in Mamba2 (continuous-time SSM awareness)
- Order book center of mass from all 10 MBP-10 levels (aggression signal)
- Retrospective hold quality bonus (teaches exit timing)
Plus: Hold action (4th direction), DSR Sharpe EMA fix, counterfactual
magnitude/order sign fix, MFT mid-price mark-to-market.
OFI embed MLP now 18→10 (was 16→8). Mamba2 width SH2+10 (was SH2+8).
Attention width D+10 (was D+8).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
target_dim 4→6: adds raw_open (OHLCV) and mid_price_open (MBP-10
best bid/ask midpoint at bar formation) to the targets buffer.
Micro-reward now uses real intra-bar move (close - open) / atr
instead of close-to-close approximation. Also adds spread cost
awareness via |open - mid| penalty. Requires fxcache recomputation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Major changes from v1:
- Fix OFI data pipeline (ofi_gpu never wired, indices 42→66)
- Expand PORTFOLIO_STRIDE 30→38 for prev-OFI storage
- Add Mamba2 d_h_history backward (2 new cuBLAS GEMMs)
- Expose attention d_input_scratch for gradient flow
- Pre-compute OFI deltas in experience collection (state[74..82])
- Lower rank normalization threshold to 1e-5 for micro-rewards
- Use close-to-close return instead of unavailable open_price
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Dense per-bar reward using order flow momentum × price confirmation.
OFI embedding MLP (16→8 via cuBLAS) feeds into Mamba2 history AND
attention input. Replaces sparse exit-only reward with continuous
temporal signal for the SSM and attention heads to learn from.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Single cuGraphExecLaunch per step. PER sampling, gather, training,
priority update ALL as child graph nodes in one parent. Direct-to-trainer
gather eliminates DtoD copies. GPU-side counters eliminate host writes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
