Adds 12 new `SL_*_GROUP_BEGIN/END` macros to `state_layout.cuh` covering the
6 feature groups (market/ofi/tlob/mtf/portfolio/plan_isv), plus
`SL_NUM_FEATURE_GROUPS=6` and `SL_MAX_FEATURE_GROUP_DIM=42` (= largest group
dim, MARKET_DIM). Six new `static_assert`s anchor each group dim ≤ max and
confirm contiguity / start-at-zero / end-at-padding invariants.
Rust mirror in `crates/ml-core/src/state_layout.rs` exposes:
- `SL_NUM_FEATURE_GROUPS`
- `SL_MAX_FEATURE_GROUP_DIM`
- `FEATURE_GROUP_RANGES: [(usize, usize); 6]` (half-open `[begin, end)`,
`plan_isv` ends at `PADDING_START` — padding is not a group)
- `FEATURE_GROUP_NAMES: [&str; 6]`
Three `const _: () = assert!(...)` blocks validate (1) first range starts at
0, (2) last range ends at `PADDING_START`, (3) every adjacent pair
satisfies `ranges[g].end == ranges[g+1].begin` (no gaps), (4) every group
dim ≤ `SL_MAX_FEATURE_GROUP_DIM`.
Group dims at this commit: market=42, ofi=32, tlob=16, mtf=16, portfolio=8,
plan_isv=7 — total 121 = `PADDING_START`.
Prerequisite for Plan 4 Task 1B (E.1 Variable Selection Network — pre-trunk
per-group softmax-over-groups gating), Task 5 Mode B (per-group ISV
diagnostics), and Task 4 follow-on (group-aware encoder interface).
Additive only — ZERO production callers in this commit. No new module /
kernel / ISV slot / param tensor / Orphan row. No fingerprint change (group
ranges are derivable from existing `SL_*_START` constants and contribute no
new structural-hash invariant). cargo check clean at 11 warnings (workspace
baseline preserved); cargo build compiles all kernel cubins (state_layout.cuh
edit triggers full kernel rebuild — no compile errors).
Audit doc: 1 new entry under "Plan 4 Task 1A".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Plan 3 Task 6b.
Portfolio-state tail-append:
- PS_REGIME_SHIFT_BAR = 40 (hold_time of first detected regime shift, 0 if none)
- PS_STRIDE 40 → 41
- All 6 hardcoded-stride sites migrated in lockstep
Detector (experience_kernels.cu):
- Adaptive threshold = clamp(0.25 × |clamp(sharpe, -2, 2)|, 0.05, 0.5)
- Fires first bar where |regime_now - PS_PLAN_ENTRY_REGIME| > threshold
- First-shift-only (short-circuits on non-zero PS_REGIME_SHIFT_BAR)
- Uses new ISV_SHARPE_EMA_IDX = 22 macro in state_layout.cuh
Consumer (segment_complete block):
- bars_late_frac = clamp(bars_late / hold_time, 0, 1)
- penalty = shaping × conviction × bars_late_frac × |reward|
- All multiplicands except |reward| in [0,1]; max penalty = |reward|
- reward -= penalty; rc[5] -= penalty (cancels with B.2/C.4/D.4a at
other (i,t) slots; ISV[68] REWARD_BONUS_EMA shows net)
- Consumer resets PS_REGIME_SHIFT_BAR after use
**Iteration history.** First pass multiplied by ISV[Q_DIR_ABS_REF] (~5–50,
an absolute Q-magnitude) AND |reward| — produced penalties 5–50× the
reward, destabilising training (smoke: Return swings ±300–900%, Sharpe
oscillating wildly). Root cause: Q_DIR_ABS_REF is an absolute
magnitude, not a [0,1] coefficient; B.1 uses it as a DENOMINATOR to
normalize q_range, not as a multiplier on an already-unbounded signal.
Fix: drop q_scale, keep |reward| as the only unbounded factor. Smoke
now passes cleanly with fold-2 best Sharpe 117.92 (up from T6a's 100.10).
No new ISV slot.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Plan 3 Task 6a.
Portfolio-state tail-append (shared-contract migration, all in same commit):
- PS_INTRA_TRADE_MIN_PNL = 39 (symmetric to PS_INTRA_TRADE_MAX_PNL = 21)
- PS_STRIDE 39 -> 40
- 6 PORTFOLIO_STRIDE hardcoded copies bumped in lockstep
Producer (experience_kernels.cu):
- MIN_PNL tracked per bar (fminf against pnl_pct) in the same block
as MAX_PNL update
- Reset to 0 at all 5 MAX_PNL reset sites (entry, reverse, 2x fold boundary)
Consumer (experience_kernels.cu segment_complete):
- Fires only on reward > 0 AND drawdown_depth > 1e-6
- persist_bonus = shaping x conviction x |min_pnl| x tanh(reward/|min_pnl|)
- reward += persist_bonus; rc[5] += persist_bonus (accumulates with
B.2 entry bonus + C.4 timing bonus — different (i,t) slots per trade)
Self-scaling via tanh: no tuned coefficients. Saturates when recovery
is large relative to drawdown; near-zero when recovery is trivial.
Attribution lands in ISV[68] REWARD_BONUS_EMA via the Task 1 kernel.
No new ISV slot.
Smoke: multi_fold_convergence PASS (fold-2 best Sharpe 100.10, threshold >=80).
HEALTH_DIAG reward_split bonus=17.21 (post-Task-5 rises with new D.4a credit
firing on profitable drawdown recoveries).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Plan 3 Task 5.
Portfolio-state tail-append (shared-contract migration, all in same commit):
- PS_PEAK_PNL_BAR = 38 (hold_time snapshotted when MAX_PNL updates)
- PS_STRIDE 38 -> 39 in state_layout.cuh and ml-core/state_layout.rs
- PORTFOLIO_STRIDE 38 -> 39 in trade_stats_kernel.cu (hardcoded copy)
- PORTFOLIO_STRIDE 38 -> 39 in gpu_experience_collector.rs allocator
- ps_stride 38 -> 39 in gpu_dqn_trainer.rs launch_kelly_cap_update
Producer (experience_kernels.cu):
- Peak bar snapshotted alongside every MAX_PNL update (uses local
hold_time, not ps[PS_HOLD_TIME], because the portfolio-state commit
block runs later in the kernel).
- Peak bar reset to 0 at every MAX_PNL reset site: plan-entry (1856),
entering_trade (2014), reversing_trade (2019), fold hard-reset (2736),
trade-complete soft-reset (2751).
Consumer (experience_kernels.cu segment_complete block):
- bars_early = max(0, segment_hold_time - PS_PEAK_PNL_BAR)
- timing_bonus = shaping_scale x (bars_early / segment_hold_time)
x |final_pnl| x conviction_core
- reward += timing_bonus; rc[5] += timing_bonus
(accumulates with Task 3 B.2 entry bonus — different (i,t) slots).
No new ISV slot — rc[5] bonus semantics unchanged; B.2 and C.4 share it
via += accumulate semantics (defensively idempotent, but the two sites
fire at distinct (i,t) by construction: entry vs exit).
Self-scaling: shaping_scale x conviction_core x |pnl| keeps the bonus
proportional to trade magnitude, no tuned coefficients.
Smoke multi_fold_convergence (RTX 3050 Ti): all 3 folds complete,
fold-2 best Sharpe 84.44 at epoch 1 (expected ~85 range).
cargo check --workspace clean at 11 warnings baseline.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds 7th plan_isv dimension: PLAN_ISV_REMAINING_FRACTION = max(0, min(1,
(plan_target_bars - hold_time) / plan_target_bars)) when plan active, else
0. Exposes temporal pressure to the policy.
State vector grows 104 → 112 (105 + 7 padding for 8-alignment).
SL_PORTFOLIO_PLAN_DIM 6 → 7. SL_PADDING_DIM 0 → 7.
Both training (experience_env_step) and val (backtest_plan_state_isv) write
the new slot identically — preserves train/val state-distribution parity.
All hardcoded stride-6 references in backtest_plan_kernel.cu replaced with
SL_PORTFOLIO_PLAN_DIM. plan_isv_buf allocation updated to n_windows * 7.
Stale offset comments ([86..92)) corrected to [98..105) across all files.
No behavioural change to existing dimensions. New signal is additive.
Plan 2 Task 6A. Spec §4.D.6.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fix kernel-read gap
Adds 12 features to the DQN input pipeline:
- 10 MicrostructureState::snapshot()[0..10] slots that were previously computed
every bar and then discarded before reaching fxcache: ofi_trajectory,
realized_variance, hawkes_intensity, book_pressure (weighted 10-level),
spread_dynamics, aggression_ratio, queue_depletion_asymmetry,
order_count_flux, intra_bar_momentum, regime_score.
- 2 TLOB-novel slots derived directly from Mbp10Snapshot:
order_count_imbalance = (Σbid_ct − Σask_ct) / Σ(bid_ct + ask_ct),
microprice_residual = (weighted_mid − mid) / mid.
Also fixes a production gap: ofi_acceleration (slot 18) and
toxicity_gradient (slot 19) were persisted to fxcache via OFI_DIM=20
but the OFI embed kernel (experience_kernels.cu:6146-6173) only read
[0..18), silently discarding them every bar. Kernel extended to
consume full SL_OFI_DIM=32.
Dimension bumps (all 8-aligned):
OFI_DIM 20 → 32
FXCACHE_VERSION 4 → 5 (invalidates existing caches; regen via
precompute_features)
STATE_DIM 96 → 104
PADDING_DIM 4 → 0 (OFI expansion consumed padding, still 8-aligned)
STATE_DIM_PADDED 128 (unchanged)
OFI_EMBED_IN 18 → 32 (MLP input width; W/grad/Adam/m/v buffers
resized in lockstep via named constants)
fxcache regen results (175874 bars ES.FUT 2024-Q1):
deltas_nonzero: 175781 / 175874 (99.9 percent)
book_aggression: 102137 / 175874 (58.1 percent)
microstructure[20-30): 175874 / 175874 (100 percent)
tlob_novel[30-32): 133615 / 175874 (76.0 percent)
Compile status: SQLX_OFFLINE=true CARGO_INCREMENTAL=0 cargo check
--workspace --tests passes cleanly (0 errors, pre-existing warnings
only).
Test results:
fxcache roundtrip (unit + integration): PASS (4+6 tests)
magnitude_distribution smoke: ran through epoch 1 successfully
(OFI_DIAG fires, state_dim=104 confirmed, feature_dim=74 in
validation kernel); epoch 2 OOM on local RTX 3050 Ti (4 GB) —
expected hardware limit from state_dim growth. Full 20-epoch run
requires L40S/H100 CI verification.
multi_fold_convergence smoke: not verified locally (same VRAM
ceiling applies). L40S/H100 CI verification required.
The new slots follow the existing OFICalculator/MicrostructureState
pattern and consume signals already computed by ml-features — no new
crate, no ONNX, no stubs. All 12 sources were audited against their
implementation before persistence; every slot traces back to real
Mbp10Snapshot or MicrostructureState math.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- ml-explainability/integrated_gradients.rs: the GPU IG path is a
stub that returns an error so callers fall through to CPU. Drop
the three TODO markers and describe the situation declaratively —
the reference CUDA source is kept as a follow-up anchor.
- ml-supervised/mamba/mod.rs: dropout in both inference and training
paths is simulated via the `1 - dropout_rate` scalar bake-in; no
dedicated GPU dropout kernel is wired on the supervised path. The
hidden-state carry-over in the SSD scan requires a 3-D narrow the
GpuTensor API does not expose, and inference only consumes the
last timestep, so the update is intentionally elided.
- ml-core/cuda_autograd/gpu_tensor.rs: the `cat` docstring claimed a
host round-trip; the implementation is already fully DtoD via
`memcpy_dtod` for dim=0 and per-row DtoD for dim>0. Rewrite the
comment to describe the real implementation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Action space 81→108: direction(4) × magnitude(3) × order(3) × urgency(3).
Short(0), Hold(1), Long(2), Flat(3). Hold keeps current position with
zero transaction cost, giving the model a "do nothing" option.
Changes: trade_physics.cuh (Hold returns current_position), env_step
(Hold skips trade execution), action.rs (ExposureLevel::Hold variant),
config (branch_0_size 3→4), all match arms updated across 11 files.
Counterfactual mirror: Short↔Long, Hold↔Hold, Flat↔Flat.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Added OwnedGpuLinear to ml-core (self-contained linear layer with owned
weight/bias CudaSlice, forward_with_slices bypasses GpuVarStore lookup)
- Removed GpuVarStore from all 12 ml-dqn modules: Sequential, branching,
distributional_dueling, dueling, network, rmsnorm, residual, curiosity,
attention, iql, quantile_regression, regime_conditional
- Removed get_q_network_vars/vars/store accessors from DQN, branching,
distributional_dueling, dueling, network, quantile_regression
- Replaced GpuLinear+GpuVarStore with OwnedGpuLinear in all cold-path modules
- Made load_from_safetensors a no-op (flat buffer path handles checkpoints)
- Made sync_to_varmap a no-op (flat buffer is source of truth)
- Moved branching->VarStore conversion to gpu_weights::branching_to_varstore
- Re-enabled bottleneck_dim=16 in config defaults + all 3 TOML configs
- Removed flatten_online_weights bottleneck skip workaround
- Zero GpuVarStore references in crates/ml-dqn/src + crates/ml/src/trainers/dqn
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The 4-branch DQN (direction x magnitude) had 3 degenerate variants
(Short25, Flat, Long25) that all mapped to 0.0 target exposure when
direction=Flat, causing 82% Flat collapse. Collapse these into a
single Flat variant, giving 7 levels (ShortSmall/Half/Full, Flat,
LongSmall/Half/Full) and 63 total factored actions (7x3x3).
- ExposureLevel enum: 9 variants -> 7 (add direction/magnitude/from_dir_mag)
- FactoredAction: 81 -> 63 total actions, from_index/to_index updated
- DQN epsilon-greedy: use from_dir_mag() instead of dir*3+mag indexing
- DQN config: num_actions default 9 -> 7
- PPO action space: 45 -> 63 actions, action masking updated
- Signal adapter CUDA kernel: 5-bin -> 7-bin exposure aggregation
- All tests updated for new variant names and index ranges
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Complete bf16 elimination across all crates (ml, ml-core, ml-dqn, ml-ppo,
ml-supervised). Zero half::bf16, __nv_bfloat16, or CudaSlice<half::bf16>
references remain (verified by grep).
CUDA: All 60+ .cu kernels and .cuh headers converted to native float.
- Half-precision intrinsics (__hmul, __hadd, __hdiv) → float operators
- atomicAddBF16 → native atomicAdd(float)
- bf16 wrapper functions → f32 identity passthroughs
Rust: All CudaSlice<half::bf16> → CudaSlice<f32> across 90+ files.
- htod_f32_to_bf16/dtoh_bf16_to_f32 → htod_f32/dtoh_f32 (direct, no conversion)
- Deleted bf16 mirror infrastructure (DuelingWeightSetBf16, alloc_bf16_mirror, etc.)
- Renamed params_bf16→params_flat, d_value_logits_bf16→d_value_logits, etc.
- Fixed .to_f32() sed damage on Decimal::to_f32() and rng.f32()
FxCache: Single f32 disk format (was bf16/f64 dual-version).
- Deleted --bf16 CLI flag from precompute_features
- PVC cache files need regeneration via precompute_features
TF32 tensor cores activated via cublasLtMatmul CUBLAS_COMPUTE_32F — no
explicit TF32 types needed. Storage is pure f32 everywhere.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three root causes found and fixed:
1. SUM-reduced gradients without 1/N: all CUDA loss gradient kernels
(C51, MSE, IQN backward, CQL) accumulated per-sample gradients as
raw SUM. At batch=16384 (H100) the raw norm was 282x larger than
batch=58, causing gradient clipping to destroy signal-to-noise ratio
and collapse training at epoch 2-3. Now all kernels multiply by
1/batch_size, making gradient scale batch-invariant.
2. ExposureLevel::target_exposure() used a flat 9-level scale that did
not match the 4-branch dir*mag encoding. The Rust backtest evaluator
computed wrong position sizes (e.g. 4x oversize for Short+Small).
Now uses dir x mag formula. Also fixed is_buy/is_sell/is_hold and
from_trading_action for 4-branch semantics.
3. Rust epsilon-greedy only explored 5/9 exposure combos (0..5 instead
of dir*3+mag), ignored the magnitude branch on greedy, and used wrong
indices for order/urgency (get(1)/get(2) instead of get(2)/get(3)).
LR recalibrated: old gradient_clip_norm was accidentally a batch-size-
dependent LR reducer (~60x at batch=58, ~16000x at batch=16384). With
mean-reduced gradients the clip rarely fires, so LR is now the sole
training speed control. Smoketest 1e-4 -> 2e-6, production 1e-4 -> 1e-5,
hyperopt range [1e-5,3e-4] -> [1e-7,1e-4].
Diagnostics: FOXHUNT_GRAD_DIAG=1 enables per-stage gradient norm logging.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Delete 3 unused bf16 scratch buffers (exp_h_b0/b1/b2) from
GpuExperienceCollector — allocated VRAM but never read.
The f32 versions (exp_h_b0_f32 etc.) remain in use.
- Fix stale doc comments: 45 actions -> 81 actions, index range 0-44 -> 0-80
- Extend round-trip tests to cover all 81 action indices
- Add TODO for future 4-branch struct refactor (direction + magnitude fields)
— 64 callers make it too invasive for this commit.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
gpu_n_episodes was manually overridden in GPU profiles, training configs,
test files, and hyperopt — all set to 0 or small fixed values that
bypassed the auto-scaling logic, causing a div-by-zero crash in
train_baseline_rl.
Now: single auto-scaling path via optimal_n_episodes() from VRAM/SM
count. No manual override field. Cap at 16384 (consistent with
AutoBatchSizer's 8192 cap pattern). Floor at 32 for small GPUs.
Removed gpu_n_episodes from:
- DQNHyperparameters, PpoHyperparameters structs
- All 4 GPU profiles (rtx3050, h100, a100, default)
- Training profiles (smoketest, localdev)
- ExperienceProfile struct + serde
- Hyperopt adapter
- All test overrides
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Deleted the hardcoded 20% fallback function. Constructor now uses
vram_fraction directly for initial PER budget. No hardcoded limits
anywhere in the sizing pipeline.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Removed MAX_REPLAY_CAPACITY = 10M and STATIC_MAX_BATCH_SIZE = 8192.
Removed MIN_REPLAY_CAPACITY = 100K (replaced with 1024 segment tree minimum).
AutoBatchSizer computes from actual free VRAM. Replay buffer uses
vram_fraction (0.70 for H100) instead of hardcoded 20%.
H100 80GB: batch ~2M ceiling, replay ~89M entries (was capped at 10M).
RTX 3050 4GB: still auto-scales to small values safely.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three root causes of sporadic NaN during training:
1. --use_fast_math (nvcc) breaks IEEE 754 NaN semantics: fmaxf(NaN,x)
returns NaN instead of x, isnan()/isinf() compile to false.
Replaced with --ftz=true --fmad=true --prec-div=true --prec-sqrt=true
across all 4 build.rs (ml, ml-dqn, ml-ppo, ml-core).
2. Cross-stream race: replay buffer wrote batch data on the device's
original stream while the trainer read it on a forked stream.
Fixed by passing the forked stream to the DQN agent via agent_device,
so all GPU components share a single CUDA stream (zero sync overhead).
3. Rewards/dones stored as bf16 in replay buffer caused done=0xFFFF NaN.
Converted entire rewards/dones pipeline to f32: experience collector,
replay buffer storage, nstep kernel, loss/grad kernels.
Also:
- Removed fast_isnan/fast_isinf/fast_isfinite wrappers — standard
isnan/isinf/isfinite work correctly without --use_fast_math
- Updated dqn-smoketest.toml: lr=1e-4, epsilon=1e-8 (f32 Adam values)
- Removed debug printfs from gather kernels
- Added curiosity_weight to training profile system
- Cleaned up smoke_params() inline overrides
11/11 smoke tests pass, 5/5 stress runs of 50-epoch test pass,
359/359 ml-dqn + 895/895 ml unit tests pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Agent fixes: kernel name mismatches, sgemm→GemmEx BF16 in linear.rs
and stream_ops.rs, shared memory sizes for BF16 kernels, BF16-safe
optimizer betas (0.999 rounds to 1.0 in BF16), argmax index readback,
test tolerances relaxed for BF16 precision (~3 decimal digits).
ml-core: 300 passed, 0 failed.
ml-dqn: 304 passed, 55 failed (4 dead nvrtc stubs — separate task).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace all 20 dangling `ptx` variable references with correct cubin
static names after nvrtc removal sed. Fix ml-dqn dead code stubs
(residual.rs, rmsnorm.rs, noisy_layers.rs, gpu_replay_buffer.rs).
Clean up unused `let ptx` variables.
Zero compilation errors across full workspace.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
GELU forward and backward now use bf16_tanh from common header
instead of float tanhf. tanh(x) = (exp(2x)-1)/(exp(2x)+1) via
bf16_exp — all native __nv_bfloat16 arithmetic. No exceptions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CUDA kernels: 36 of 37 compiled with __nv_bfloat16* (dt_kernels.cu remaining)
Rust types: ALL CudaSlice<f32> → CudaSlice<half::bf16> across ml + ml-core
Build: all kernels now compiled with common_device_functions.cuh (BF16 helpers)
BF16 math wrappers: bf16_sqrt, bf16_log, bf16_exp, bf16_pow, bf16_fabs, bf16_fmax,
bf16_fmin, bf16_cos, bf16_zero, bf16_one, bf16(), atomicAddBF16
NOT YET COMPILING — ml-core boundary errors (Vec<f32> → Vec<half::bf16>)
and dt_kernels.cu float*__nv_bfloat16 ambiguity remain.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
check_err() now logs warnings for real kernel errors instead of let _ =.
Removed max_steps_per_epoch from smoketest TOML to match working config.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Smoke test loads all hyperparams from TOML profile instead of hardcoding.
TOML: hidden_dim=64, batch=64, lr=0.0003 (stable on RTX 3050 + H100).
Restored check_err() drain in device.rs — required to clear stale CUDA
errors from primary context reuse between tests.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- test_embedded_h100_parses: update assertions to match h100.toml values
(gpu_n_episodes=2048, gpu_timesteps_per_episode=100)
- dqn_training_smoke_test: apply dqn-smoketest profile to cap hidden_dim=32.
H100's gpu profile sets hidden_dim_base=256 which causes loss explosion
(375x in 3 epochs) with lr=0.001.
- Revert gpu-test-pipeline DAG to compile-and-test (RWO PVC constraint)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove 6 eprintln!("[GPU-DEBUG]...") statements from gpu_dqn_trainer.rs,
constructor.rs, and elementwise.rs
- Delete log_gpu_memory() function and all 18 callers in smoke test files
- Remove check_err() drains from GpuDqnTrainer::new() and
GpuExperienceCollector::new() — root cause is fixed (per-context kernel cache)
- Keep check_err() after CUDA Graph capture (legitimate — drains event tracking errors)
- Fix broken import lines in smoke test files after sed cleanup
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: ElementwiseKernels was cached in a static OnceLock, compiled
once on the first test's CudaContext. When the second test created a new
CudaContext (fresh Arc), the cached CudaFunction handles were stale,
causing CUDA_ERROR_INVALID_VALUE on alloc_zeros (n=128, op=abs).
Fix: Replace OnceLock with a Mutex<HashMap<usize, Arc<ElementwiseKernels>>>
keyed by Arc<CudaContext> pointer address. Each distinct CudaContext gets
fresh kernel compilation. Old entries are evicted on context change.
Also: eliminate ALL GpuTensor from DQN training path (metrics.rs,
training_loop.rs). Replace with CPU computation for validation (cold path)
and raw CudaSlice for replay buffer insertion. Log deferred CUDA errors
instead of silently swallowing.
Result: ALL 6 sequential GPU smoke tests pass (was 1 failing).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: `SMOKE_CUDA: OnceLock<MlDevice>` held a static Arc<CudaContext>
for the process lifetime. CudaSlice Drop from test N recorded errors on
this shared context's error_state, causing test N+1's bind_to_thread() to
fail with CUDA_ERROR_INVALID_VALUE.
Fix: create a fresh MlDevice per test (no static caching). Each test gets
its own CudaContext Arc with clean error_state.
Also: convert all GPU smoke tests from #[tokio::test] to synchronous #[test]
with explicit tokio::runtime::Builder::new_current_thread(). The runtime is
explicitly dropped between tests, ensuring all Arc<CudaContext> refs are freed.
Result: 5 of 6 sequential smoke tests now pass. The remaining 1 failure is
a real Candle GpuTensor bug: the replay buffer insertion path still uses
Candle's elementwise kernels, which cache CudaFunction handles that become
stale across test boundaries. Fix: eliminate Candle from replay buffer path.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Created config/gpu/{default,rtx3050,h100,a100}.toml with all GPU-specific
parameters: batch_size, num_atoms, buffer_size, hidden_dim_base,
replay_buffer_vram_fraction, gpu_n_episodes, gpu_timesteps_per_episode,
cuda_stack_bytes.
GpuProfile::load() auto-detects GPU by device name, falls back to
embedded defaults (include_str!). Override via FOXHUNT_GPU_PROFILE env.
Removed dead code:
- detect_vram_mb(), vram_scaled_hidden_dims(), vram_scaled_base_dim(),
resolve_hidden_dim_base() + 18 tests for these functions
All callers updated: train_baseline_rl, DQNTrainer constructor,
PPO trainer, smoke tests, pipeline tests.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace scattered VRAM-based if/else chains with a declarative TOML profile
system. GPU profiles (rtx3050, a100, h100, default) are selected by device
name and embedded at compile time via include_str! for zero-filesystem
fallback in CI/containers, with filesystem and env var overrides.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
