HER relabeling now uses a dedicated CUDA kernel (her_inplace_relabel)
that writes goal columns + rewards directly into the trainer's padded
staging buffers. Zero intermediate GpuBatch allocation.
Kernel: her_inplace_relabel in dqn_utility_kernels.cu
- One warp (32 threads) per relabeled sample
- Coalesced column writes for goal_dim columns
- Sets rewards = 1.0 for HER samples
- Works for any goal_dim (1..state_dim)
Added to GpuDqnTrainer as 27th utility kernel.
launch_her_inplace_relabel() method takes raw pointers — zero cudarc
event recording overhead.
Deleted 6 test files that tested the removed CPU training path:
- dqn_checkpoint_loading_test.rs
- ensemble_real_models_validation_test.rs
- dqn_iqn_integration_test.rs
- validation_real_data_test.rs
- dqn_training_smoke_test.rs
- validation_harness_integration_test.rs
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
EMA (target_ema_update): removed cuStreamSynchronize + check_err.
EventTrackingGuard alone disables cudarc event recording.
Stream ordering guarantees graph replay completed before EMA.
First call still syncs for initial flatten_target_weights.
Attention (apply_attention_forward): removed cuStreamSynchronize
+ check_err. EventTrackingGuard sufficient. Stream ordering
guarantees save_h_s2 is written before attention reads it.
Root cause of previous hangs was the sync memcpy_htod for
adam_step (fixed in previous commit with cuMemcpyHtoDAsync_v2),
not these syncs. With async adam_step, the EventTrackingGuard
alone prevents stale cudarc events without pipeline drain.
Per-step hot path now has ZERO cuStreamSynchronize calls.
Only remaining sync: 1-step-lagged readback event.synchronize()
which fires only if previous async DtoH hasn't completed (~never).
Expected: 129ms/step → ~10-15ms/step (GPU compute only).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The batch_size >128 hang was caused by cudarc's synchronous
memcpy_htod for the adam_step counter. At batch_size=128 the
sync completes fast enough, but at 509+ the pipeline drain
from the sync interacts with CUDA Graph replay timing and
deadlocks the stream.
Replaced all 3 memcpy_htod(&[self.adam_step]) calls with raw
cuMemcpyHtoDAsync_v2 — zero pipeline drain, zero CPU sync.
Production TOML set to batch_size=0 (AutoBatchSizer drives it).
AutoBatchSizer caps at 8192 for RL training.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
batch_size=509 (from VRAM floor) and batch_size=1024 (from TOML)
both hang the training loop after CUDA graph capture. Only
batch_size=128 is known to work. Root cause unknown — likely a
cudarc or CUDA Graph limitation with larger buffer sizes.
Removed the VRAM floor that scaled 128→509. Production TOML
set to batch_size=128 (known working). The batch_size hang
investigation is part of the mega-graph refactor plan.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
cudarc's EventTrackingGuard + check_err alone is NOT sufficient.
The cuStreamSynchronize is required every step because cudarc's
internal event state machine becomes corrupted without it —
training hangs indefinitely after graph capture.
This is a cudarc limitation: CUDA Graph replay generates stale
events that accumulate and eventually block kernel launches.
The sync drains the pipeline (~150µs) but prevents the hang.
Removing these syncs requires either:
1. Patching cudarc to not record events on graph-replayed buffers
2. Using raw CUDA driver API without cudarc's event tracking
3. The mega-graph refactor (all ops in one graph, no cudarc ops between)
The other perf wins (vaccine throttle, actions DtoD, HER GPU-native,
causal interval) remain active. Expected epoch: ~500s (vs 614s baseline).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The hang was caused by cudarc accumulating stale CudaEvent errors
from CUDA Graph replays. EventTrackingGuard + check_err must run
every step to drain these errors. The cuStreamSynchronize is only
needed on the very first call to clear the initial backlog from
graph capture — subsequent steps rely on stream ordering.
Guard (~100ns) + check_err (~50ns) per step is negligible.
The cuStreamSynchronize (~150µs pipeline drain) is eliminated
on all but the first step.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The full cuStreamSynchronize removal hung the training loop because
cudarc's EventTrackingGuard + check_err need one initial sync to clear
stale events from CUDA Graph capture. After the first call, stream
ordering is sufficient.
- target_ema_update: sync on first call only (target_params_initialized)
- apply_attention_forward: sync on first call only (attention_initialized)
- All subsequent steps: zero sync, pure async kernel dispatch
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
HER donor indices: removed memcpy_dtoh (was ~200B per step).
gpu_her_relabel_batch_with_donors now accepts CudaSlice<i32>
directly — gather kernel reads GPU-resident donor indices.
Reinterpret i32→u32 via pointer cast (safe for non-negative).
HER source indices: pre-computed once at init on GPU.
relabel_batch_with_strategy now accepts CudaSlice<i32>
instead of &[i32] — DtoD copy replaces per-step HtoD upload.
HER reward ones: pre-allocated CudaSlice<f32> at init.
Both Random and Future/Final HER paths use DtoD copy from
pre-allocated buffer instead of per-step vec![1.0; N] + HtoD.
Result: fused_training.rs now has ZERO memcpy_dtoh calls.
The only remaining GPU→CPU transfer in the entire training step
is the 1-step-lagged async loss/grad_norm readback via CUDA event
(non-blocking, overlapped with next step's compute).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove 3 per-step CPU-GPU synchronization points that dominated the
300ms/step wall time (pure compute for 323K params is <1ms on H100):
1. target_ema_update: removed cuStreamSynchronize — stream ordering
guarantees EMA kernel sees Adam-updated weights (same stream).
2. apply_attention_forward: removed cuStreamSynchronize — save_h_s2
is written by graph_forward on the same stream.
3. Actions DtoH round-trip: GpuBatch.actions changed from GpuTensor
(bf16) to CudaSlice<i32>. Eliminates synchronous GPU→CPU→GPU
round-trip (bf16 download → i32 cast → upload) every training step.
Actions now flow u32 → i32 via async DtoD in the replay buffer.
Throttle expensive per-step features:
4. Gradient vaccine: runs every 10 steps (was every step). Full
ungraphed forward+backward pass was ~100-150ms — the single
largest bottleneck. 10-step amortization preserves gradient
quality with ~90% cost reduction.
5. Causal intervention interval: 10 → 100. Each invocation runs
14 cuBLAS forward passes + sync + readback.
Dead code removed:
- u32_slice_to_gpu_tensor_gpu (56 lines) — obsolete bf16 cast path
- u32_to_f32 CastKernel field — no longer needed
- Old train_step fallback in training_loop — fused path only
Expected: ~300ms/step → ~25ms/step → ~50s/epoch (was 614s)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sharpe calculation:
- Use per-trade returns (sum_returns/sum_sq_returns) instead of per-bar
step_returns. Per-bar Sharpe collapsed variance → bogus 19.75 with PF=0.03.
- Annualize by sqrt(trades_per_year) not sqrt(bars_per_year).
Batch sizing:
- Cap auto-computed batch_size at 8192 (VRAM ceiling of 2M is OOM limit, not
optimal RL batch).
- Add VRAM floor: batch_size < ceiling/4096 gets scaled UP (128 → 512 on H100).
- Only let hyperopt override batch_size when explicitly non-zero — preserve
profile's batch_size=0 auto-compute sentinel.
Replay buffer:
- Divide per_max_memory_bytes by 3.0 for regime heads (PER budget was 3x too
large, causing OOM cascade 74M → 37M → 18M → 9M → 4.6M).
- HER buffer uses original_buffer_size (pre-autosizer), not inflated 74M.
Q-value clipping:
- Wire hyperparams.q_clip_min/max to DQNConfig (was hardcoded ±500, production
TOML has ±50). Prevents Q-value overestimation ratio of 94.6x at epoch 2.
Training stability:
- Anti-LR warmup: skip first 5 epochs (early Sharpe unreliable from random
policy). Prevents bogus 3x LR boost at epoch 2.
- min_replay_size from profile (1000), not hyperopt batch_size (128).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Deleted the hardcoded 20% fallback function. Constructor now uses
vram_fraction directly for initial PER budget. No hardcoded limits
anywhere in the sizing pipeline.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Removed MAX_REPLAY_CAPACITY = 10M and STATIC_MAX_BATCH_SIZE = 8192.
Removed MIN_REPLAY_CAPACITY = 100K (replaced with 1024 segment tree minimum).
AutoBatchSizer computes from actual free VRAM. Replay buffer uses
vram_fraction (0.70 for H100) instead of hardcoded 20%.
H100 80GB: batch ~2M ceiling, replay ~89M entries (was capped at 10M).
RTX 3050 4GB: still auto-scales to small values safely.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Change PREFETCH_K from 32 to usize::MAX so the chunked
step_by loop iterates exactly once, acquiring the read lock
a single time per epoch instead of every 32 steps.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace host-side `current_t` scalar parameter in experience_env_step
with a GPU-resident counter buffer. The kernel now reads the timestep
index from device memory (step_counter_gpu[0]) instead of receiving it
as a kernel argument that changes every iteration. A tiny single-thread
step_counter_advance kernel increments the counter after each env_step.
This eliminates per-timestep host→GPU parameter variation in the 100-
iteration experience collection loop, making all iterations dispatch
identical kernel argument sets — a prerequisite for future CUDA Graph
capture of the timestep sequence.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Pass 2 (target forward) and Pass 3 (Double DQN online forward)
run concurrently on main stream + forked double_dqn_stream.
Fork/join via CUDA events captured in graph topology.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fork 3 CUDA streams at CublasForward construction. In forward_online_raw
and forward_online_f32, the trunk records an event, each branch stream
waits on it, submits its GEMM+bias ops on its own stream (via
cublasSetStream), then the main stream joins all three. This overlaps
the 3 independent advantage head computations (exposure, order, urgency)
that previously executed sequentially.
Safety: a distinct_branches guard checks h_b0!=h_b1!=h_b2 at runtime;
callers that alias branch hidden buffers (Pass 3 Double DQN scratch
reuse) fall back to the sequential loop. CUDA Graph capture (CUDA 12+)
captures the fork/join pattern as graph dependencies.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace stream.synchronize() between experience collection and training
with a CUDA event recorded on the forked stream. The event is checked
lazily at the start of run_training_steps, allowing CPU-side setup
(can_train, ensure_fused_ctx, guard init, variable capture) to overlap
with the tail of GPU experience collection kernels.
Also converts the curiosity kernel sync to the event pattern with
non-blocking is_complete() poll before synchronize().
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Set batch_size=0 in config/gpu/h100.toml as sentinel for auto-compute.
The constructor now treats batch_size==0 as "let AutoBatchSizer drive":
it uses the VRAM-computed ceiling capped at 8192, instead of the static
1024 that was wasting >90% of H100's 80GB VRAM bandwidth.
Previously AutoBatchSizer computed the optimal batch (e.g. 2085808) but
the profile's batch_size=1024 always won. Now with batch_size=0 the
sizer's result flows through, enabling full SM occupancy on H100.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace synchronous memcpy_dtoh in reduce_current_q_stats() with
cuMemcpyDtoHAsync_v2 + CudaEvent pattern. The function now returns the
PREVIOUS call's results (one-step delay) while the current reduction
runs asynchronously, eliminating a GPU stall every 50 training steps.
Adds flush_q_stats_readback() to drain the final in-flight readback at
epoch end, matching the existing flush_readback() pattern for loss/grad_norm.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace the blocking stream.synchronize() in replay_adam_and_readback()
with async DtoH copies + CUDA event recording. Each step now returns
the previous step's scalars while the current step's readback lands
asynchronously. A flush_readback() call at epoch end collects the
final in-flight transfer. This eliminates 341 per-epoch CPU stalls
where the H100 sat idle waiting for Rust to re-enter the loop.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- DAG: compile (ci-compile-cpu) + gpu-warmup → train (H100)
- Compiles from source targeting compute cap 90
- fxcache REQUIRED — fails if no .fxcache found
- Error chains printed with {:#} for full cause visibility
- Uses cargo-target-pvc for compile cache + sccache
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each record now starts with an i64 timestamp (nanoseconds since epoch)
before the feature/target/OFI data. v1 records grow from 432 to 440
bytes, v2 from 112 to 120 bytes. The timestamp is always i64 even in
bf16 mode. train_baseline_rl reconstructs bars with real timestamps
instead of placeholders so the walk-forward windower can split by month.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Auto-discovers fxcache via env var or sibling directory. Falls back
to DBN loading if no cache found. Reconstructs aligned bars from
cached targets for walk-forward windowing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Cache key hashes filename + size + mtime instead of full paths.
Resolves mbp10/trades relative paths by walking up from data_dir.
Ensures precompute binary and cargo test produce the same key
regardless of working directory.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Precompute binary now uses calculate_dbn_cache_key_full from the
library instead of its own compute_cache_key. Canonicalize all
paths in the key function so relative/absolute produce the same
key. Walk up parent dirs for sibling feature-cache discovery.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Exact key match only — no fallback to loading unvalidated caches.
On miss, stale .fxcache files are deleted to prevent disk bloat
and confusion from outdated cached data.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Removed 2.6GB of uncompressed .dbn files (derivable from .zst).
Renamed directory to mbp10-fixtures to distinguish unit test
fixtures from training data. Updated all test references to use
.dbn.zst paths directly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
load_training_data() checks for .fxcache in:
1. Explicit feature_cache_dir (set via with_feature_cache())
2. $FOXHUNT_FEATURE_CACHE_DIR env var
3. Sibling feature-cache/ directory next to data dir
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extract standalone functions from DQNTrainer:
- extract_features_from_bars(): 42-dim feature extraction
- extract_ohlcv_bars_from_dbn(): DBN to OHLCVBar conversion
- collect_dbn_files_recursive(): file discovery
DQNTrainer delegates to these, ensuring consistency.
Precompute binary no longer initializes CUDA — runs on any
CPU node (ci-compile-cpu at EUR 0.85/hr vs GPU at EUR 12.60/hr).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
DQNTrainer constructor requires CUDA for regime-conditional heads.
Use DQNTrainer::new() (GPU) instead of new_with_device(Cpu).
Set buffer_size=100_000 to satisfy PER minimum threshold.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CLI binary that reads OHLCV + MBP-10 + trades DBN data via DQNTrainer,
runs the full feature extraction pipeline, and writes a .fxcache file
for zero-overhead GPU loading during training. Uses CPU device to avoid
CUDA dependency for data-only workloads.
Also promotes load_training_data() and ofi_features to pub visibility
so the example binary can access them.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
3 retries with exponential backoff (2s, 4s, 8s) for transient
connection failures (IncompleteBody/UnexpectedEof). Partial files
are deleted before retry so skip-existing logic works correctly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Integration test now loads imbalance_bar_threshold and ewma_alpha from
config/training/dqn-smoketest.toml. Single source of truth for all
config values. Production threshold lowered to 0.5 for maximum bar yield.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
First load: full MBP-10 decompression (~450s for 19G).
Subsequent loads: bincode deserialize (<1s).
Cache key: hash(dir + symbol + threshold + alpha).
Auto-invalidates when any source .dbn file is newer than cache.
Cache failures are non-fatal — falls through to recomputation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Relative TOML paths now resolved via CARGO_MANIFEST_DIR → workspace root,
matching test_data_dir() behavior. MBP-10 smoke test passes: 654 imbalance
bars from Q1+Q2 2024, threshold=25.0.
Smoketest config switched to data_source="mbp10" for MBP-10 validation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace if-let-Ok pattern with #[ignore] test that asserts on results.
Uses CARGO_MANIFEST_DIR to resolve workspace root. Validates bar
ordering, positive volume, valid OHLC. Produces 654 imbalance bars
from Q1+Q2 2024 ES MBP-10 data (448s load time).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add --min-hold-bars CLI arg to train_baseline_rl and hyperopt_baseline_rl.
Wire through Argo workflow as parameter. Default 5 (TOML), override via
CLI for quick A/B experiments without config changes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Wire MBP-10 order book data through ImbalanceBarSampler into the DQN
training pipeline. No fallback — explicit data_source config: "mbp10"
(imbalance bars from 10-level order book) or "ohlcv" (1-minute candles).
Fails loudly if chosen source's data doesn't exist.
Pipeline: MBP-10 .dbn.zst → trade extraction (action=='T', native side
classification) → adaptive ImbalanceBarSampler (EWMA threshold) →
OHLCVBar → existing feature extraction.
Production: data_source="mbp10", smoketest/localdev: data_source="ohlcv".
8 files, +483/-48 lines. 3 new tests for trade extraction and pipeline.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove 8 enable_* from FeatureConfig (ml-features) and 24 from
DQNHyperparameters (ml). All features are always active — no boolean
toggles, no dead conditional branches, no false impression of optionality.
FeatureConfig reduced to single `phase: FeaturePhase` field.
DQNHyperparameters loses 24 fields, downstream conditionals collapsed.
TOML configs cleaned of all enable_* lines.
16 files changed, -461/+181 lines.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
train_baseline_rl now accepts --initial-capital (default $35K) matching
hyperopt. Argo compile-and-train passes the workflow parameter to the
train-best step. Both hyperopt and training now use consistent capital.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace pure time-sequential expanding-window splits with stratified
splits that adjust fold boundaries to balance Trending/Ranging/Volatile
proportions. Slides boundaries up to 25% of validation window when
deviation exceeds 10pp from global average. Strictly temporal — no
data shuffling, only boundary adjustments.
This directly addresses the R²=1.0 finding: IS→OOS Sharpe gap was
entirely regime-driven because folds had wildly different regime mixes.
Stratification ensures each fold sees similar market conditions.
7 unit tests for regime classification, distribution, deviation, and
fold generation with both uniform and imbalanced data.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add futures-baseline/ to the auto-detection search paths so smoke tests
work without FOXHUNT_TEST_DATA env var. All 11 smoke tests pass on
RTX 3050 Ti (960s, Sharpe 7.76).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase system now controls which family intensities are searchable vs
pinned in the 14D layout. Fast (default) searches ALL 14 dims — phased
search was needed for old 30D space but 14D is within PSO's efficient
range. Full/Reward/Risk phases remain for targeted refinement.
Fix test_phase_fast_bounds to verify all dims searchable in Fast.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Task 14: Normalize OOS Sharpe by regime difficulty (volatile folds weighted
up, trending folds down). Compute R² between volatile% and Sharpe across
folds — logs verdict: regime-driven (R²>0.7), model-driven (R²<0.3),
or mixed.
Task 15: Adaptive regime_replay_decay per fold. More concentrated regime →
lower decay → more aggressive PER bias toward dominant regime. Wired
through ReplayBufferType::sample_regime_biased() into the training loop.
Vaccine batches remain unbiased for gradient diversity.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
