Pass 2 (target forward) and Pass 3 (Double DQN online forward)
run concurrently on main stream + forked double_dqn_stream.
Fork/join via CUDA events captured in graph topology.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fork 3 CUDA streams at CublasForward construction. In forward_online_raw
and forward_online_f32, the trunk records an event, each branch stream
waits on it, submits its GEMM+bias ops on its own stream (via
cublasSetStream), then the main stream joins all three. This overlaps
the 3 independent advantage head computations (exposure, order, urgency)
that previously executed sequentially.
Safety: a distinct_branches guard checks h_b0!=h_b1!=h_b2 at runtime;
callers that alias branch hidden buffers (Pass 3 Double DQN scratch
reuse) fall back to the sequential loop. CUDA Graph capture (CUDA 12+)
captures the fork/join pattern as graph dependencies.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace stream.synchronize() between experience collection and training
with a CUDA event recorded on the forked stream. The event is checked
lazily at the start of run_training_steps, allowing CPU-side setup
(can_train, ensure_fused_ctx, guard init, variable capture) to overlap
with the tail of GPU experience collection kernels.
Also converts the curiosity kernel sync to the event pattern with
non-blocking is_complete() poll before synchronize().
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Set batch_size=0 in config/gpu/h100.toml as sentinel for auto-compute.
The constructor now treats batch_size==0 as "let AutoBatchSizer drive":
it uses the VRAM-computed ceiling capped at 8192, instead of the static
1024 that was wasting >90% of H100's 80GB VRAM bandwidth.
Previously AutoBatchSizer computed the optimal batch (e.g. 2085808) but
the profile's batch_size=1024 always won. Now with batch_size=0 the
sizer's result flows through, enabling full SM occupancy on H100.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace synchronous memcpy_dtoh in reduce_current_q_stats() with
cuMemcpyDtoHAsync_v2 + CudaEvent pattern. The function now returns the
PREVIOUS call's results (one-step delay) while the current reduction
runs asynchronously, eliminating a GPU stall every 50 training steps.
Adds flush_q_stats_readback() to drain the final in-flight readback at
epoch end, matching the existing flush_readback() pattern for loss/grad_norm.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace the blocking stream.synchronize() in replay_adam_and_readback()
with async DtoH copies + CUDA event recording. Each step now returns
the previous step's scalars while the current step's readback lands
asynchronously. A flush_readback() call at epoch end collects the
final in-flight transfer. This eliminates 341 per-epoch CPU stalls
where the H100 sat idle waiting for Rust to re-enter the loop.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
precompute passes data-dir/SYMBOL to match how train_baseline_rl
calls load_all_bars(data_dir, symbol). compile-and-train gets
FOXHUNT_FEATURE_CACHE_DIR env var and --epochs fix.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Reuses compile-training, gpu-warmup, fetch-binary templates from
compile-and-train. Own train step runs train_baseline_rl directly
with --epochs, FOXHUNT_FEATURE_CACHE_DIR, and fxcache validation.
No hyperopt — pure baseline with default params.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- DAG: compile (ci-compile-cpu) + gpu-warmup → train (H100)
- Compiles from source targeting compute cap 90
- fxcache REQUIRED — fails if no .fxcache found
- Error chains printed with {:#} for full cause visibility
- Uses cargo-target-pvc for compile cache + sccache
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Dedicated workflow for baseline RL training without hyperopt.
Runs train_baseline_rl binary on GPU node with fxcache.
Fetches pre-compiled binary from MinIO, falls back to PVC.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each record now starts with an i64 timestamp (nanoseconds since epoch)
before the feature/target/OFI data. v1 records grow from 432 to 440
bytes, v2 from 112 to 120 bytes. The timestamp is always i64 even in
bf16 mode. train_baseline_rl reconstructs bars with real timestamps
instead of placeholders so the walk-forward windower can split by month.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Auto-discovers fxcache via env var or sibling directory. Falls back
to DBN loading if no cache found. Reconstructs aligned bars from
cached targets for walk-forward windowing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All pods with app.kubernetes.io/part-of=foxhunt get base infra
access. Fixes MinIO log archival timeout for precompute-features
and databento-download workflows. Component-specific policies
still add extra rules (GitLab SSH, external HTTPS, etc.).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Previous 32Gi OOM'd with 148GB of MBP-10 data (61M snapshots ≈ 15GB
in memory + trades + OHLCV). ci-compile-cpu node has 64GB.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Cache key hashes filename + size + mtime instead of full paths.
Resolves mbp10/trades relative paths by walking up from data_dir.
Ensures precompute binary and cargo test produce the same key
regardless of working directory.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Precompute binary now uses calculate_dbn_cache_key_full from the
library instead of its own compute_cache_key. Canonicalize all
paths in the key function so relative/absolute produce the same
key. Walk up parent dirs for sibling feature-cache discovery.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Exact key match only — no fallback to loading unvalidated caches.
On miss, stale .fxcache files are deleted to prevent disk bloat
and confusion from outdated cached data.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Removed 2.6GB of uncompressed .dbn files (derivable from .zst).
Renamed directory to mbp10-fixtures to distinguish unit test
fixtures from training data. Updated all test references to use
.dbn.zst paths directly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
load_training_data() checks for .fxcache in:
1. Explicit feature_cache_dir (set via with_feature_cache())
2. $FOXHUNT_FEATURE_CACHE_DIR env var
3. Sibling feature-cache/ directory next to data dir
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extract standalone functions from DQNTrainer:
- extract_features_from_bars(): 42-dim feature extraction
- extract_ohlcv_bars_from_dbn(): DBN to OHLCVBar conversion
- collect_dbn_files_recursive(): file discovery
DQNTrainer delegates to these, ensuring consistency.
Precompute binary no longer initializes CUDA — runs on any
CPU node (ci-compile-cpu at EUR 0.85/hr vs GPU at EUR 12.60/hr).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
DQNTrainer constructor requires CUDA for regime-conditional heads.
Use DQNTrainer::new() (GPU) instead of new_with_device(Cpu).
Set buffer_size=100_000 to satisfy PER minimum threshold.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds feature-cache-dir parameter (default: /data/feature-cache) so
the training binary can load pre-computed .fxcache files.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CLI binary that reads OHLCV + MBP-10 + trades DBN data via DQNTrainer,
runs the full feature extraction pipeline, and writes a .fxcache file
for zero-overhead GPU loading during training. Uses CPU device to avoid
CUDA dependency for data-only workloads.
Also promotes load_training_data() and ofi_features to pub visibility
so the example binary can access them.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All downloads now use the Rust download_baseline binary via the
Argo WorkflowTemplate (databento-download).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
500m/512Mi request fits alongside existing platform workloads.
Download is I/O-bound, not CPU-bound.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
3 retries with exponential backoff (2s, 4s, 8s) for transient
connection failures (IncompleteBody/UnexpectedEof). Partial files
are deleted before retry so skip-existing logic works correctly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove runAsUser:1000 so apt-get install ca-certificates works.
Fix double-nested ES.FUT path (output-dir is the schema root, not ES.FUT subdir).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: ubuntu:24.04 has no CA certificates, rustls-native-certs
fails TLS handshake. Fix: apt-get install ca-certificates before
running download binary. Also add port 80 to network policy for apt.
Downloads OHLCV-1m, Trades, OHLCV-1s, MBP-10 via streaming API
(get_range) — parallel quarterly downloads, individual .dbn.zst files.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Set HOME=/tmp to fix pip cache ownership, --no-cache-dir for clean install.
Label compile-and-train matches network policy for HTTPS egress.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Downloads completed batch jobs (mbp-10, trades, ohlcv-1m, ohlcv-1s, mbp-1)
to training-data-pvc. Cleans old data per schema before downloading.
Uses existing databento-credentials secret.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Integration test now loads imbalance_bar_threshold and ewma_alpha from
config/training/dqn-smoketest.toml. Single source of truth for all
config values. Production threshold lowered to 0.5 for maximum bar yield.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
First load: full MBP-10 decompression (~450s for 19G).
Subsequent loads: bincode deserialize (<1s).
Cache key: hash(dir + symbol + threshold + alpha).
Auto-invalidates when any source .dbn file is newer than cache.
Cache failures are non-fatal — falls through to recomputation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
