Auto-discovers fxcache via env var or sibling directory. Falls back
to DBN loading if no cache found. Reconstructs aligned bars from
cached targets for walk-forward windowing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All pods with app.kubernetes.io/part-of=foxhunt get base infra
access. Fixes MinIO log archival timeout for precompute-features
and databento-download workflows. Component-specific policies
still add extra rules (GitLab SSH, external HTTPS, etc.).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Previous 32Gi OOM'd with 148GB of MBP-10 data (61M snapshots ≈ 15GB
in memory + trades + OHLCV). ci-compile-cpu node has 64GB.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Cache key hashes filename + size + mtime instead of full paths.
Resolves mbp10/trades relative paths by walking up from data_dir.
Ensures precompute binary and cargo test produce the same key
regardless of working directory.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Precompute binary now uses calculate_dbn_cache_key_full from the
library instead of its own compute_cache_key. Canonicalize all
paths in the key function so relative/absolute produce the same
key. Walk up parent dirs for sibling feature-cache discovery.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Exact key match only — no fallback to loading unvalidated caches.
On miss, stale .fxcache files are deleted to prevent disk bloat
and confusion from outdated cached data.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Removed 2.6GB of uncompressed .dbn files (derivable from .zst).
Renamed directory to mbp10-fixtures to distinguish unit test
fixtures from training data. Updated all test references to use
.dbn.zst paths directly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
load_training_data() checks for .fxcache in:
1. Explicit feature_cache_dir (set via with_feature_cache())
2. $FOXHUNT_FEATURE_CACHE_DIR env var
3. Sibling feature-cache/ directory next to data dir
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extract standalone functions from DQNTrainer:
- extract_features_from_bars(): 42-dim feature extraction
- extract_ohlcv_bars_from_dbn(): DBN to OHLCVBar conversion
- collect_dbn_files_recursive(): file discovery
DQNTrainer delegates to these, ensuring consistency.
Precompute binary no longer initializes CUDA — runs on any
CPU node (ci-compile-cpu at EUR 0.85/hr vs GPU at EUR 12.60/hr).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
DQNTrainer constructor requires CUDA for regime-conditional heads.
Use DQNTrainer::new() (GPU) instead of new_with_device(Cpu).
Set buffer_size=100_000 to satisfy PER minimum threshold.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds feature-cache-dir parameter (default: /data/feature-cache) so
the training binary can load pre-computed .fxcache files.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CLI binary that reads OHLCV + MBP-10 + trades DBN data via DQNTrainer,
runs the full feature extraction pipeline, and writes a .fxcache file
for zero-overhead GPU loading during training. Uses CPU device to avoid
CUDA dependency for data-only workloads.
Also promotes load_training_data() and ofi_features to pub visibility
so the example binary can access them.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All downloads now use the Rust download_baseline binary via the
Argo WorkflowTemplate (databento-download).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
500m/512Mi request fits alongside existing platform workloads.
Download is I/O-bound, not CPU-bound.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
3 retries with exponential backoff (2s, 4s, 8s) for transient
connection failures (IncompleteBody/UnexpectedEof). Partial files
are deleted before retry so skip-existing logic works correctly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove runAsUser:1000 so apt-get install ca-certificates works.
Fix double-nested ES.FUT path (output-dir is the schema root, not ES.FUT subdir).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: ubuntu:24.04 has no CA certificates, rustls-native-certs
fails TLS handshake. Fix: apt-get install ca-certificates before
running download binary. Also add port 80 to network policy for apt.
Downloads OHLCV-1m, Trades, OHLCV-1s, MBP-10 via streaming API
(get_range) — parallel quarterly downloads, individual .dbn.zst files.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Set HOME=/tmp to fix pip cache ownership, --no-cache-dir for clean install.
Label compile-and-train matches network policy for HTTPS egress.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Downloads completed batch jobs (mbp-10, trades, ohlcv-1m, ohlcv-1s, mbp-1)
to training-data-pvc. Cleans old data per schema before downloading.
Uses existing databento-credentials secret.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Integration test now loads imbalance_bar_threshold and ewma_alpha from
config/training/dqn-smoketest.toml. Single source of truth for all
config values. Production threshold lowered to 0.5 for maximum bar yield.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
First load: full MBP-10 decompression (~450s for 19G).
Subsequent loads: bincode deserialize (<1s).
Cache key: hash(dir + symbol + threshold + alpha).
Auto-invalidates when any source .dbn file is newer than cache.
Cache failures are non-fatal — falls through to recomputation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Relative TOML paths now resolved via CARGO_MANIFEST_DIR → workspace root,
matching test_data_dir() behavior. MBP-10 smoke test passes: 654 imbalance
bars from Q1+Q2 2024, threshold=25.0.
Smoketest config switched to data_source="mbp10" for MBP-10 validation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace if-let-Ok pattern with #[ignore] test that asserts on results.
Uses CARGO_MANIFEST_DIR to resolve workspace root. Validates bar
ordering, positive volume, valid OHLC. Produces 654 imbalance bars
from Q1+Q2 2024 ES MBP-10 data (448s load time).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add --min-hold-bars CLI arg to train_baseline_rl and hyperopt_baseline_rl.
Wire through Argo workflow as parameter. Default 5 (TOML), override via
CLI for quick A/B experiments without config changes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Wire MBP-10 order book data through ImbalanceBarSampler into the DQN
training pipeline. No fallback — explicit data_source config: "mbp10"
(imbalance bars from 10-level order book) or "ohlcv" (1-minute candles).
Fails loudly if chosen source's data doesn't exist.
Pipeline: MBP-10 .dbn.zst → trade extraction (action=='T', native side
classification) → adaptive ImbalanceBarSampler (EWMA threshold) →
OHLCVBar → existing feature extraction.
Production: data_source="mbp10", smoketest/localdev: data_source="ohlcv".
8 files, +483/-48 lines. 3 new tests for trade extraction and pipeline.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove 8 enable_* from FeatureConfig (ml-features) and 24 from
DQNHyperparameters (ml). All features are always active — no boolean
toggles, no dead conditional branches, no false impression of optionality.
FeatureConfig reduced to single `phase: FeaturePhase` field.
DQNHyperparameters loses 24 fields, downstream conditionals collapsed.
TOML configs cleaned of all enable_* lines.
16 files changed, -461/+181 lines.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
templateRef doesn't inherit workflow-level volumes from the referenced
template. All 3 train wrappers (DQN, PPO, supervised) now define
git-ssh-key, training-data, cargo-target-cuda volumes and workspace
volumeClaimTemplate locally.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Migrate from training-pipeline (requires pre-built binary-tag) to
compile-and-train (compiles from source). All 3 train templates now
use the same pattern: thin wrapper → compile-and-train → pipeline.
Consistent params: commit-sha + git-branch instead of binary-tag.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Was using commit-ref which doesn't exist in compile-and-train template.
Now passes commit-sha=HEAD + git-branch=main matching the parent template.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Line 18 had 6-space indent instead of 4 under metadata.labels.
This caused all CI pipelines to fail at apply-argo-templates since our
compile-and-train changes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
train_baseline_rl now accepts --initial-capital (default $35K) matching
hyperopt. Argo compile-and-train passes the workflow parameter to the
train-best step. Both hyperopt and training now use consistent capital.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Was $100K (default) but workflow passes $35K and hyperopt optimizes for
$35K. Model must learn conservative position sizing for real retail
capital, not train with 3× the actual account size.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
With family-based 14D search space, Phase Fast already searches all dims.
Phases 2-4 (Full/Reward/Risk) were designed for the old 30D space where
each phase explored different parameter subsets. Now redundant — saves
3× hyperopt compute per training run.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace pure time-sequential expanding-window splits with stratified
splits that adjust fold boundaries to balance Trending/Ranging/Volatile
proportions. Slides boundaries up to 25% of validation window when
deviation exceeds 10pp from global average. Strictly temporal — no
data shuffling, only boundary adjustments.
This directly addresses the R²=1.0 finding: IS→OOS Sharpe gap was
entirely regime-driven because folds had wildly different regime mixes.
Stratification ensures each fold sees similar market conditions.
7 unit tests for regime classification, distribution, deviation, and
fold generation with both uniform and imbalanced data.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add futures-baseline/ to the auto-detection search paths so smoke tests
work without FOXHUNT_TEST_DATA env var. All 11 smoke tests pass on
RTX 3050 Ti (960s, Sharpe 7.76).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase system now controls which family intensities are searchable vs
pinned in the 14D layout. Fast (default) searches ALL 14 dims — phased
search was needed for old 30D space but 14D is within PSO's efficient
range. Full/Reward/Risk phases remain for targeted refinement.
Fix test_phase_fast_bounds to verify all dims searchable in Fast.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Task 14: Normalize OOS Sharpe by regime difficulty (volatile folds weighted
up, trending folds down). Compute R² between volatile% and Sharpe across
folds — logs verdict: regime-driven (R²>0.7), model-driven (R²<0.3),
or mixed.
Task 15: Adaptive regime_replay_decay per fold. More concentrated regime →
lower decay → more aggressive PER bias toward dominant regime. Wired
through ReplayBufferType::sample_regime_biased() into the training loop.
Vaccine batches remain unbiased for gradient diversity.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fix mbp10_data_dir/trades_data_dir TOML→DQNHyperparameters wiring gap.
Fields were deserialized but silently dropped — now applied in
training_profile.rs apply_to(). Production TOML activates OFI (8 features
from MBP-10 order book), smoketest leaves it off for fast iteration.
3 new OFI integration tests: state vector positioning, graceful
degradation (zero-fill without MBP-10), feature name ordering.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add scripts/cuda-kernel-guard.sh — catches CPU/host leaks in .cu files:
printf, cudaMalloc/Free, host API calls, double-precision math (exp/log
instead of expf/logf), assert, file I/O. Filters comments and trailing
/* */ annotations. Zero false positives across 38 production kernels.
Extend gpu-hotpath-hook.sh to route .cu/.cuh files to the new guard.
Configure ruflo hooks in .claude/settings.json: pre-edit context loading,
post-edit pattern learning, post-command metrics, session-end persistence.
Fires for subagents too via Claude Code hook system.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
