Commit Graph

12 Commits

Author SHA1 Message Date
jgrusewski
8a42af0bc6 refactor: LHS is the correct algorithm for small budgets, not a fallback
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-29 19:18:47 +02:00
jgrusewski
3e1f3fd0cc fix(hyperopt): LHS fallback when budget < 2× swarm size
With 10 trials and n_initial=5, PSO got 5 remaining evals for a
20-particle swarm. Only 5 of 20 particles were evaluated before the
budget observer killed the run — 75% of the swarm had no cost value.
PSO can't compute a proper gbest from partial data.

Fix: when remaining_trials < n_particles × 2, skip PSO entirely and
use additional LHS samples instead. LHS gives better space coverage
than an incomplete swarm iteration.

For 10 trials: 5 initial LHS + 5 additional LHS = 10 independent
space-filling samples. Much better than 5 LHS + 5 broken PSO particles.

PSO still runs when budget is sufficient (≥40 remaining for 20 particles).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-29 19:12:47 +02:00
jgrusewski
ad4e22e87d fix(hyperopt): PSO premature convergence — tune inertia/cognitive/social
Argmin defaults: inertia=0.72, cognitive=1.19, social=1.19
Problem: social >> inertia causes particles to collapse toward the
global best immediately. With only 1 LHS trial as the initial best,
the entire swarm clusters around that point and can't explore.
Every PSO trial was worse than the random LHS trial.

Fix: inertia=0.9 (high momentum, maintains exploration),
cognitive=1.5 (strong personal best memory),
social=0.8 (weak global pull, prevents premature convergence).

Applied to both sequential and parallel optimizer paths.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-29 19:05:16 +02:00
jgrusewski
d485dde5a2 fix: eliminate NVRTC, fix evaluator SIGSEGV, 22D search space, trade physics
Major changes:
- Eliminate ALL runtime NVRTC: c51_loss + mse_loss kernels converted to
  precompiled cubins with runtime num_atoms/v_min/v_max/branch params
- Fix evaluator SIGSEGV: rng_states/q_gaps sized for chunked batch (cn)
  not n_windows; NULL pointer guard in action_select kernel
- Add trade_physics.cuh shared header for train/eval consistency
- Add equity circuit breaker (25% DD from peak) + margin-aware position cap
- Consolidate 46D→22D hyperopt search space, enable ensemble by default
- Fix trade counting: use exposure index not factored action
- Fix Calmar overflow: clamp to ±100 in kernel
- Softer CVaR penalty (cap 3.0 not 10.0) for undertrained models
- Fix win_rate display (ratio→percentage)
- Remove dead code (normalize_reward, calculate_completion_penalty)
- Add tracing subscriber to hyperopt test for visible metrics
- Per-chunk sync in evaluator for reliable error reporting

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 00:33:05 +01:00
jgrusewski
4ab8ca0162 fix: parameterize IQN_SHARED_H1 + IQN_HIDDEN in all 9 IQN kernels
Build-time #define was 256 but runtime config can be 128 (hidden_dim_base).
Caused CUDA_ERROR_ILLEGAL_ADDRESS in IQN trunk gradient. Now passed as
kernel params matching the STATE_DIM parameterization pattern.

Also: warn→error for training failures in optimizer, eprintln for debugging.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 01:55:52 +01:00
jgrusewski
9b4395b0df fix: trial budget observer — shared AtomicUsize counter enforces trial limit
increment_trial() was never called — budget enforcement was broken because
TrialBudgetObserver had its own internal Arc<AtomicUsize> disconnected from
the trial_counter used by evaluate_point() and cost(). With num_trials=2,
PSO ran 20 particle evaluations instead of stopping at 2.

Now TrialBudgetObserver::new() takes the caller-owned Arc<AtomicUsize> so
all evaluation paths (LHS via evaluate_point + PSO via cost()) share one
counter. ObjectiveFunction::cost() and ParallelObjectiveFunction::cost()
increment trial_counter directly (fetch_add SeqCst) and check budget inline;
observe_iter() reads the same counter to terminate PSO between iterations.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-26 01:18:19 +01:00
jgrusewski
502fc469fe fix: TPE evaluate_point catches training errors as penalty, not abort
NaN/divergence during training is a valid PSO/TPE outcome — it means
the sampled hyperparameters are unstable. Score failed trials with 1e6
penalty so the optimizer learns to avoid that region, instead of
aborting all remaining trials.

The PSO/argmin paths already handled this (lines 1118, 1261). Only
the shared evaluate_point (TPE path) propagated errors as fatal.

Tested: 6 trials × 20 epochs on RTX 3050 — 2 NaN trials scored as
penalty, optimizer completed all 6 trials in 597s.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-22 14:48:40 +01:00
jgrusewski
a5ea8713b6 feat(dqn): Branching DQN + KernelWeightPack + metrics propagation
Branching DQN (Tavakoli et al., 2018):
- 3 independent advantage heads (exposure=5, order=3, urgency=3) in CUDA kernel
- `use_branching` as hyperopt-tunable parameter (index 11 in 29D space)
- Branch weight pointers packed into KernelWeightPack struct

KernelWeightPack refactor:
- 87→38 CUDA kernel params by packing 48 weight pointers into 384-byte #[repr(C)] struct
- UNPACK_WEIGHT_PTRS macro in CUDA header for clean kernel-side access
- Single .arg(&weight_pack) replaces 48 individual .arg() calls

Hyperopt metrics in JSON output:
- Added `metrics: Option<serde_json::Value>` to TrialResult<P>
- `extract_metrics()` trait method with default None (DQN overrides)
- JSON output now includes per-trial backtest metrics (Sharpe, Sortino,
  Calmar, Omega, drawdown, win_rate, trades) + top-level best_metrics

Prometheus backtest gauges (Rust-side, no CUDA):
- 8 new gauges: foxhunt_hyperopt_best_{sharpe,sortino,calmar,omega,
  max_drawdown_pct,win_rate,total_trades,total_return_pct}

Dynamic episode scaling:
- Removed hardcoded 256 episode cap on small GPUs
- VRAM budget calculation in optimal_n_episodes() handles scaling naturally
- Removed stale .min(256) in PPO trainer

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-11 00:46:53 +01:00
jgrusewski
d06c99b0e1 fix(clippy): achieve zero warnings across entire workspace
- Fix format_push_string: write!() instead of push_str(&format!()) (25 sites)
- Fix str_to_string: .to_owned() instead of .to_string() on &str (6 sites)
- Fix unseparated_literal_suffix: add _ separator (6 sites)
- Fix multiple_inherent_impl: merge split impl blocks in TGGN, TFT, OFI (3)
- Fix else_if_without_else: add exhaustive else clauses (3 sites)
- Fix if_then_some_else_none: use .then().transpose() (1 site)
- Fix unwrap_in_result: replace expect() with match + ? (2 sites)
- Fix wildcard_enum_match_arm: enumerate Storage variants explicitly (2)
- Fix decimal_literal_representation: use hex for power-of-2 constants (5)
- Fix rc_buffer: Arc<Vec<T>> → Arc<[T]> for OFI features
- Fix needless_range_loop: convert to iterator patterns (17 sites)
- Fix used_underscore_binding: remove prefix on used vars (6 sites)
- Fix doc list item indentation (7 sites)
- Allow too_many_arguments on ML training functions (4)
- Allow multiple_unsafe_ops_per_block on CUDA FFI functions (3)
- Allow upper_case_acronyms on SLSTM/MLSTM model names (2)
- Add ML-crate pedantic allows: shadow, similar_names, type_complexity,
  indexing_slicing, partial_pub_fields, non_ascii_literal, same_name_method
  (following existing ml-labeling/ml-universe pattern)

Result: cargo clippy --workspace -- -D warnings passes with zero warnings.
All 2758+ lib tests pass (2 pre-existing backtesting failures unchanged).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-10 13:18:57 +01:00
jgrusewski
fca2495a73 fix(clippy): add doc backticks and const fn across ML crates
- Add backticks to type names in doc comments (doc_markdown)
- Mark eligible functions as const fn (missing_const_for_fn)
No behavior changes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-10 11:51:31 +01:00
jgrusewski
7ef92983f9 fix(clippy): apply cargo clippy --fix across workspace
Mechanical auto-fixes: redundant borrows, clone on Copy, or_insert_with,
single-char push_str, get(0) → first(), needless borrow, let_and_return.
150 files, no behavior changes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-10 11:17:51 +01:00
jgrusewski
d313486dc2 refactor(ml): split monolith into 9 sub-crates + delete dead code
Extract 9 new sub-crates from the ml monolith to enable parallel
compilation across the workspace:

New crates (this commit):
- ml-features (282 tests): feature engineering, 21 modules
- ml-labeling (45 tests): triple barrier, meta-labeling, fractional diff
- ml-ensemble (116 tests): ensemble coordination, voting, confidence
- ml-hyperopt (47 tests): core PSO/TPE optimizer, parameter space
- ml-checkpoint (41 tests): checkpoint persistence, compression, signing
- ml-regime (68 tests): CUSUM, Bayesian changepoint, regime classification
- ml-data-validation (67 tests): FDR correction, CPCV, data quality
- ml-risk (33 tests): neural VaR, Kelly criterion, circuit breakers
- ml-validation (43 tests): statistical validation, walk-forward, DSR

Extended existing crates:
- ml-dqn: added evaluation/ (backtesting engine, metrics, reports)
  and checkpoint implementation
- ml-supervised: added checkpoint implementations
- ml-core: added shared types needed by new sub-crates

Pattern: each module in ml/ becomes a thin facade (pub use subcrate::*)
with bridge modules staying in ml for cross-model adapter code.

Dead code deleted (~7K lines):
- 13 undeclared files in microstructure/ (never compiled)
- 7 undeclared files + tests/ in risk/ (never compiled)
- parquet_io, cache_service, cache_storage, minio_integration (unused)
- extraction_wave_d_impl.rs (bare fn outside impl block)

All 2,746 sub-crate tests + 951 ml tests pass.
Full workspace builds clean.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 15:17:22 +01:00