Fixes clippy::rc_buffer lint in 8 hyperopt adapter files.
Arc<[T]> avoids double indirection vs Arc<Vec<T>>.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create EnsembleModelAdapter in ml::ensemble wrapping model IDs
- Add build_production_strategy() factory: 10-model ensemble with
ProductionFeatureExtractorAdapter + graceful degradation (zero confidence
when no checkpoints loaded)
- Wire backtesting_service to use the factory function
- Harden metrics HTTP server: 5s read timeout, 8KB request limit,
correct Content-Type charset=utf-8
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Delete push_metrics.rs (Pushgateway client, zero consumers)
- Delete 5 legacy test/bench files for MLFeatureExtractor + SimpleDQNAdapter
- Remove MLFeatureExtractor from trading_agent_service, use direct bar scoring
- Remove with_feature_extractor() method and Arc<MLFeatureExtractor> parameter
- Remove bench target from common/Cargo.toml
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Quality audit found 2 dead code paths in the GPU optimization commit:
1. Data caching: preload_data() was defined on all 10 hyperopt adapters
but never called. Now wired in both hyperopt binaries (RL + supervised)
before the trial loop. Each model preloads training data once into
Arc<Vec<...>>, eliminating per-trial disk I/O.
2. PPO mixed precision: config.mixed_precision was stored but never used
in forward passes. Added forward_mixed() to PolicyNetwork and
ValueNetwork (same BF16/FP16 pattern as DQN's NetworkLayers). Stored
on network structs and auto-applied via forward(). Wired in
PPO::with_device() for MLP networks.
Also fixes missing mixed_precision field in 2 test files and
trading_service PPOConfig literal.
5 files changed, +152/-20. 2418 tests pass, workspace compiles clean.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wire BF16/FP16 mixed precision end-to-end for DQN and PPO with auto-detection
from GPU name (Ampere+ → BF16, Volta/Turing → FP16). Add hidden_dim_base to
hyperopt and wire through training/eval binaries. Reduce GPU sync points: make
DQN NaN checks periodic (every 100 steps), replace PPO GAE GPU round-trip with
pure CPU implementation. Cache training data across hyperopt trials for all 10
models via Arc. Batch DQN experience storage (128x fewer lock acquisitions).
Correct VRAM constants and batch bounds for all 9 supervised model adapters.
28 files changed, +1207/-208 lines. 2418 tests pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PPO was only checking for NaN losses every 10th epoch, allowing NaN
to propagate for up to 9 epochs and corrupt model weights before
detection. Now checks every mini-batch in both MLP and LSTM paths.
Delete three dead gradient clipping methods from DQN agent:
- compute_gradients_and_clip (never called, hardcoded max_norm=1.0)
- clip_gradients (returns error, deprecated)
- clip_gradients_map (computes clip factor but never applies it)
Active DQN training uses AdamOptimizer::backward_step_with_monitoring
which correctly delegates to gradient_utils::clip_grad_norm.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move Prometheus training metrics from example-local baseline_common/
to common::metrics::{server,training_metrics} following the existing
grpc_metrics.rs pattern. Fix 29 let_underscore_must_use clippy errors
in push_metrics.rs, 3 shadow lint errors in training binaries, and
demote gradient clipping log from warn to debug.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add metrics server lifecycle (init, port 9094, active_workers) to both
hyperopt_baseline_rl and hyperopt_baseline_supervised. All 18 metrics
are registered and exposed; inner PSO trial loops can be instrumented
incrementally.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Create shared baseline_common/metrics.rs module that registers all 18
dashboard-expected metrics (11 gauges, 5 counters, 2 histograms) and
spawns a lightweight HTTP metrics server on port 9094.
Instrument train_baseline_supervised and train_baseline_rl with:
- Epoch progress, training/validation loss gauges
- Checkpoint save timing, size, and failure counters
- NaN/gradient explosion detection counters
- Data loading latency histograms
- Active workers lifecycle (1 on start, 0 on exit)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Training jobs are short-lived K8s Jobs that can't be reliably scraped
by Prometheus. This adds a TrainingMetricsPusher that pushes epoch-level
metrics (loss, accuracy, throughput, gradient health, checkpoints) to
the Pushgateway so they persist for Prometheus to scrape.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The #[instrument] import was incorrectly inserted inside a `use super {}`
block by the previous commit, causing syntax errors.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Web-gateway routing:
- Point TRADING_SERVICE_URL at api-gateway (proto mismatch fix)
Web-gateway uses foxhunt.tli.TradingService proto but was connecting
directly to trading-service which implements trading.TradingService.
api-gateway already proxies Subscribe* → Stream* correctly.
GitLab KAS:
- Disable gitlab_kas in appConfig to stop sidekiq NotifyGitPushWorker
errors (KAS pod was already disabled but Rails still tried to connect)
Trading service monitoring (3 stubs → real):
- AcknowledgeAlert: real alert lookup + state mutation in shared store
- GetActiveAlerts: returns actual active alerts from in-memory store
- StreamAlerts: now persists generated alerts (capped at 1000 entries)
Trading service ML streams (2 stubs → real):
- StreamModelMetrics: emits real inference_count, error_count, latency
per model every N seconds from the RuntimeModelInfo registry
- StreamSignalStrength: emits per-symbol signal aggregation from model
ensemble weights and latency confidence
Backtesting service:
- stop_backtest: real CancellationToken cancellation (was no-op)
Tokens stored per-backtest, execute_backtest wraps strategy call
in tokio::select! for immediate cancellation
Deleted 7 empty placeholder files:
- 4 Wave D regime stubs (dynamic_stops, ensemble, performance_tracker,
position_sizer) — comment-only files, never wired
- 2 Wave 3 feature stubs (microstructure, statistical)
- 1 PPO stub (unified_ppo.rs — empty struct definitions)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove artificial swarm_size cap from plan_hyperopt() that limited
concurrent trials to n_particles (default 20). Now concurrency is
purely VRAM-driven with a hardware cap of 128 threads.
optimize_parallel() auto-scales n_particles to match GPU budget:
- L4 24GB: ~65 concurrent DQN trials (was 20)
- H100 80GB: 128 concurrent DQN trials (was 20)
- CPU/small GPU: falls back to configured n_particles
max_trials scales proportionally to ensure 3+ PSO iterations
for convergence with larger swarms.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Auto-detect now considers both CPU count and GPU VRAM when choosing
concurrent trial count. Prevents OOM on smaller GPUs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
optimize_parallel() now uses HardwareBudget::plan_hyperopt() to compute
max concurrent trials based on model VRAM footprint. Includes ramp-up
(start at half concurrency), VRAM watchdog (check free memory before
spawning), and graceful degradation to sequential.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Computes optimal (concurrency, batch_size_bounds) based on model memory
footprint vs detected GPU VRAM. Scales from CPU-only (sequential) through
H100 80GB (full swarm parallel). 20% safety margin for CUDA allocator.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PSO now controls batch_size within [64, 4096] range. HardwareBudget
adjusts upper bound based on detected VRAM. The .min(160.0) clamp was
silently overriding PSO's choices on any GPU larger than 3050 Ti.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Single model_memory_mb field now serves as the sole source of truth for
base model memory. Removes the redundant base_model_memory_mb field,
legacy dual-path branches in calculate_optimal_batch_size() and
max_safe_batch_size(), and the associated legacy test.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
AutoBatchSizer was computing an "optimal" batch_size (128 on L4) and
overriding the PSO-chosen value. On an L4 24GB this meant 3.5% VRAM
utilization per trial. Now the trainer only clamps if the batch_size
would actually OOM, letting PSO explore the full [64, 4096] range.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The GPU experience collector only supported DQNAgentType::Standard, but
enable_regime_qnetwork defaults to true, creating RegimeConditional
agents. This silently fell back to CPU with a debug! message invisible
at INFO log level.
- Add primary_head() accessor to RegimeConditionalDQN (returns trending head)
- Extract weights from primary head for both collector init and weight sync
- Upgrade debug! to warn! for missing GPU collection prerequisites
- Add warn! to PPO for silent GPU skip when raw market data not uploaded
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
These files were part of the Phase 2 cuda_pipeline but were untracked
and not included in previous commits. Required for compilation with
--features ml/cuda.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Phase 3 of the CUDA pipeline: both trainers now take a GPU-first path
for experience collection (128×500 = 64K experiences per kernel launch,
zero CPU-GPU roundtrips per timestep) with automatic CPU fallback.
- DQN: upload features alongside targets, GPU collection branch before
CPU loop, gpu_batch_to_experiences() conversion into replay buffer
- PPO: set_raw_market_data() for CudaSlice upload, GPU collection
branch bypasses collect_rollouts + prepare_training_batch entirely,
gpu_batch_to_trajectory_batch() conversion with in-kernel GAE
- Configurable GPU batch sizes (gpu_n_episodes, gpu_timesteps_per_episode)
and trading params (initial_capital, avg_spread) via hyperparameters
- SAFETY training diagnostics downgraded from warn! to debug!
- 4 new batch index-math validation tests in cuda_pipeline
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move 10 shared device functions (gpu_random, leaky_relu, matvec_leaky_relu,
action_to_exposure, action_to_tx_cost, barrier_init/check/reset,
diversity_entropy, curiosity_inference) and shared constants from
dqn_experience_kernel.cu into a new common_device_functions.cuh header.
The DQN kernel now expects the common header to be prepended via NVRTC
source concatenation at compile time. DQN-specific functions
(q_forward_dueling, argmax_q) remain in the kernel file. This enables
the upcoming PPO kernel to reuse the same shared functions.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds GpuExperienceCollector field to DQNTrainer, initializes it when
dueling networks and curiosity module are present on CUDA device, and
syncs GPU weight copies after each training epoch.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three fixes from spec review:
1. diversity_entropy returns -0.1 penalty (threshold < 1.0) instead of raw entropy
2. barrier_done included in episode termination (barrier hit ends episode)
3. step_in_episode counter resets properly on mid-loop episode reset
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace all `.to_string().parse::<f32/f64>()` patterns with
`num_traits::ToPrimitive` methods (`.to_f32()`, `.to_f64()`).
Each string roundtrip heap-allocated per conversion — fatal in
DQN hot loop (300K+ bars × epochs). Decimal stays as canonical
financial type; conversions happen at GPU/float boundaries only.
Also fixes blocking_read() in async context (risk_integration.rs).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
P2-C reversal/cash warns fire thousands of times per hyperopt trial
during backtest simulation. Also removes stale Kelly patch file (already
applied).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
These fire every training step when portfolio isn't populated yet —
extremely noisy during hyperopt with parallel trials.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
