Commit Graph

9 Commits

Author SHA1 Message Date
jgrusewski
64ca8f97ce fix(observability): dedicated OTLP runtime for sync training binaries
The batch span processor needs a tokio runtime for gRPC transport and
periodic flush. Async services already have one via #[tokio::main], but
sync training binaries (hyperopt, train, evaluate) don't.

Previous approach (making binaries async with #[tokio::main]) caused
"Cannot start a runtime from within a runtime" panics because the ML
crate's internal code creates its own tokio runtimes for block_on().

New approach: build_otel_tracer() detects runtime context via
Handle::try_current(). If absent, it creates a dedicated 1-worker
multi-thread runtime stored in a process-lifetime OnceLock. The worker
thread actively polls the OTLP batch export task.

Reverts training binaries to sync fn main() so internal runtime creation
(hyperopt adapters, DQN/PPO trainers) continues working as before.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 18:54:10 +01:00
jgrusewski
fcf87a5f72 fix(ml): add tokio runtime to training binaries for OTLP export
All 6 training binaries (hyperopt_baseline_rl, hyperopt_baseline_supervised,
train_baseline_rl, train_baseline_supervised, evaluate_baseline,
evaluate_supervised) used sync fn main() but the OTLP batch exporter
requires a tokio runtime (tonic/hyper-util gRPC transport). This caused
an immediate panic on CI when OTEL_EXPORTER_OTLP_ENDPOINT was set.

Fix: #[tokio::main(flavor = "current_thread")] on all 6 binaries.
Also fix pre-existing clippy warnings (shadow, let_underscore_must_use,
doc_markdown, cognitive_complexity, integer_division, unsafe_code).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 18:16:37 +01:00
jgrusewski
118ceca4d5 fix(ml): update evaluate_baseline for 45 factored actions
Both DQN and PPO eval paths used old 3-action index matching (0=Buy,
1=Sell, 2=Hold). Now uses FactoredAction.target_exposure() for
exposure-weighted returns and order-type-specific transaction costs.
PPO path had .to_int() which doesn't exist on FactoredAction.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 15:41:24 +01:00
jgrusewski
6c829d59a8 feat(ml): wire Prometheus eval metrics into evaluate_baseline
Emit set_eval_metrics (win_rate, sharpe, profit_factor, return) per fold
for DQN/PPO evaluation. Start metrics server on :9094. Feeds training
cockpit Grafana dashboard eval panels.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 23:03:13 +01:00
jgrusewski
2cc68af7d6 feat(ml): add OTLP tracing to all 6 training/eval binaries
Replace tracing_subscriber::fmt() with init_observability() which adds
JSON structured logging + optional OTLP export to Tempo. When
OTEL_EXPORTER_OTLP_ENDPOINT env var is unset, falls back to fmt-only.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 23:03:13 +01:00
jgrusewski
2b2ff4ffa5 feat(ml): maximize GPU utilization — BF16 mixed precision, dynamic sizing, sync reduction
Wire BF16/FP16 mixed precision end-to-end for DQN and PPO with auto-detection
from GPU name (Ampere+ → BF16, Volta/Turing → FP16). Add hidden_dim_base to
hyperopt and wire through training/eval binaries. Reduce GPU sync points: make
DQN NaN checks periodic (every 100 steps), replace PPO GAE GPU round-trip with
pure CPU implementation. Cache training data across hyperopt trials for all 10
models via Arc. Batch DQN experience storage (128x fewer lock acquisitions).
Correct VRAM constants and batch bounds for all 9 supervised model adapters.

28 files changed, +1207/-208 lines. 2418 tests pass.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 17:54:25 +01:00
jgrusewski
afd85b2f8f chore: clean up examples, update ML binaries and risk tests
- Delete 14 unused example files (-3,543 lines): config, adaptive-strategy,
  data, storage, trading_engine, api_gateway, backtesting, trading_service, chaos
- Update ML training/eval binaries: improved CLI args, completion tracking,
  CUDA test cleanup, hyperopt enhancements
- Fix KAN network and TFT module adjustments
- Update risk test assertions for consistency
- Fix backtesting repositories and promotion manager
- Update .serena project config and Cargo dependencies

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 01:33:18 +01:00
jgrusewski
e72e4db235 refactor: delete 22 dead examples, 4 CSVs, consolidate data to test_data/
- Delete 22 dead/placeholder/broken example files (-3,489 lines code)
- Delete 4 tracked CSV files (-1.1M lines, were accidentally committed)
- Move baseline training data default from data/cache/ to test_data/
- Update 5 unified binary defaults, gitignore, k8s upload comment, docs
- Consolidate all training data under test_data/futures-baseline/

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 01:24:02 +01:00
jgrusewski
9c3d741a08 refactor: restructure repo — crates/, bin/, testing/ layout
Move 17 library crates into crates/, CLI binary into bin/fxt,
consolidate 10 test crates into testing/, split config crate
from deployment config files.

Root directory reduced from 38+ to ~17 directories.
All Cargo.toml paths and build.rs proto refs updated.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 11:56:00 +01:00