The batch span processor needs a tokio runtime for gRPC transport and
periodic flush. Async services already have one via #[tokio::main], but
sync training binaries (hyperopt, train, evaluate) don't.
Previous approach (making binaries async with #[tokio::main]) caused
"Cannot start a runtime from within a runtime" panics because the ML
crate's internal code creates its own tokio runtimes for block_on().
New approach: build_otel_tracer() detects runtime context via
Handle::try_current(). If absent, it creates a dedicated 1-worker
multi-thread runtime stored in a process-lifetime OnceLock. The worker
thread actively polls the OTLP batch export task.
Reverts training binaries to sync fn main() so internal runtime creation
(hyperopt adapters, DQN/PPO trainers) continues working as before.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All 6 training binaries (hyperopt_baseline_rl, hyperopt_baseline_supervised,
train_baseline_rl, train_baseline_supervised, evaluate_baseline,
evaluate_supervised) used sync fn main() but the OTLP batch exporter
requires a tokio runtime (tonic/hyper-util gRPC transport). This caused
an immediate panic on CI when OTEL_EXPORTER_OTLP_ENDPOINT was set.
Fix: #[tokio::main(flavor = "current_thread")] on all 6 binaries.
Also fix pre-existing clippy warnings (shadow, let_underscore_must_use,
doc_markdown, cognitive_complexity, integer_division, unsafe_code).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace tracing_subscriber::fmt() with init_observability() which adds
JSON structured logging + optional OTLP export to Tempo. When
OTEL_EXPORTER_OTLP_ENDPOINT env var is unset, falls back to fmt-only.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move Prometheus training metrics from example-local baseline_common/
to common::metrics::{server,training_metrics} following the existing
grpc_metrics.rs pattern. Fix 29 let_underscore_must_use clippy errors
in push_metrics.rs, 3 shadow lint errors in training binaries, and
demote gradient clipping log from warn to debug.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Create shared baseline_common/metrics.rs module that registers all 18
dashboard-expected metrics (11 gauges, 5 counters, 2 histograms) and
spawns a lightweight HTTP metrics server on port 9094.
Instrument train_baseline_supervised and train_baseline_rl with:
- Epoch progress, training/validation loss gauges
- Checkpoint save timing, size, and failure counters
- NaN/gradient explosion detection counters
- Data loading latency histograms
- Active workers lifecycle (1 on start, 0 on exit)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
evaluate_supervised looked for checkpoints at {models_dir}/{model}_fold{N}_best
but training saves to {models_dir}/{model}/{model}_fold{N}_best (model subdirectory).
Also saves NormStats JSON per fold during training so evaluation uses
training-time normalization instead of computing from test data (data leakage).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The training binary creates flat [batch, 51] feature vectors (one per
bar), but TFTConfig::default() had sequence_length=50, making the
adapter expect [batch, 51*50=2550]. Set sequence_length=1 and
prediction_horizon=1 to match the actual point-wise sample format.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
TFT create_model used TFTConfig::default() values for num_known_features(10)
and num_unknown_features(210) totaling 220, but input_dim was 51 from the
feature extractor. Set both explicitly: known=0, unknown=feature_dim.
S3 uploader now uses path-style requests (required for Scaleway S3) and
explicitly passes AWS credentials from env vars instead of relying on the
instance metadata credential provider (unavailable on Kapsule).
Also fix runner tags lost during session (kapsule, rust, docker restored).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
TFT create_model used TFTConfig::default() values for num_known_features(10)
and num_unknown_features(210) totaling 220, but input_dim was 51 from the
feature extractor. Set both explicitly: known=0, unknown=feature_dim.
S3 uploader now uses path-style requests (required for Scaleway S3) and
explicitly passes AWS credentials from env vars instead of relying on the
instance metadata credential provider (unavailable on Kapsule).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Extend ml_training_service to dispatch GPU training jobs as K8s batch/v1
Jobs, collect results via a Rust sidecar uploader, and support model
promotion with operator approval via fxt CLI.
- K8s dispatcher creates Jobs on gpu-training pool with native sidecar
- training_uploader crate: watches DONE/FAILED marker, uploads to S3,
reports completion via ReportJobCompletion gRPC
- PromotionManager compares metrics, queues better models for approval
- 4 new proto RPCs: ReportJobCompletion, ListPendingPromotions,
ApprovePromotion, RejectPromotion
- fxt commands: train start, model list/approve/reject
- Training binaries write DONE/FAILED markers + metrics.json
- Dockerfile, K8s job template, and CI pipeline updated
- StartTraining gracefully falls back to in-process when outside K8s
- 27 new tests (16 service + 11 promotion), 141 total service tests pass
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a parent/child GitLab CI pipeline for ML model training:
- Generator script produces per-model hyperopt/train/evaluate jobs
- Parent pipeline (.gitlab-ci-training.yml) with manual trigger
- NFS-backed ReadWriteMany PVC for shared training outputs
- Hyperopt params wired into training binaries (DQN, PPO, TFT, Mamba2)
- Shared DBN loader eliminates duplicate code across hyperopt adapters
- Supervised hyperopt unified to DBN data (was parquet-only)
Pipeline: hyperopt (4 models) → train (10 models) → evaluate ensemble
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Delete 22 dead/placeholder/broken example files (-3,489 lines code)
- Delete 4 tracked CSV files (-1.1M lines, were accidentally committed)
- Move baseline training data default from data/cache/ to test_data/
- Update 5 unified binary defaults, gitignore, k8s upload comment, docs
- Consolidate all training data under test_data/futures-baseline/
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
