- Remove duplicate checkpoint existence checks in RealTFTModel and
RealMamba2Model (each had two identical `!checkpoint_path.exists()`
guards — reduced to one)
- Move unused ModelPerformanceMetrics struct into test module (only
consumer; was generating dead_code warning)
- Restrict visibility of internal types to pub(crate): RuntimeModelInfo,
FeatureNormStats, FeaturePreprocessor, EnsembleConfig,
ModelPerformanceMetrics — none have external consumers
- Add #[allow(dead_code)] with documentation on RuntimeModelInfo fields
(model_id, fallback_priority) that are stored for Debug output and
future fallback ordering but not yet read in hot paths
- Remove emoji from PPO checkpoint log message for consistency
- No proto contract changes; all gRPC method signatures preserved
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove entire trading_engine/src/compliance/ directory (9 files, 16,068 LOC)
and 18 associated test files (18,069 LOC). Comprehensive audit confirmed
zero external callers for all types (ISO 27001, SOX, MiFID II, best
execution, automated reporting). Remove unused cron dependency.
Independent compliance modules in risk/ and risk-data/ are preserved.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The storage crate had its own RetryConfig struct (max_attempts,
initial_delay, max_delay, backoff_multiplier) duplicating
common::resilience::retry::RetryConfig.
Added backoff_multiplier field to common's RetryConfig and updated
storage to re-export and use common's version with its field names
(max_retries, base_delay). Updated all storage tests accordingly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three duplicate ErrorSeverity enums (data/error.rs, data/validation.rs,
database/error.rs) with variants Low/Medium/High/Critical now use the
canonical definition in common::error::ErrorSeverity.
Extended common's ErrorSeverity with Low, Medium, High variants (alongside
existing Debug, Info, Warn, Error, Critical) and added PartialOrd/Ord derives
so both severity models coexist in a single type.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The RetryStrategy enum in trading_engine/src/types/error.rs was an exact
duplicate of common::error::RetryStrategy with zero callers in the crate.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create EnsembleModelAdapter in ml::ensemble wrapping model IDs
- Add build_production_strategy() factory: 10-model ensemble with
ProductionFeatureExtractorAdapter + graceful degradation (zero confidence
when no checkpoints loaded)
- Wire backtesting_service to use the factory function
- Harden metrics HTTP server: 5s read timeout, 8KB request limit,
correct Content-Type charset=utf-8
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Delete push_metrics.rs (Pushgateway client, zero consumers)
- Delete 5 legacy test/bench files for MLFeatureExtractor + SimpleDQNAdapter
- Remove MLFeatureExtractor from trading_agent_service, use direct bar scoring
- Remove with_feature_extractor() method and Arc<MLFeatureExtractor> parameter
- Remove bench target from common/Cargo.toml
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Quality audit found 2 dead code paths in the GPU optimization commit:
1. Data caching: preload_data() was defined on all 10 hyperopt adapters
but never called. Now wired in both hyperopt binaries (RL + supervised)
before the trial loop. Each model preloads training data once into
Arc<Vec<...>>, eliminating per-trial disk I/O.
2. PPO mixed precision: config.mixed_precision was stored but never used
in forward passes. Added forward_mixed() to PolicyNetwork and
ValueNetwork (same BF16/FP16 pattern as DQN's NetworkLayers). Stored
on network structs and auto-applied via forward(). Wired in
PPO::with_device() for MLP networks.
Also fixes missing mixed_precision field in 2 test files and
trading_service PPOConfig literal.
5 files changed, +152/-20. 2418 tests pass, workspace compiles clean.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wire BF16/FP16 mixed precision end-to-end for DQN and PPO with auto-detection
from GPU name (Ampere+ → BF16, Volta/Turing → FP16). Add hidden_dim_base to
hyperopt and wire through training/eval binaries. Reduce GPU sync points: make
DQN NaN checks periodic (every 100 steps), replace PPO GAE GPU round-trip with
pure CPU implementation. Cache training data across hyperopt trials for all 10
models via Arc. Batch DQN experience storage (128x fewer lock acquisitions).
Correct VRAM constants and batch bounds for all 9 supervised model adapters.
28 files changed, +1207/-208 lines. 2418 tests pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Training Prometheus metrics + production cleanup design:
- Prometheus metrics export from all 4 training binaries (18 metrics)
- Metrics server in common::metrics (TcpListener on port 9094)
- OpenTelemetry 0.27→0.31 upgrade
- DCGM exporter fix for GPU monitoring
- PPO NaN detection + dead DQN clipping code removed
- All clippy errors resolved in common test targets
- ML inference production cleanup design doc
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
OTLP fix: opentelemetry-otlp 0.27 bundled tonic 0.12 while the workspace
uses tonic 0.14, making with_channel() impossible (type mismatch). Upgrading
to otel-otlp 0.31 aligns both on tonic 0.14, enabling explicit Channel
construction that respects http:// scheme (no spurious TLS negotiation).
API migrations (otel 0.27→0.31):
- TracerProvider → SdkTracerProvider
- with_batch_exporter(exporter, runtime) → with_batch_exporter(exporter)
- Resource::new(vec![...]) → Resource::builder().with_service_name().build()
- global::shutdown_tracer_provider() removed (Drop-based shutdown)
- opentelemetry-otlp feature "tonic" → "grpc-tonic"
DCGM fix: remove runtimeClassName: nvidia from DaemonSet — Scaleway Kapsule
GPU pools use nvidia runtime as default containerd handler. The RuntimeClass
CRD is only created by the full GPU Operator, not the device plugin alone.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PPO was only checking for NaN losses every 10th epoch, allowing NaN
to propagate for up to 9 epochs and corrupt model weights before
detection. Now checks every mini-batch in both MLP and LSTM paths.
Delete three dead gradient clipping methods from DQN agent:
- compute_gradients_and_clip (never called, hardcoded max_norm=1.0)
- clip_gradients (returns error, deprecated)
- clip_gradients_map (computes clip factor but never applies it)
Active DQN training uses AdamOptimizer::backward_step_with_monitoring
which correctly delegates to gradient_utils::clip_grad_norm.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move Prometheus training metrics from example-local baseline_common/
to common::metrics::{server,training_metrics} following the existing
grpc_metrics.rs pattern. Fix 29 let_underscore_must_use clippy errors
in push_metrics.rs, 3 shadow lint errors in training binaries, and
demote gradient clipping log from warn to debug.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add metrics server lifecycle (init, port 9094, active_workers) to both
hyperopt_baseline_rl and hyperopt_baseline_supervised. All 18 metrics
are registered and exposed; inner PSO trial loops can be instrumented
incrementally.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Create shared baseline_common/metrics.rs module that registers all 18
dashboard-expected metrics (11 gauges, 5 counters, 2 histograms) and
spawns a lightweight HTTP metrics server on port 9094.
Instrument train_baseline_supervised and train_baseline_rl with:
- Epoch progress, training/validation loss gauges
- Checkpoint save timing, size, and failure counters
- NaN/gradient explosion detection counters
- Data loading latency histograms
- Active workers lifecycle (1 on start, 0 on exit)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DCGM exporter v3.x emits lowercase metric names (dcgm_gpu_utilization)
but dashboards used the old uppercase format (DCGM_FI_DEV_GPU_UTIL).
Fixed across 4 dashboard files:
- foxhunt-cockpit.json (Infrastructure Cockpit)
- foxhunt-training-cockpit.json (Training Cockpit)
- foxhunt-gpu-training.json (GPU Training)
- gpu-overview.json (GPU Overview)
Replaced DCGM_FI_DEV_FB_TOTAL with (dcgm_fb_used + dcgm_fb_free)
since DCGM v3 doesn't emit a total metric.
Redeployed all dashboard ConfigMaps and restarted Grafana.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three fixes for GPU metrics not appearing in dashboards:
1. Added runtimeClassName: nvidia so DCGM can access NVML/GPU devices
2. Custom counters ConfigMap without DCP/profiling metrics (avoids
SYS_ADMIN requirement that caused fatal crash)
3. Added gitlab.com/prometheus_* annotations so GitLab's Prometheus
discovers and scrapes the DCGM pods
Increased memory limit 128Mi → 1Gi (was OOMKilled).
Metrics now flowing: gpu_utilization, fb_used/free, power_usage, temps.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The L4 pool was unused: all training routes to ci-training (L40S) and
all compilation routes to ci-compile-cpu (POP2). Disabled in terragrunt
and applied to destroy the pool. Removed all ci-rl references from K8s
manifests and CI comments.
Final pool layout:
- ci-compile-cpu (POP2-32C-128G) — Rust compilation
- ci-training (L40S-1-48G) — all GPU training + hyperopt
- services (DEV1-L) — production services
- gitlab (GP1-XS) — GitLab CE
- gpu-dev (GP1-L) — DevPod development
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Applied terragrunt to recreate the L4 GPU pool with the correct name
`ci-rl` (was `ci-compile` due to immutable Scaleway pool names).
Updated all K8s manifests and comments to match. Removed the 3 stale
`moved` blocks from main.tf since the state renames are now applied.
Pool naming is now consistent across Terraform, Scaleway, and K8s configs:
- ci-compile-cpu (POP2-32C-128G) — CPU compilation
- ci-rl (L4-1-24G) — RL training / CUDA compile
- ci-training (L40S-1-48G) — supervised training + hyperopt
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RL runner only handles GPU training jobs, so global nvidia runtime
is safe and required (KUBERNETES_RUNTIME_CLASS_NAME override wasn't
being applied). Main runner keeps overwrite-allowed pattern.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pool mapping:
- ci-compile-cpu (POP2-32C-128G): Rust compile, web dashboard, manifest
- ci-compile (L4): CUDA compile with stubs only
- ci-training (L40S): ALL training (RL + supervised)
Main runner default → ci-compile-cpu
RL runner default → ci-training
.train-rl-base → explicit ci-training node selector
Replaced all stale ci-rl references with correct pool names.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tempo was OOMKilled every ~30 min at 512Mi, causing web-gateway OTLP
export errors. Increased to 1Gi.
RL runner had cpu_request_overwrite_max_allowed=4000m but hyperopt RL
jobs need 6000m for parallel PSO trials. Increased to 8000m.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove runtime_class_name="nvidia" from runner configs (applied to ALL
pods, broke compile on CPU-only ci-compile-cpu nodes). Instead use
runtime_class_name_overwrite_allowed + KUBERNETES_RUNTIME_CLASS_NAME
per-job on GPU templates (.train-rl-base, .train-validate-base).
Also routes build-web-dashboard and write-manifest to ci-compile-cpu.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GitLab Runner expects KUBERNETES_NODE_SELECTOR_<NAME>: "key=value" format,
not KUBERNETES_NODE_SELECTOR_<key>: "value". Fix both compile (ci-compile-cpu)
and supervised training (ci-training) node selector overrides.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
.rust-base-cpu was missing KUBERNETES_NODE_SELECTOR override, so compile
jobs defaulted to the runner's ci-rl pool which has scale-to-zero and
timed out. ci-compile-cpu has a persistent POP2-32C-128G node.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
cert-manager stores certs as tls.crt/tls.key but MinIO expects
public.crt/private.key — add volume items projection.
mc client doesn't use MC_CA_CERTIFICATE env; instead copy CA cert to
~/.mc/certs/CAs/ before running mc alias set.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Switch all MinIO communication from HTTP to HTTPS across the entire stack:
- MinIO deployment: mount TLS secret, tcpSocket probes, HTTPS init-buckets
- 11 service YAMLs: HTTPS rclone endpoint + CA cert volume mount
- Training job template + train.sh: HTTPS for fetch-binaries and uploader
- CI pipeline (.gitlab-ci.yml): all 7 rclone exports use HTTPS + CA cert
- GitLab runner values: minio-ca-cert ConfigMap volume for CI pods
- Rust: CA cert loading via MINIO_CA_CERT_PATH in training_uploader,
storage backend, data_acquisition uploader, and k8s_dispatcher
- Cert generation script (infra/scripts/generate-minio-tls.sh)
Fixes training uploader S3 upload failures caused by with_allow_http(false)
connecting to an HTTP endpoint.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The job template referenced a non-existent 'gpu-training' pool. The
actual L40S GPU pool in Kapsule is named 'ci-training' (scale-from-zero,
0→1 autoscaling).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GitLab's bundled Prometheus only scrapes pods with gitlab.com/prometheus_scrape
annotations, not the standard prometheus.io/* ones. Without this, the
Infrastructure Cockpit shows no data for node/pod/cluster metrics.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The existing `global.appConfig.gitlab_kas.enabled: false` only affected
the KAS subchart config but was ignored by the chart template that
generates gitlab.yml. Adding `global.kas.enabled: false` properly sets
`gitlab_kas.enabled: false` in gitlab.yml and removes the orphaned
gitlab-kas Service, stopping ~5 GRPC::Unavailable errors per git push.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move all 3 cockpit dashboards (Cockpit, Trading Cockpit, Training
Cockpit) into a single "Cockpits" folder. GPU & Training Jobs moved
to Infrastructure. 4 folders total: Cockpits, Operations,
Infrastructure, CI-CD.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Strip "Foxhunt - " prefix from all dashboard titles since the folder
structure already provides context. Use clean folder names without
slashes (Trading, Training, Operations, Infrastructure, CI-CD) and
per-folder provisioning providers to prevent duplicate folder creation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Dashboard Loki queries used naive `|= "ERROR"` string matching which
created a self-referential feedback loop: Loki and Grafana log query
execution text containing "ERROR" at INFO level, inflating the error
count by ~19/min with phantom matches.
Add `container!~"loki|grafana|grafana-sc-dashboard|prometheus-server"`
filter to all 11 affected queries across 4 dashboards (cockpit,
overview, logs, services).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replaces single "Foxhunt" folder with Trading, Training, Operations,
Infrastructure, and CI/CD groupings. Updates ConfigMap definitions in
import.sh and Grafana Helm values to use foldersFromFilesStructure.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Training jobs are short-lived K8s Jobs that can't be reliably scraped
by Prometheus. This adds a TrainingMetricsPusher that pushes epoch-level
metrics (loss, accuracy, throughput, gradient health, checkpoints) to
the Pushgateway so they persist for Prometheus to scrape.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
