OTLP fix: opentelemetry-otlp 0.27 bundled tonic 0.12 while the workspace
uses tonic 0.14, making with_channel() impossible (type mismatch). Upgrading
to otel-otlp 0.31 aligns both on tonic 0.14, enabling explicit Channel
construction that respects http:// scheme (no spurious TLS negotiation).
API migrations (otel 0.27→0.31):
- TracerProvider → SdkTracerProvider
- with_batch_exporter(exporter, runtime) → with_batch_exporter(exporter)
- Resource::new(vec![...]) → Resource::builder().with_service_name().build()
- global::shutdown_tracer_provider() removed (Drop-based shutdown)
- opentelemetry-otlp feature "tonic" → "grpc-tonic"
DCGM fix: remove runtimeClassName: nvidia from DaemonSet — Scaleway Kapsule
GPU pools use nvidia runtime as default containerd handler. The RuntimeClass
CRD is only created by the full GPU Operator, not the device plugin alone.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PPO was only checking for NaN losses every 10th epoch, allowing NaN
to propagate for up to 9 epochs and corrupt model weights before
detection. Now checks every mini-batch in both MLP and LSTM paths.
Delete three dead gradient clipping methods from DQN agent:
- compute_gradients_and_clip (never called, hardcoded max_norm=1.0)
- clip_gradients (returns error, deprecated)
- clip_gradients_map (computes clip factor but never applies it)
Active DQN training uses AdamOptimizer::backward_step_with_monitoring
which correctly delegates to gradient_utils::clip_grad_norm.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move Prometheus training metrics from example-local baseline_common/
to common::metrics::{server,training_metrics} following the existing
grpc_metrics.rs pattern. Fix 29 let_underscore_must_use clippy errors
in push_metrics.rs, 3 shadow lint errors in training binaries, and
demote gradient clipping log from warn to debug.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add metrics server lifecycle (init, port 9094, active_workers) to both
hyperopt_baseline_rl and hyperopt_baseline_supervised. All 18 metrics
are registered and exposed; inner PSO trial loops can be instrumented
incrementally.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Create shared baseline_common/metrics.rs module that registers all 18
dashboard-expected metrics (11 gauges, 5 counters, 2 histograms) and
spawns a lightweight HTTP metrics server on port 9094.
Instrument train_baseline_supervised and train_baseline_rl with:
- Epoch progress, training/validation loss gauges
- Checkpoint save timing, size, and failure counters
- NaN/gradient explosion detection counters
- Data loading latency histograms
- Active workers lifecycle (1 on start, 0 on exit)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DCGM exporter v3.x emits lowercase metric names (dcgm_gpu_utilization)
but dashboards used the old uppercase format (DCGM_FI_DEV_GPU_UTIL).
Fixed across 4 dashboard files:
- foxhunt-cockpit.json (Infrastructure Cockpit)
- foxhunt-training-cockpit.json (Training Cockpit)
- foxhunt-gpu-training.json (GPU Training)
- gpu-overview.json (GPU Overview)
Replaced DCGM_FI_DEV_FB_TOTAL with (dcgm_fb_used + dcgm_fb_free)
since DCGM v3 doesn't emit a total metric.
Redeployed all dashboard ConfigMaps and restarted Grafana.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three fixes for GPU metrics not appearing in dashboards:
1. Added runtimeClassName: nvidia so DCGM can access NVML/GPU devices
2. Custom counters ConfigMap without DCP/profiling metrics (avoids
SYS_ADMIN requirement that caused fatal crash)
3. Added gitlab.com/prometheus_* annotations so GitLab's Prometheus
discovers and scrapes the DCGM pods
Increased memory limit 128Mi → 1Gi (was OOMKilled).
Metrics now flowing: gpu_utilization, fb_used/free, power_usage, temps.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The L4 pool was unused: all training routes to ci-training (L40S) and
all compilation routes to ci-compile-cpu (POP2). Disabled in terragrunt
and applied to destroy the pool. Removed all ci-rl references from K8s
manifests and CI comments.
Final pool layout:
- ci-compile-cpu (POP2-32C-128G) — Rust compilation
- ci-training (L40S-1-48G) — all GPU training + hyperopt
- services (DEV1-L) — production services
- gitlab (GP1-XS) — GitLab CE
- gpu-dev (GP1-L) — DevPod development
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Applied terragrunt to recreate the L4 GPU pool with the correct name
`ci-rl` (was `ci-compile` due to immutable Scaleway pool names).
Updated all K8s manifests and comments to match. Removed the 3 stale
`moved` blocks from main.tf since the state renames are now applied.
Pool naming is now consistent across Terraform, Scaleway, and K8s configs:
- ci-compile-cpu (POP2-32C-128G) — CPU compilation
- ci-rl (L4-1-24G) — RL training / CUDA compile
- ci-training (L40S-1-48G) — supervised training + hyperopt
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RL runner only handles GPU training jobs, so global nvidia runtime
is safe and required (KUBERNETES_RUNTIME_CLASS_NAME override wasn't
being applied). Main runner keeps overwrite-allowed pattern.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pool mapping:
- ci-compile-cpu (POP2-32C-128G): Rust compile, web dashboard, manifest
- ci-compile (L4): CUDA compile with stubs only
- ci-training (L40S): ALL training (RL + supervised)
Main runner default → ci-compile-cpu
RL runner default → ci-training
.train-rl-base → explicit ci-training node selector
Replaced all stale ci-rl references with correct pool names.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tempo was OOMKilled every ~30 min at 512Mi, causing web-gateway OTLP
export errors. Increased to 1Gi.
RL runner had cpu_request_overwrite_max_allowed=4000m but hyperopt RL
jobs need 6000m for parallel PSO trials. Increased to 8000m.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove runtime_class_name="nvidia" from runner configs (applied to ALL
pods, broke compile on CPU-only ci-compile-cpu nodes). Instead use
runtime_class_name_overwrite_allowed + KUBERNETES_RUNTIME_CLASS_NAME
per-job on GPU templates (.train-rl-base, .train-validate-base).
Also routes build-web-dashboard and write-manifest to ci-compile-cpu.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GitLab Runner expects KUBERNETES_NODE_SELECTOR_<NAME>: "key=value" format,
not KUBERNETES_NODE_SELECTOR_<key>: "value". Fix both compile (ci-compile-cpu)
and supervised training (ci-training) node selector overrides.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
.rust-base-cpu was missing KUBERNETES_NODE_SELECTOR override, so compile
jobs defaulted to the runner's ci-rl pool which has scale-to-zero and
timed out. ci-compile-cpu has a persistent POP2-32C-128G node.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
cert-manager stores certs as tls.crt/tls.key but MinIO expects
public.crt/private.key — add volume items projection.
mc client doesn't use MC_CA_CERTIFICATE env; instead copy CA cert to
~/.mc/certs/CAs/ before running mc alias set.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Switch all MinIO communication from HTTP to HTTPS across the entire stack:
- MinIO deployment: mount TLS secret, tcpSocket probes, HTTPS init-buckets
- 11 service YAMLs: HTTPS rclone endpoint + CA cert volume mount
- Training job template + train.sh: HTTPS for fetch-binaries and uploader
- CI pipeline (.gitlab-ci.yml): all 7 rclone exports use HTTPS + CA cert
- GitLab runner values: minio-ca-cert ConfigMap volume for CI pods
- Rust: CA cert loading via MINIO_CA_CERT_PATH in training_uploader,
storage backend, data_acquisition uploader, and k8s_dispatcher
- Cert generation script (infra/scripts/generate-minio-tls.sh)
Fixes training uploader S3 upload failures caused by with_allow_http(false)
connecting to an HTTP endpoint.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The job template referenced a non-existent 'gpu-training' pool. The
actual L40S GPU pool in Kapsule is named 'ci-training' (scale-from-zero,
0→1 autoscaling).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GitLab's bundled Prometheus only scrapes pods with gitlab.com/prometheus_scrape
annotations, not the standard prometheus.io/* ones. Without this, the
Infrastructure Cockpit shows no data for node/pod/cluster metrics.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The existing `global.appConfig.gitlab_kas.enabled: false` only affected
the KAS subchart config but was ignored by the chart template that
generates gitlab.yml. Adding `global.kas.enabled: false` properly sets
`gitlab_kas.enabled: false` in gitlab.yml and removes the orphaned
gitlab-kas Service, stopping ~5 GRPC::Unavailable errors per git push.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move all 3 cockpit dashboards (Cockpit, Trading Cockpit, Training
Cockpit) into a single "Cockpits" folder. GPU & Training Jobs moved
to Infrastructure. 4 folders total: Cockpits, Operations,
Infrastructure, CI-CD.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Strip "Foxhunt - " prefix from all dashboard titles since the folder
structure already provides context. Use clean folder names without
slashes (Trading, Training, Operations, Infrastructure, CI-CD) and
per-folder provisioning providers to prevent duplicate folder creation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Dashboard Loki queries used naive `|= "ERROR"` string matching which
created a self-referential feedback loop: Loki and Grafana log query
execution text containing "ERROR" at INFO level, inflating the error
count by ~19/min with phantom matches.
Add `container!~"loki|grafana|grafana-sc-dashboard|prometheus-server"`
filter to all 11 affected queries across 4 dashboards (cockpit,
overview, logs, services).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replaces single "Foxhunt" folder with Trading, Training, Operations,
Infrastructure, and CI/CD groupings. Updates ConfigMap definitions in
import.sh and Grafana Helm values to use foldersFromFilesStructure.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Training jobs are short-lived K8s Jobs that can't be reliably scraped
by Prometheus. This adds a TrainingMetricsPusher that pushes epoch-level
metrics (loss, accuracy, throughput, gradient health, checkpoints) to
the Pushgateway so they persist for Prometheus to scrape.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The deploy job's Grafana dashboard ConfigMap creation used `declare -A`
associative arrays which require bash 4+. The CI runner's shell is
/bin/sh (dash), causing "cannot convert indexed to associative array"
errors and failing the deploy stage.
Rewrites the logic as a POSIX-compatible shell function that takes the
ConfigMap name, folder, and dashboard files as positional arguments.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add a reusable GrpcMetricsLayer that can be applied to any tonic server
via Server::builder().layer() to automatically emit three Prometheus
metric families for every gRPC handler: started_total, handled_total
(with status code), and handling_seconds histogram.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a dedicated Prometheus metrics endpoint on port 9098 (configurable
via METRICS_PORT env var) with counters for HTTP requests, histograms for
request duration, gauges for uptime and active WebSocket connections, and
counters for WebSocket messages. Metrics are registered using once_cell
Lazy statics with abort-on-failure to satisfy the crate's deny(unwrap)
lint policy.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add metrics module with uptime, active feeds, records received,
errors, and feed latency metrics. Spawn HTTP /metrics endpoint
using axum, following the same inline pattern as ml-training-service.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace 28 placeholder PromQL expressions with real metric names:
- Active Jobs: kube_job_status_active from kube-state-metrics
- Training Progress: foxhunt_training_epoch_loss, current_epoch, etc.
- GPU Resources: DCGM_FI_DEV_GPU_UTIL/FB_USED/GPU_TEMP/POWER_USAGE
- Model Quality: foxhunt_training_eval_accuracy/f1/precision/recall
- Checkpoints: foxhunt_training_checkpoint_saves/failures/duration/size
- Data Pipeline: foxhunt_training_batches_processed, data_load_seconds
- Errors: foxhunt_training_nan_detected/gradient_explosion/feature_errors
- Job History: kube_job_complete_time/start_time for duration
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace placeholder metric names (traces_spanmetrics_calls_total,
traces_spanmetrics_latency_*) with real Tempo metrics
(tempo_distributor_spans_received_total, tempo_spanmetrics_latency_*).
Switch Recent Traces panel from nativeSearch to TraceQL query.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace all placeholder PromQL expressions with real metric names
that match actual Prometheus metrics emitted by the trading system.
Add $model, $symbol, and $service template variables for filtering.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace traces_spanmetrics_calls_total with tempo_distributor_spans_received_total
in all 3 Trace Health panels (Spans/sec, Trace Error Rate, Spans/sec by Service).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
13-task plan across 4 layers: fix dashboard PromQL, add metrics
servers to data-acquisition and web-gateway, create gRPC metrics
Tower layer, deploy Pushgateway for training jobs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Four-layer plan: fix dashboard queries to use real metric names,
add metrics servers to data-acquisition and web-gateway, add shared
gRPC metrics tower layer, deploy Pushgateway for training job metrics.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
13-task plan across 4 layers: fix dashboard PromQL, add metrics
servers to data-acquisition and web-gateway, create gRPC metrics
Tower layer, deploy Pushgateway for training jobs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Four-layer plan: fix dashboard queries to use real metric names,
add metrics servers to data-acquisition and web-gateway, add shared
gRPC metrics tower layer, deploy Pushgateway for training job metrics.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Grafana sidecar supports grafana_folder annotations to place
dashboards in specific folders. Without this, all provisioned
dashboards land in the General folder. Add Foxhunt/Infrastructure/CI-CD
folder mapping to both CI pipeline and import.sh.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two bugs found during validation:
1. rclone defaults to HTTPS when endpoint has no scheme — MinIO serves
plain HTTP. All RCLONE_S3_ENDPOINT values now include http:// prefix.
2. rclone copyto returns exit 0 for non-existent S3 keys (empty bucket).
initContainers crashed on chmod of missing file instead of falling
through to PVC cache. Added `&& [ -f /binaries/$SERVICE ]` guard.
Also: deploy stage uses `kubectl apply -k` (not -f) for minio/ dir
to properly handle kustomization.yaml.
Deleted stale s3-credentials secret from cluster.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
