Pool mapping:
- ci-compile-cpu (POP2-32C-128G): Rust compile, web dashboard, manifest
- ci-compile (L4): CUDA compile with stubs only
- ci-training (L40S): ALL training (RL + supervised)
Main runner default → ci-compile-cpu
RL runner default → ci-training
.train-rl-base → explicit ci-training node selector
Replaced all stale ci-rl references with correct pool names.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tempo was OOMKilled every ~30 min at 512Mi, causing web-gateway OTLP
export errors. Increased to 1Gi.
RL runner had cpu_request_overwrite_max_allowed=4000m but hyperopt RL
jobs need 6000m for parallel PSO trials. Increased to 8000m.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove runtime_class_name="nvidia" from runner configs (applied to ALL
pods, broke compile on CPU-only ci-compile-cpu nodes). Instead use
runtime_class_name_overwrite_allowed + KUBERNETES_RUNTIME_CLASS_NAME
per-job on GPU templates (.train-rl-base, .train-validate-base).
Also routes build-web-dashboard and write-manifest to ci-compile-cpu.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GitLab Runner expects KUBERNETES_NODE_SELECTOR_<NAME>: "key=value" format,
not KUBERNETES_NODE_SELECTOR_<key>: "value". Fix both compile (ci-compile-cpu)
and supervised training (ci-training) node selector overrides.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
.rust-base-cpu was missing KUBERNETES_NODE_SELECTOR override, so compile
jobs defaulted to the runner's ci-rl pool which has scale-to-zero and
timed out. ci-compile-cpu has a persistent POP2-32C-128G node.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
cert-manager stores certs as tls.crt/tls.key but MinIO expects
public.crt/private.key — add volume items projection.
mc client doesn't use MC_CA_CERTIFICATE env; instead copy CA cert to
~/.mc/certs/CAs/ before running mc alias set.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Switch all MinIO communication from HTTP to HTTPS across the entire stack:
- MinIO deployment: mount TLS secret, tcpSocket probes, HTTPS init-buckets
- 11 service YAMLs: HTTPS rclone endpoint + CA cert volume mount
- Training job template + train.sh: HTTPS for fetch-binaries and uploader
- CI pipeline (.gitlab-ci.yml): all 7 rclone exports use HTTPS + CA cert
- GitLab runner values: minio-ca-cert ConfigMap volume for CI pods
- Rust: CA cert loading via MINIO_CA_CERT_PATH in training_uploader,
storage backend, data_acquisition uploader, and k8s_dispatcher
- Cert generation script (infra/scripts/generate-minio-tls.sh)
Fixes training uploader S3 upload failures caused by with_allow_http(false)
connecting to an HTTP endpoint.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The job template referenced a non-existent 'gpu-training' pool. The
actual L40S GPU pool in Kapsule is named 'ci-training' (scale-from-zero,
0→1 autoscaling).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GitLab's bundled Prometheus only scrapes pods with gitlab.com/prometheus_scrape
annotations, not the standard prometheus.io/* ones. Without this, the
Infrastructure Cockpit shows no data for node/pod/cluster metrics.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The existing `global.appConfig.gitlab_kas.enabled: false` only affected
the KAS subchart config but was ignored by the chart template that
generates gitlab.yml. Adding `global.kas.enabled: false` properly sets
`gitlab_kas.enabled: false` in gitlab.yml and removes the orphaned
gitlab-kas Service, stopping ~5 GRPC::Unavailable errors per git push.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move all 3 cockpit dashboards (Cockpit, Trading Cockpit, Training
Cockpit) into a single "Cockpits" folder. GPU & Training Jobs moved
to Infrastructure. 4 folders total: Cockpits, Operations,
Infrastructure, CI-CD.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Strip "Foxhunt - " prefix from all dashboard titles since the folder
structure already provides context. Use clean folder names without
slashes (Trading, Training, Operations, Infrastructure, CI-CD) and
per-folder provisioning providers to prevent duplicate folder creation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Dashboard Loki queries used naive `|= "ERROR"` string matching which
created a self-referential feedback loop: Loki and Grafana log query
execution text containing "ERROR" at INFO level, inflating the error
count by ~19/min with phantom matches.
Add `container!~"loki|grafana|grafana-sc-dashboard|prometheus-server"`
filter to all 11 affected queries across 4 dashboards (cockpit,
overview, logs, services).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replaces single "Foxhunt" folder with Trading, Training, Operations,
Infrastructure, and CI/CD groupings. Updates ConfigMap definitions in
import.sh and Grafana Helm values to use foldersFromFilesStructure.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Training jobs are short-lived K8s Jobs that can't be reliably scraped
by Prometheus. This adds a TrainingMetricsPusher that pushes epoch-level
metrics (loss, accuracy, throughput, gradient health, checkpoints) to
the Pushgateway so they persist for Prometheus to scrape.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The deploy job's Grafana dashboard ConfigMap creation used `declare -A`
associative arrays which require bash 4+. The CI runner's shell is
/bin/sh (dash), causing "cannot convert indexed to associative array"
errors and failing the deploy stage.
Rewrites the logic as a POSIX-compatible shell function that takes the
ConfigMap name, folder, and dashboard files as positional arguments.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add a reusable GrpcMetricsLayer that can be applied to any tonic server
via Server::builder().layer() to automatically emit three Prometheus
metric families for every gRPC handler: started_total, handled_total
(with status code), and handling_seconds histogram.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a dedicated Prometheus metrics endpoint on port 9098 (configurable
via METRICS_PORT env var) with counters for HTTP requests, histograms for
request duration, gauges for uptime and active WebSocket connections, and
counters for WebSocket messages. Metrics are registered using once_cell
Lazy statics with abort-on-failure to satisfy the crate's deny(unwrap)
lint policy.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add metrics module with uptime, active feeds, records received,
errors, and feed latency metrics. Spawn HTTP /metrics endpoint
using axum, following the same inline pattern as ml-training-service.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace 28 placeholder PromQL expressions with real metric names:
- Active Jobs: kube_job_status_active from kube-state-metrics
- Training Progress: foxhunt_training_epoch_loss, current_epoch, etc.
- GPU Resources: DCGM_FI_DEV_GPU_UTIL/FB_USED/GPU_TEMP/POWER_USAGE
- Model Quality: foxhunt_training_eval_accuracy/f1/precision/recall
- Checkpoints: foxhunt_training_checkpoint_saves/failures/duration/size
- Data Pipeline: foxhunt_training_batches_processed, data_load_seconds
- Errors: foxhunt_training_nan_detected/gradient_explosion/feature_errors
- Job History: kube_job_complete_time/start_time for duration
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace placeholder metric names (traces_spanmetrics_calls_total,
traces_spanmetrics_latency_*) with real Tempo metrics
(tempo_distributor_spans_received_total, tempo_spanmetrics_latency_*).
Switch Recent Traces panel from nativeSearch to TraceQL query.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace all placeholder PromQL expressions with real metric names
that match actual Prometheus metrics emitted by the trading system.
Add $model, $symbol, and $service template variables for filtering.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace traces_spanmetrics_calls_total with tempo_distributor_spans_received_total
in all 3 Trace Health panels (Spans/sec, Trace Error Rate, Spans/sec by Service).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
13-task plan across 4 layers: fix dashboard PromQL, add metrics
servers to data-acquisition and web-gateway, create gRPC metrics
Tower layer, deploy Pushgateway for training jobs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Four-layer plan: fix dashboard queries to use real metric names,
add metrics servers to data-acquisition and web-gateway, add shared
gRPC metrics tower layer, deploy Pushgateway for training job metrics.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
13-task plan across 4 layers: fix dashboard PromQL, add metrics
servers to data-acquisition and web-gateway, create gRPC metrics
Tower layer, deploy Pushgateway for training jobs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Four-layer plan: fix dashboard queries to use real metric names,
add metrics servers to data-acquisition and web-gateway, add shared
gRPC metrics tower layer, deploy Pushgateway for training job metrics.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Grafana sidecar supports grafana_folder annotations to place
dashboards in specific folders. Without this, all provisioned
dashboards land in the General folder. Add Foxhunt/Infrastructure/CI-CD
folder mapping to both CI pipeline and import.sh.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two bugs found during validation:
1. rclone defaults to HTTPS when endpoint has no scheme — MinIO serves
plain HTTP. All RCLONE_S3_ENDPOINT values now include http:// prefix.
2. rclone copyto returns exit 0 for non-existent S3 keys (empty bucket).
initContainers crashed on chmod of missing file instead of falling
through to PVC cache. Added `&& [ -f /binaries/$SERVICE ]` guard.
Also: deploy stage uses `kubectl apply -k` (not -f) for minio/ dir
to properly handle kustomization.yaml.
Deleted stale s3-credentials secret from cluster.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add tracing::instrument(skip_all) to gRPC handlers across all services
for distributed trace spans via OTLP. Pairs with Prometheus scrape
annotations from previous commit.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move all compiled binary uploads/downloads from public Scaleway S3 to
MinIO running inside Kapsule. Keeps proprietary binaries and trained
models off the public internet.
- CI compile/upload jobs: Scaleway → MinIO endpoint + credentials
- train.sh: all 3 rclone blocks → RCLONE_CONFIG_MINIO_*
- Deploy stage: apply MinIO manifests + wait for readiness before rollout
- Add infra/k8s/minio/ (Deployment, Service, Secret, PVC, init-buckets Job)
- Update s3-credentials.secret.example → minio-credentials format
IaC Terragrunt jobs and SCW registry pulls intentionally unchanged.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- foxhunt-trading-cockpit.json: 22 panels covering signal quality,
execution, risk, ML model health, and latency
- foxhunt-training-cockpit.json: 36 panels covering active jobs,
training progress, GPU resources, model quality, checkpoints,
data pipeline, errors, and job history
- foxhunt-traces.json: 12 panels with service map, trace search,
latency by service, and error rate from Tempo
- foxhunt-cockpit.json: added Trace Health row (spans/sec, error rate,
spans by service) and DS_TEMPO datasource input
- import.sh: added 3 new dashboards to ConfigMap group and API fallback
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The #[instrument] import was incorrectly inserted inside a `use super {}`
block by the previous commit, causing syntax errors.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Enable GitLab Prometheus to auto-discover and scrape metrics from all
foxhunt pods by adding gitlab.com/prometheus_scrape annotations to the
pod template metadata of 9 service deployments and the training job
template.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace direct tracing_subscriber::fmt::init() calls with the unified
common::observability::init_observability() in api_gateway, web-gateway,
data_acquisition_service, broker_gateway_service, and trading_agent_service.
All 8 services now emit JSON logs + OTLP traces to Tempo.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previously init_observability() called init_json_logger() (which set the
global subscriber with a fmt layer) and then init_tracing() (which only
set the global tracer provider). Because no tracing_opentelemetry::layer()
was ever added to the subscriber, #[instrument] spans never reached Tempo.
Now init_observability() builds a single Registry subscriber that
composes an EnvFilter, a JSON fmt layer, and an optional OTLP layer
(via tracing_opentelemetry). The OTLP layer gracefully degrades to None
when Tempo is unavailable, so logging always works.
Also adds build_otel_tracer() to tracing_config.rs which returns
Option<Tracer> for the caller to wrap in an OpenTelemetryLayer inside
their subscriber stack, letting the S type parameter be inferred
correctly by the compiler.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Web-gateway routes through api-gateway for the foxhunt.tli proto, but
api-gateway requires JWT Bearer auth on all gRPC requests. This adds a
service_auth module that mints short-lived service JWTs (matching the
api-gateway's expected claims: iss, aud, roles, permissions) and injects
them into all 15 REST proxy calls and 4 gRPC stream bridges.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Web-gateway routing:
- Point TRADING_SERVICE_URL at api-gateway (proto mismatch fix)
Web-gateway uses foxhunt.tli.TradingService proto but was connecting
directly to trading-service which implements trading.TradingService.
api-gateway already proxies Subscribe* → Stream* correctly.
GitLab KAS:
- Disable gitlab_kas in appConfig to stop sidekiq NotifyGitPushWorker
errors (KAS pod was already disabled but Rails still tried to connect)
Trading service monitoring (3 stubs → real):
- AcknowledgeAlert: real alert lookup + state mutation in shared store
- GetActiveAlerts: returns actual active alerts from in-memory store
- StreamAlerts: now persists generated alerts (capped at 1000 entries)
Trading service ML streams (2 stubs → real):
- StreamModelMetrics: emits real inference_count, error_count, latency
per model every N seconds from the RuntimeModelInfo registry
- StreamSignalStrength: emits per-symbol signal aggregation from model
ensemble weights and latency confidence
Backtesting service:
- stop_backtest: real CancellationToken cancellation (was no-op)
Tokens stored per-backtest, execute_backtest wraps strategy call
in tokio::select! for immediate cancellation
Deleted 7 empty placeholder files:
- 4 Wave D regime stubs (dynamic_stops, ensemble, performance_tracker,
position_sizer) — comment-only files, never wired
- 2 Wave 3 feature stubs (microstructure, statistical)
- 1 PPO stub (unified_ppo.rs — empty struct definitions)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add ohlcv_bars table migration (050) for trading_service prediction loop
- Fix kube-state-metrics RBAC: add admissionregistration.k8s.io and
certificates.k8s.io API groups to suppress forbidden errors
- Fix web-gateway gRPC stream reconnect: detect Unimplemented via both
status code and error message text (proxy may remap codes)
- Add automated migration step to CI deploy pipeline with tracking table
(_applied_migrations) so migrations are applied before service rollouts
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
write-manifest needs compile-services and compile-training, but those
jobs use changes: rules. When a commit doesn't touch source code (e.g.
infra-only changes), those jobs don't exist in the pipeline, causing
GitLab to abort pipeline creation with 0 jobs for ALL stages.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Grafana dashboards are provisioned via sidecar ConfigMaps (label
grafana_dashboard=1), making the API endpoint reject overwrites. The
import script now resolves ${DS_*} variables and deploys via kubectl
ConfigMap updates with automatic Grafana restart. Falls back to API
import when kubectl is unavailable or SKIP_CONFIGMAP=1.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add foxhunt-cockpit.json — single-page overview with 7 rows (health bar,
service status, resources, GPU/training, restarts/network, errors, live
logs) pulling critical signals from all domain dashboards.
Migrate all 10 dashboards from hardcoded datasource UIDs to ${DS_PROMETHEUS},
${DS_LOKI}, ${DS_TEMPO} variables with __inputs sections for portable import.
Update import.sh to auto-detect datasource UIDs from Grafana API at deploy
time, using /api/dashboards/import endpoint.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Disable file logging in init_observability — use console-only (stdout).
Containerd captures stdout, Promtail ships to Loki. Eliminates
"Failed to create log directory: Permission denied" errors on 3 services.
- Remove LOG_DIR env vars and /app/logs emptyDir volumes from 5 deployments.
- Add OTEL_EXPORTER_OTLP_ENDPOINT=tempo to 5 services that were missing it
(api-gateway, web-gateway, data-acquisition, trading-agent, broker-gateway).
- Make OTLP tracing init non-fatal — services degrade gracefully if Tempo is
unavailable instead of failing entire observability stack.
- Stop web-gateway gRPC stream retry on Unimplemented status. When trading-service
doesn't implement streaming endpoints, log once and stop instead of retrying
every 60s indefinitely.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
