OTLP fix: opentelemetry-otlp 0.27 bundled tonic 0.12 while the workspace
uses tonic 0.14, making with_channel() impossible (type mismatch). Upgrading
to otel-otlp 0.31 aligns both on tonic 0.14, enabling explicit Channel
construction that respects http:// scheme (no spurious TLS negotiation).
API migrations (otel 0.27→0.31):
- TracerProvider → SdkTracerProvider
- with_batch_exporter(exporter, runtime) → with_batch_exporter(exporter)
- Resource::new(vec![...]) → Resource::builder().with_service_name().build()
- global::shutdown_tracer_provider() removed (Drop-based shutdown)
- opentelemetry-otlp feature "tonic" → "grpc-tonic"
DCGM fix: remove runtimeClassName: nvidia from DaemonSet — Scaleway Kapsule
GPU pools use nvidia runtime as default containerd handler. The RuntimeClass
CRD is only created by the full GPU Operator, not the device plugin alone.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DCGM exporter v3.x emits lowercase metric names (dcgm_gpu_utilization)
but dashboards used the old uppercase format (DCGM_FI_DEV_GPU_UTIL).
Fixed across 4 dashboard files:
- foxhunt-cockpit.json (Infrastructure Cockpit)
- foxhunt-training-cockpit.json (Training Cockpit)
- foxhunt-gpu-training.json (GPU Training)
- gpu-overview.json (GPU Overview)
Replaced DCGM_FI_DEV_FB_TOTAL with (dcgm_fb_used + dcgm_fb_free)
since DCGM v3 doesn't emit a total metric.
Redeployed all dashboard ConfigMaps and restarted Grafana.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three fixes for GPU metrics not appearing in dashboards:
1. Added runtimeClassName: nvidia so DCGM can access NVML/GPU devices
2. Custom counters ConfigMap without DCP/profiling metrics (avoids
SYS_ADMIN requirement that caused fatal crash)
3. Added gitlab.com/prometheus_* annotations so GitLab's Prometheus
discovers and scrapes the DCGM pods
Increased memory limit 128Mi → 1Gi (was OOMKilled).
Metrics now flowing: gpu_utilization, fb_used/free, power_usage, temps.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The L4 pool was unused: all training routes to ci-training (L40S) and
all compilation routes to ci-compile-cpu (POP2). Disabled in terragrunt
and applied to destroy the pool. Removed all ci-rl references from K8s
manifests and CI comments.
Final pool layout:
- ci-compile-cpu (POP2-32C-128G) — Rust compilation
- ci-training (L40S-1-48G) — all GPU training + hyperopt
- services (DEV1-L) — production services
- gitlab (GP1-XS) — GitLab CE
- gpu-dev (GP1-L) — DevPod development
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Applied terragrunt to recreate the L4 GPU pool with the correct name
`ci-rl` (was `ci-compile` due to immutable Scaleway pool names).
Updated all K8s manifests and comments to match. Removed the 3 stale
`moved` blocks from main.tf since the state renames are now applied.
Pool naming is now consistent across Terraform, Scaleway, and K8s configs:
- ci-compile-cpu (POP2-32C-128G) — CPU compilation
- ci-rl (L4-1-24G) — RL training / CUDA compile
- ci-training (L40S-1-48G) — supervised training + hyperopt
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RL runner only handles GPU training jobs, so global nvidia runtime
is safe and required (KUBERNETES_RUNTIME_CLASS_NAME override wasn't
being applied). Main runner keeps overwrite-allowed pattern.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pool mapping:
- ci-compile-cpu (POP2-32C-128G): Rust compile, web dashboard, manifest
- ci-compile (L4): CUDA compile with stubs only
- ci-training (L40S): ALL training (RL + supervised)
Main runner default → ci-compile-cpu
RL runner default → ci-training
.train-rl-base → explicit ci-training node selector
Replaced all stale ci-rl references with correct pool names.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tempo was OOMKilled every ~30 min at 512Mi, causing web-gateway OTLP
export errors. Increased to 1Gi.
RL runner had cpu_request_overwrite_max_allowed=4000m but hyperopt RL
jobs need 6000m for parallel PSO trials. Increased to 8000m.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove runtime_class_name="nvidia" from runner configs (applied to ALL
pods, broke compile on CPU-only ci-compile-cpu nodes). Instead use
runtime_class_name_overwrite_allowed + KUBERNETES_RUNTIME_CLASS_NAME
per-job on GPU templates (.train-rl-base, .train-validate-base).
Also routes build-web-dashboard and write-manifest to ci-compile-cpu.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
cert-manager stores certs as tls.crt/tls.key but MinIO expects
public.crt/private.key — add volume items projection.
mc client doesn't use MC_CA_CERTIFICATE env; instead copy CA cert to
~/.mc/certs/CAs/ before running mc alias set.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Switch all MinIO communication from HTTP to HTTPS across the entire stack:
- MinIO deployment: mount TLS secret, tcpSocket probes, HTTPS init-buckets
- 11 service YAMLs: HTTPS rclone endpoint + CA cert volume mount
- Training job template + train.sh: HTTPS for fetch-binaries and uploader
- CI pipeline (.gitlab-ci.yml): all 7 rclone exports use HTTPS + CA cert
- GitLab runner values: minio-ca-cert ConfigMap volume for CI pods
- Rust: CA cert loading via MINIO_CA_CERT_PATH in training_uploader,
storage backend, data_acquisition uploader, and k8s_dispatcher
- Cert generation script (infra/scripts/generate-minio-tls.sh)
Fixes training uploader S3 upload failures caused by with_allow_http(false)
connecting to an HTTP endpoint.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The job template referenced a non-existent 'gpu-training' pool. The
actual L40S GPU pool in Kapsule is named 'ci-training' (scale-from-zero,
0→1 autoscaling).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GitLab's bundled Prometheus only scrapes pods with gitlab.com/prometheus_scrape
annotations, not the standard prometheus.io/* ones. Without this, the
Infrastructure Cockpit shows no data for node/pod/cluster metrics.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The existing `global.appConfig.gitlab_kas.enabled: false` only affected
the KAS subchart config but was ignored by the chart template that
generates gitlab.yml. Adding `global.kas.enabled: false` properly sets
`gitlab_kas.enabled: false` in gitlab.yml and removes the orphaned
gitlab-kas Service, stopping ~5 GRPC::Unavailable errors per git push.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move all 3 cockpit dashboards (Cockpit, Trading Cockpit, Training
Cockpit) into a single "Cockpits" folder. GPU & Training Jobs moved
to Infrastructure. 4 folders total: Cockpits, Operations,
Infrastructure, CI-CD.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Strip "Foxhunt - " prefix from all dashboard titles since the folder
structure already provides context. Use clean folder names without
slashes (Trading, Training, Operations, Infrastructure, CI-CD) and
per-folder provisioning providers to prevent duplicate folder creation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Dashboard Loki queries used naive `|= "ERROR"` string matching which
created a self-referential feedback loop: Loki and Grafana log query
execution text containing "ERROR" at INFO level, inflating the error
count by ~19/min with phantom matches.
Add `container!~"loki|grafana|grafana-sc-dashboard|prometheus-server"`
filter to all 11 affected queries across 4 dashboards (cockpit,
overview, logs, services).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replaces single "Foxhunt" folder with Trading, Training, Operations,
Infrastructure, and CI/CD groupings. Updates ConfigMap definitions in
import.sh and Grafana Helm values to use foldersFromFilesStructure.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace 28 placeholder PromQL expressions with real metric names:
- Active Jobs: kube_job_status_active from kube-state-metrics
- Training Progress: foxhunt_training_epoch_loss, current_epoch, etc.
- GPU Resources: DCGM_FI_DEV_GPU_UTIL/FB_USED/GPU_TEMP/POWER_USAGE
- Model Quality: foxhunt_training_eval_accuracy/f1/precision/recall
- Checkpoints: foxhunt_training_checkpoint_saves/failures/duration/size
- Data Pipeline: foxhunt_training_batches_processed, data_load_seconds
- Errors: foxhunt_training_nan_detected/gradient_explosion/feature_errors
- Job History: kube_job_complete_time/start_time for duration
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace placeholder metric names (traces_spanmetrics_calls_total,
traces_spanmetrics_latency_*) with real Tempo metrics
(tempo_distributor_spans_received_total, tempo_spanmetrics_latency_*).
Switch Recent Traces panel from nativeSearch to TraceQL query.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace all placeholder PromQL expressions with real metric names
that match actual Prometheus metrics emitted by the trading system.
Add $model, $symbol, and $service template variables for filtering.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace traces_spanmetrics_calls_total with tempo_distributor_spans_received_total
in all 3 Trace Health panels (Spans/sec, Trace Error Rate, Spans/sec by Service).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Grafana sidecar supports grafana_folder annotations to place
dashboards in specific folders. Without this, all provisioned
dashboards land in the General folder. Add Foxhunt/Infrastructure/CI-CD
folder mapping to both CI pipeline and import.sh.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two bugs found during validation:
1. rclone defaults to HTTPS when endpoint has no scheme — MinIO serves
plain HTTP. All RCLONE_S3_ENDPOINT values now include http:// prefix.
2. rclone copyto returns exit 0 for non-existent S3 keys (empty bucket).
initContainers crashed on chmod of missing file instead of falling
through to PVC cache. Added `&& [ -f /binaries/$SERVICE ]` guard.
Also: deploy stage uses `kubectl apply -k` (not -f) for minio/ dir
to properly handle kustomization.yaml.
Deleted stale s3-credentials secret from cluster.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move all compiled binary uploads/downloads from public Scaleway S3 to
MinIO running inside Kapsule. Keeps proprietary binaries and trained
models off the public internet.
- CI compile/upload jobs: Scaleway → MinIO endpoint + credentials
- train.sh: all 3 rclone blocks → RCLONE_CONFIG_MINIO_*
- Deploy stage: apply MinIO manifests + wait for readiness before rollout
- Add infra/k8s/minio/ (Deployment, Service, Secret, PVC, init-buckets Job)
- Update s3-credentials.secret.example → minio-credentials format
IaC Terragrunt jobs and SCW registry pulls intentionally unchanged.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- foxhunt-trading-cockpit.json: 22 panels covering signal quality,
execution, risk, ML model health, and latency
- foxhunt-training-cockpit.json: 36 panels covering active jobs,
training progress, GPU resources, model quality, checkpoints,
data pipeline, errors, and job history
- foxhunt-traces.json: 12 panels with service map, trace search,
latency by service, and error rate from Tempo
- foxhunt-cockpit.json: added Trace Health row (spans/sec, error rate,
spans by service) and DS_TEMPO datasource input
- import.sh: added 3 new dashboards to ConfigMap group and API fallback
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Enable GitLab Prometheus to auto-discover and scrape metrics from all
foxhunt pods by adding gitlab.com/prometheus_scrape annotations to the
pod template metadata of 9 service deployments and the training job
template.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Web-gateway routing:
- Point TRADING_SERVICE_URL at api-gateway (proto mismatch fix)
Web-gateway uses foxhunt.tli.TradingService proto but was connecting
directly to trading-service which implements trading.TradingService.
api-gateway already proxies Subscribe* → Stream* correctly.
GitLab KAS:
- Disable gitlab_kas in appConfig to stop sidekiq NotifyGitPushWorker
errors (KAS pod was already disabled but Rails still tried to connect)
Trading service monitoring (3 stubs → real):
- AcknowledgeAlert: real alert lookup + state mutation in shared store
- GetActiveAlerts: returns actual active alerts from in-memory store
- StreamAlerts: now persists generated alerts (capped at 1000 entries)
Trading service ML streams (2 stubs → real):
- StreamModelMetrics: emits real inference_count, error_count, latency
per model every N seconds from the RuntimeModelInfo registry
- StreamSignalStrength: emits per-symbol signal aggregation from model
ensemble weights and latency confidence
Backtesting service:
- stop_backtest: real CancellationToken cancellation (was no-op)
Tokens stored per-backtest, execute_backtest wraps strategy call
in tokio::select! for immediate cancellation
Deleted 7 empty placeholder files:
- 4 Wave D regime stubs (dynamic_stops, ensemble, performance_tracker,
position_sizer) — comment-only files, never wired
- 2 Wave 3 feature stubs (microstructure, statistical)
- 1 PPO stub (unified_ppo.rs — empty struct definitions)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add ohlcv_bars table migration (050) for trading_service prediction loop
- Fix kube-state-metrics RBAC: add admissionregistration.k8s.io and
certificates.k8s.io API groups to suppress forbidden errors
- Fix web-gateway gRPC stream reconnect: detect Unimplemented via both
status code and error message text (proxy may remap codes)
- Add automated migration step to CI deploy pipeline with tracking table
(_applied_migrations) so migrations are applied before service rollouts
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Grafana dashboards are provisioned via sidecar ConfigMaps (label
grafana_dashboard=1), making the API endpoint reject overwrites. The
import script now resolves ${DS_*} variables and deploys via kubectl
ConfigMap updates with automatic Grafana restart. Falls back to API
import when kubectl is unavailable or SKIP_CONFIGMAP=1.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add foxhunt-cockpit.json — single-page overview with 7 rows (health bar,
service status, resources, GPU/training, restarts/network, errors, live
logs) pulling critical signals from all domain dashboards.
Migrate all 10 dashboards from hardcoded datasource UIDs to ${DS_PROMETHEUS},
${DS_LOKI}, ${DS_TEMPO} variables with __inputs sections for portable import.
Update import.sh to auto-detect datasource UIDs from Grafana API at deploy
time, using /api/dashboards/import endpoint.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Disable file logging in init_observability — use console-only (stdout).
Containerd captures stdout, Promtail ships to Loki. Eliminates
"Failed to create log directory: Permission denied" errors on 3 services.
- Remove LOG_DIR env vars and /app/logs emptyDir volumes from 5 deployments.
- Add OTEL_EXPORTER_OTLP_ENDPOINT=tempo to 5 services that were missing it
(api-gateway, web-gateway, data-acquisition, trading-agent, broker-gateway).
- Make OTLP tracing init non-fatal — services degrade gracefully if Tempo is
unavailable instead of failing entire observability stack.
- Stop web-gateway gRPC stream retry on Unimplemented status. When trading-service
doesn't implement streaming endpoints, log once and stop instead of retrying
every 60s indefinitely.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add nginx server block for api.fxhnt.ai with grpc_pass to the
api-gateway ClusterIP service. Add Scaleway DNS A record via Terraform
module. Fix fxt gRPC client to enable TLS when connecting to https://
endpoints (tonic requires explicit ClientTlsConfig).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- init_observability now reads LOG_DIR env var for log directory (falls back
to relative logs/ path). Gracefully disables file logging if directory
cannot be created instead of failing the entire observability init.
- Add ServiceAccount, Role, and RoleBinding for ml-training-service to
create/get/list/watch/delete batch/v1 Jobs (K8s training dispatch).
- Add LOG_DIR=/app/logs env to ml-training-service, trading-service, and
backtesting-service deployment YAMLs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- All services: maxSurge=0, maxUnavailable=1 (ReadWriteOnce PVCs
deadlock with maxSurge=1 — new pod can't attach volume while old
pod still holds it)
- ml-training-service: switch httpGet /health:8080 to tcpSocket:50053
(service crashes on log dir permission, never binds health port)
- GPU overlays: same strategy and probe fixes
- job-template.yaml: use generateName instead of placeholder name
(training-MODEL-TIMESTAMP fails k8s RFC 1123 validation)
- deploy job: increase rollout timeout 120s → 300s
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Scaleway quota limits POP2-32C-128G to 1 instance. Reduce each compile
job from 28→14 CPUs so compile-services and compile-training run in
parallel on the same 32-vCPU node instead of sequentially.
Also fix data-acquisition-service health probe: service doesn't implement
gRPC health protocol, switch to tcpSocket probe on port 50057.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The foxhunt-runtime base image runs as uid 1000 (foxhunt user), but
new PVC volumes are owned by root. The initContainer needs root to
write the cached binary to the PVC. Main container still runs as
foxhunt (non-root) for security.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Scaleway block storage minimum volume size is 1Gi. The 500Mi PVCs
were stuck in Pending state, blocking all service pod scheduling.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
